Compare with arduino CI as a test backend

[ianfixes]

moved as per request from bxparks#63

@bxparks Any discussion of features and potential deficiencies is welcome, but I want to make clear that I understand & appreciate that you're volunteering your time -- it's your initial reactions to this as a contribution that I'm most curious about (since I assume that you have those front-of-mind without having to read too much code). For example, what is the gap between this contribution and one you'd want to accept?

If you do want to dive into the arduino_ci codebase itself, I'll be happy to make time to do that -- on a schedule that works for you.

[bxparks]

So first of all, I know and understand my frameworks far far better than yours, so when I make comparative remarks, you have to forgive any misunderstanding that I have about your framework. When I created AUnit 2-3 years ago (looks like 2018-03 was my first commit), I was vaguely aware of arduino_cl but I had 2 immediate problems with it:

1. I couldn't understand what exactly it did. Probably your documentation was not as extensive. Was it a unit testing framework? Was it a continuous integration framework? To me, those are orthogonal things. I think your framework is both. Initially, I could not figure out whether your unit test code ran on the microcontrollers themselves (like ArdunoUnit did), or whether you provide an alternative Arduino API environment which executes on the host computer, using the host compilers.

2. But more importantly, I couldn't figure out how your framework worked. I don't know Ruby, and I was not going to spend the time and effort to learn it, for the sole purpose of using a unit testing framework for Arduino. Ruby and Python are similar to each other, and are in similar problem-solving space. I already know Python, there is not much incentive for me to learn Ruby. Learning Ruby is not just learning the language, which probably is not difficult. It is learning the language, the common idioms of the language (every language has them), learning the standard and popular 3rd party libraries, and often worst of all, learning the package management systems of the language. The package management systems are bad enough in Python, with multiple competing ones, I don't have the energy to learn yet another package management system. So when your arduino_ci system talks about GemFiles and Gem configs, and various other Gem stuff, I have no idea what it is talking about.

The most valuable thing about my current testing infrastructure is that I understand every single line of it, because I built the whole thing myself. When something goes wrong, I know exactly how to fix it. It will be really difficult for you to convince me to use someone else's system. So that's a subjective thing.

But more objectively, I think my system has a couple of advantages, if I correctly understand arduino_ci:

1. I have decoupled the unit testing framework from the continuous integration framework.

   • My unit testing framework is AUnit. It allows unit tests to be compiled and executed using the Arduino IDE or CLI, under each hardware platform, and run the executable on the microcontrollers themselves. Early on, 2-3 years ago, this was absolutely important for me, because I wanted to catch errors caused by compiler differences (e.g. sizeof(int) problems), and platform differences (e.g. existence of FPSTR() macros, etc). I created the UnixHostDuino project to allow the exact same AUnit tests to run under a Unix environment, with its 64-bit compilers, and 64-bit sizeof(long), etc. Not only that, UnixHostDuino allows me to compile AUnit tests using both g++ and clang++ compilers. Often the clang++ compilers have much better warning messages, so I'm able to catch certain incorrect or ambiguous C++ constructs that g++ ignores.
   • Then my continuous integration framework can be either Jenkins running on a local machine, or a cloud service like GitHub Actions. The same AUnit tests can run in either environments. Using Jenkins, I can run my tests against a small farm of actual Arduino microcontrollers, so that the unit tests are verified on actual hardware. Using GitHub Actions, I can run those same tests in the cloud on every Pull requests.
   • I think the flexibility and composability of the various tools and frameworks (AUnit, UnixHostDuino, Arduino IDE, Arduino CLI, Jenkins, GitHub Actions) is a compelling property of my system.

2. I deliberately designed my system to have as few external dependencies as possible. A system with lots of external dependencies means more complexity, more things to break, more version mismatch problems, more things to install, more things to configure, more things that could go wrong.

   • AUnit is a self-contained library. It has no external dependencies.
   • UnixHostDuino depends only on GNU Make. Makefiles are almost as old as Unix. Most Docker containers use images which automatically include make. For example, the GitHub Actions ubuntu container contains both make and g++, I don't have to install anything else. I don't need even need Python.
   • Jenkins needs the Java JVM, and requires huge amounts of configuration and maintenance for all of its various plugins. I think it's because of this complexity that I have gradually moved away from it as a daily-driver. But my system is composable, and not dependent on Jenkins, so I'm able to use other CI systems, like GitHub Actions.
   • My AUniter toolset, which is used to upload the AUnit tests to the microcontrollers, and validate the test results from the Serial port, requires bash, Python and one python library. I justified this as acceptable because the ESP8266 and ESP32 platforms already require Python for their esptool.py tool, so I figured it was not a huge additional dependency. (Edit: I just realized that the python library that auniter.sh requires is the same python library used by esptool.py, so no extra dependency.)

So I look at your PR, and I see that the various *.ino files have been moved to a new directory test, and they are now all *.cpp files. That's a no-go for me, because I want the ability to upload and run the tests on the actual hardware themselves.

You also have a couple more configuration files than I would like. As a general rule, I am not a fan of configuration files, because that means that I have to understand the framework that is slurping in the config file in order to debug things when something goes wrong. You got a .arduino-ci.yml which seems to configure the arduino_ci, but at first glance, I don't understand why it's necessary, since the whole thing runs under a Unix execution environment. Then you have a Gemfile.lock configuration, which I don't understand at all because I don't know Ruby. Then you have a GitHub Arduino-CI/action@lates thing, which I don't understand what it does, because I haven't learn how to write custom GitHub action thingy. I mean, I could probably figure it out eventually, but if I don't have to, why not make it simple? Then you got your arduino_ci.rb which is your test runner, that I don't understand how it works... because I don't understand Ruby. Compared to my make, which to me, is so much simpler. Granted, my UnixHostDuino.mk is where I have hidden my complexity, but even that file is only 63-lines of GNU Make if you strip out 62-lines of comments.

So those are my first impressions, with the huge caveat that I don't understand how your system works at the detailed level, mostly because I don't understand Ruby. Hope that helps.

[ianfixes]

Thanks. There's a lot of good info in here, I'll try to reply in chunks as way of diving into some specific topics.

[ianfixes]

It's definitely worth acknowledging that we worked to different goals. I actually really like the model you have set up with Jenkins, where hardware is in the loop -- what I would consider "optimal" setup for anyone developing a library. I had to sacrifice that to pursue a way of making an identical build/test environment between the library owner, possible contributors, and CI machine in the middle. I tried to replicate Adafruit's ci-arduino (completely ignoring any attached hardware), which was an enormous bash script with somewhat kludgy config. Trying to wrestle unit testing into that script was a non-starter, so Ruby was my choice for a multiplatform-capable rewrite.

The ruby side of things is uninspiring: find the platform config, iterate over those; find the unit tests, run them; find the examples, compile them. Without arduino-cli available as it is now, that was messy work -- the IDE was not really set up to be automated.

One thing I'm hearing that I hadn't considered before is that by asking users to include the Gemfile in one's project, I'm giving the impression that a user would have to dive into ruby at any point. I think I can/should clear that up by just accepting that the project is mature enough to have them gem install arduino_ci and leave it at that. I'll have to test that out a bit.

I put both the GitHub action and shell-script-based methods of using arduino_ci in this PR, which is probably more confusing than (as I had hoped) illustrative. The GitHub action runs arduino_ci.rb under the hood, but it saves you from having to use the Gemfile in your own project. I can split this into 2 PRs if that would clarify things a bit.

This is all really good feedback, and I appreciate the detail you went into. I have some follow up questions that I'll write up tomorrow about the compiler stuff.

[ianfixes]

One minor clarification to get out of the way: you mentioned that I have too many configuration files. But on the other hand, I deleted almost 30 instances of a Makefile with this PR. What's your feeling on what makes a file more or less necessary in this type of setup?

[ianfixes]

I think what I'm most curious about here is UnixHostDuino, because we both have some incarnation of this. And for the same reason -- if you want to run anything without hardware, you have to mock either the compilation (providing all of the platform support, like avr-libc) or the execution (emulating a microcontroller in an effort similar to QEMU). It's quite a bit easier to mock the software side, at least for me.

Can you talk more about what sort of compiler errors you ran across by using native compilation & unit testing instead of UnixHostDuino? I would definitely be interested in adding some test cases that show shortcomings of my own framework.

[bxparks]

Your remark about the Makefiles is a good one, and it occurred to me as well. It might be a personal thing, but I tend to think of make as having less cognitive overhead than a new config file (in yaml or json or some other format), with all the Ruby code behind it. I've been using makefiles in some form or another for 32 years, though I wouldn't call myself an expert, I'm just very comfortable with it. I think it comes preinstalled on basically all Linux/MacOS flavors that we would care about in this context, so I think of it as part of the OS. So there's no external dependencies, which is one of the goals of my system. If the Makefile is kept reasonably small, debugging is fairly straightforward because it's not a full programming language. Using Makefiles also allows me to add additional processing steps, in front of the Arduino compilation. You probably didn't see that in AceButton but in some of my other libraries, I use code generators to programmatically generate C++ code, and Makefiles is the perfect place for those things, with no additional dependencies, or external scripting language, or config files.

With regards to compiler issues, definitely the most frequent is the sizeof(int) and sizeof(long) problems. The following code will compile perfectly with no compiler warnings on an 8-bit AVR, 32-bit ESP8266, and 64-bit Linux:

const uint16_t DELAY = 1000;
uint16_t myMillis();
uint16_t start = myMillis();
if (millis() - start > DELAY) { // unsigned overflow/underflow perfectly allowed by C++
  ...
}

(Edit: Changed millis() to myMillis() that returns uint16_t instead). However, the code will be functionally wrong on 32-bit and 64-bit, because the uint16_t will be automatically upcast to a 32-bit int, it will do a signed subtraction, and the whole thing will break. The correct code for all architectures is:

if ((uint16_t) (millis() - start) > DELAY) {
  ...
}

For some reason, I hit this a lot in my early Arduino programming stage. Thinking about this some more, I think it's because I don't do this sort of raw programming anymore, I have created enough tools, libraries, and infrastructure that this calculation is handled by something that's already been well-tested.

The sizeof problem peeks out in printf()-like statements, where if you have want to print a uint32_t, you have to explicitly remember to cast it to a (unsigned long) and remember to use a %ld instead of a %d.

UnixHostDuino cannot emulate PROGMEM perfectly, so if the code is even slightly sloppy and does not track the different meanings of const char* accurately, it will run perfectly on Linux, and yet fail on an AVR.

Sometimes, things go in the opposite direction. It segfaults on Linux, but runs perfectly fine on an AVR. If I recall, it sometimes has to do with deferencing 0 which is supposed to be undefined behavior in C++, but I think AVR allows it because of its Harvard architecture.

I didn't keep a detailed record of all the differences and issues. Maybe I should have. Hope this helps?

[ianfixes]

This has been extremely helpful, and I really appreciate the time you took to provide these comments!