Testenvironment for micropython: **microOctopus**

[hmaerki]

Hi micropython core developers!

I was working for the last four years for a quantum computing company (zhinst.com) building fully automated testenvironments for complex hardware devices.
In a few years, I will be retired and then I will even have more time to do fun stuff.

Lets talk about a vision: microOctopus!

Main goals

• Automated regression testing
  • Automated firmware updates of micropython boards.
  • Test runs of these boards against a set of sensors (gadgets).
• Triggered from the github workflow, for example on commits or pull requests.
• Ability to measure signals and timing using a scope.
• Remote access to microOctopus similar to github codespaces (but with real hardware attached).

Benefits

• Fast feedback about code quality.
• High test coverage.

Sketch of microOctopus

https://github.com/hmaerki/experiment_microoctopus

Questions to the core micropython developers

• Would microOctopus ease your daily work?
• Which micropython boards, and which sensors (gadgets) would be of the highest interest?
• Which tests would be of the highest interest?
• Would you be interested in
  • owning your personal microOctopus instance?
  • using the microOctopus instance which is hosted by me?
  • hosting a microOctopus instance and share it with your team?

[peterhinch]

See this comment: it seems the maintainers have plans for a "hardware in the loop" test environment. Perhaps there is scope for collaboration?

[Josverl]

I like the goal, but have a few questions on the approach.

• i've ran tests hardware integration tests before using a github selfhosted runner with directly attached boards. While that worked most of the times , the software still cannot the buttons needed for some testcases, or to recover from failed tests. This lead to quite a few false positives.
  I understand your approach is to reduce this by using these 'hardware tentacles', but $50 to run a $4 board seems skewed to me.

• What do you plan to host the core microOctupus on, a Pie 4 or similar

• In your setup I lack the component I personally test the most: a serial/USB connection to the host, and in my testing to hosts running different OSes ( Win / different Linuxes / Mac )

• for instance the mpremote pytests that I have written to find and fix a few of the pyboard.py-esp32 issues needed to be validated across different OSes and host CPU architectures. Similar approach for the the micropython-magic tests and the pymakr tests I was involved in before.
  For that not only would a tentacle need to be extended with a capability to connect USB, but testing would also need to be coordinated with a test running on a specific host that needs to be USB-connected to just the correct DUT. not impossible , but certainly not trivial with lots of devices sharing the same USB VID/PID

In my personal experience I regularly need flash and run tools on some 8-15 devices, and similar for testing.
Currently for infra, this costs me $30 for a USB hub, and a few hours of my time.
Most of the effort is in determining what/how to flash to which board, and some of the the details of this change a bit each release.

If I understand your design this would cost me somewhere in the range of a couple of hundred dollar for my own kit, which I would find prohibitive, but might want to try out on a limited scale.
if there was a way to self assemble a (prototype) tentacle, that would give me a way to evaluate.

I would definitely be interested to see if, and which, process I could run on your setup.

[hmaerki]

Thanks for your reply!
This is exactly what I need: Feedback from someone who already did intense testing!

i've ran tests hardware integration tests before using a github selfhosted runner with directly attached boards. While that worked most of the times , the software still cannot the buttons needed for some testcases, or to recover from failed tests. This lead to quite a few false positives.
I understand your approach is to reduce this by using these 'hardware tentacles', but $50 to run a $4 board seems skewed to me.

A tentacle is a rp2040 with a FET which powers the DUT and ~10 photocouplers: jlcpcb.com produces assembled pcb's for ~USD 20). Is it worth the time it saves?

Why couldn't you recover from failed tests? Is powercycling and the boot-button not sufficient?

What do you plan to host the core microOctupus on, a Pie 4 or similar

A Pie and big USB hub (2 usb connections per tentacle) is ok.

In your setup I lack the component I personally test the most: a serial/USB connection to the host

The DUT is connected directly to the host via USB, the tentacle is not involved here. As long as the DUT is not powered, it is invisible to the host. Oups - thats not correct - the DUT is powerered via USB... I need some time to think about this...

running different OSes ( Win / different Linuxes / Mac )

microOctopus is not intended to test USB. However why not?

If I understand your design this would cost me somewhere in the range of a couple of hundred dollar for my own kit, which I would find prohibitive, but might want to try out on a limited scale.

The investment pays off, when tests can be fully automated. When Pull requests automatically will be tested etc.

if there was a way to self assemble a (prototype) tentacle, that would give me a way to evaluate

The intention is, that you get lets say 10 tentacles. They are generic and have a breadboard area where you may mount your DUTs and connect the to power and the bus to the sensors etc.

I would definitely be interested to see if, and which, process I could run on your setup.

Nice to hear!
From my side, I would be interested to use some of your pytests to verify usefulness and stability of microOctopus.

[mattytrentini]

This is absolutely something we want! For the record, you probably know this, but is generally referred to as HIL (Hardware In the Loop) testing.

I've started heading down this road; I have put aside a PC (i5 NUC), USB hub and enough microcontrollers to cover a large portion of ports.

I've also started a similar design sketch for the 'tentacle' you mentioned, though I was thinking of basing it on an STMH5 or even H7 since the RP2040 has some limitations (though the PIO is compelling for some testing). My design was in two parts however; a base board and an adaptor. The base board would host the test micro (H5/H7) and would be consistent for all 'tentacles', it would also provide a couple of high-density sockets breaking out all the necessary GPIOs. The adaptor would plug into the sockets and provide an interface to match each of the devices under test.

Just for everyone else's benefit, in the testing space, the micro being tested is known as the DUT (device under test) and the micro performing the testing is the ATE (automated test equipment).

I've also started the skeleton of a MicroPython High-Level Test Suite which could provide some initial tests.. It's intended to execute on hardware and instantiate many of the standard MicroPython classes to ensure they conform to the appropriate interfaces. No ATE is required. It's a shallow set of tests but may be a good place to start given that it's relatively simple to implement.

To address your questions directly:

Would microOctopus ease your daily work?

Regressions between builds, particularly for releases, is a major issue that HIL testing can help solve. I use MicroPython commercially and updating between major releases can be challenging.

Maybe not a 'daily' problem but still important!

I'd also just generally like to help out the MicroPython project here; HIL tests in CI are highly desirable.

Which micropython boards, and which sensors (gadgets) would be of the highest interest?

I have enough hardware to cover most ports. Most important to me: stm32, esp32, rp2. Then: mimxrt, samd, renesas, esp8266, nrf.

I'd like to see sensors tested too but it's far less important than testing the micros, for me.

Which tests would be of the highest interest?

Probably start with the most simple (Pin, I2C, SPI) and build on it.

Would you be interested in

Owning/maintaining it myself. Or contributing to tests if someone else is sufficiently motivated to run a system.

[peterhinch]

Out of interest how does "hardware in loop" deal with interfaces such as SPI and I2C? I2C in particular has challenges like clock stretching and error conditions.

[mattytrentini]

Agree that some thought needs to go into the edge cases. Bur right now we have no tests. Testing basic SPI and I2C comms can be done relatively easily - let's start there.

[dhalbert]

Maybe of slight interest: we attempted something similar for CircuitPython years ago. But it did not progress due to various logistical issues. Using an RPi as the host was a mistake: the flash cards were going bad too fast.

https://github.com/adafruit/rosie-ci

(after Rosie the Robot)

[Josverl]

flash cards are perfect for occasional use - good ones last a bit longer, but still degrade too quick.
For continuous use, a small USB3 connected SSD works well, but adds to the BOM

[Josverl]

How would one go about testing incoming signals ?
Someone touching a button , capacitive input should be automatable.
in other cases a "Human in the Loop" will be needed

But for incoming signal processing there should be an option to test as well.
For instance one of my in flight projects is on a rp2040s sending and receiving DMX - ( essentially UART with some additional funk)

This is still in experimentation phase , so I do not want to spend effort on maintaining test scripts that i need to change with every iteration
In order to test that I have a setup with two boards and some probes connected to a logic analyzer (saleae basic) that I can control from my notebook / test to take captures.

Something like :

Currently I control this using a Jupyter notebook with micropython-magic to control the two DUTs
The challenge is that the effort to write the TX code is multiplied when I also need to use that to generate the test signal
and any incorrect assumption that i make on the signal, may/will give false positives / negatives.

So I would suggest that an essential part of this setup would be to have something that can record and replay signals that can be used as a test input. A logic analyzer can record , but not replay. ( id be interested in a solution to that)

[andrewleech]

Out of interest how does "hardware in loop" deal with interfaces such as SPI and I2C? I2C in particular has challenges like clock stretching and error conditions.

It's an important question, the reality is generally application specific.

Ideally, the host micro could present itself as a SPI & I2C peripheral device, wired to the device under test (DUT,) then write some multitests to send data back and forth in various forms to test.

The DUT boards can gradually be made more complicated over time, adding things like flash, sdcards, I2S, PWM, UART checks etc.
This would actually be a reason to use something like an esp32 as the host chip as it can often use any pin for any peripheral, so has more flexibility for running as a test peripheral wired to many/most pins of the DUT.
The rp2 PIO is probably better again though, potentially being used to measure pwm, I2S signals etc.

Compatibility with particular peripheral devices isn't really the goal of using here, it's proofing that the MCU peripheral is still working just as well as it was on the previous commit - I think of HIL tests as more about catching regressions than testing during initial development.

[andrewleech]

For instance one of my in flight projects is on a rp2040s sending and receiving DMX - ( essentially UART with some additional funk)

The way I generally look at it, the first step to this kind of testing is to break it down as much as possible and test everything you can on PC / Unix port first. Then only go to HIL for things that can't be done that way.

Eg. application specific signals like this can often be tested in two parts; the UART hardware needs to be tested to prove it's working to the Stream protocol, then separately the DMX protocol can be unit tested over any Stream, eg unix port socket.

For code that's under active development, it's often good to use "test driven development", testing your DMX protocol on Unix port in small fast tests.
Then only move it to hardware once it's working.
Changes to the protocol shouldn't impact whether the UART/stream test passes (until you get into the weeds like break signal etc).

Though obviously this Unix testing reaches its limit when the test needs to also involve the custom hardware receiving the DMX... then we're back to full HIL of the whole product.

Long story short, HIL test hardware is always application specific, wiring up custom IO via labjack or a host micropython device that can trigger inputs and outputs of the device under test in an automated fashion.

For a generic micropython test suite we will have to focus on a minimum set of MCU / DUT signals we want to test on everything and make the tentacle board handle them.

Too little capability and we won't try enough to bring value. Too much capability and we'll never finish designing and building it!

Start small, get it working, grow over time.

[Josverl]

The way I generally look at it, the first step to this kind of testing is to break it down as much as possible and test everything you can on PC / Unix port first. Then only go to HIL for things that can't be done that way.

The code in question is written in rp2040 PIO (so the unix port is of no use for that) and debugged, tested with a RP2040 PIO emulator first.
So TDD = ✅

• start small & KISS : could not agree more ( Minimal Lovable Product)
• try to make sure that design is open ended and can be extended later : should be part of any design

[andrewleech]

A logic analyzer can record , but not replay. ( id be interested in a solution to that)

I haven't tried this, but it's another vote for using rp2040 as the tentacle processor: https://github.com/dotcypress/ula

If needed, the PIO can act as a logic analyser, could possibly replay lots of things too

[mattytrentini]

The RMT of the ESP32 is a strong alternative for this purpose too.

[hmaerki]

@andrewleech: I haven't tried this, but it's another vote for using rp2040

microOctopus should - at least in the first versions - just implement the infrastructure on the tentacles which is:

• Updating the BoardDUT with a given firmware
• connect to BoardDUT tentacles with gadgets tentacles
• And this is a stable fully automated way

Now everybody may assemble tentacles: Implement scope/playback mechanisms as @andrewleech and @Josverl described.
I personally would profit from a tentacle with a PicoScope 2205A which is great to analyze protocols like I2C or Modbus RTU.

So the primary questions for me are: How to build a generic tentacle that may cover the most important use cases.

A test setup to test I2C slave functionality, for example clock streching, could be:

• Tentacle DUT A: pyboard as I2C slave
• Tentacle DUT B: rp2040 as I2C slave
• Tentacle DUT C: esp32 as I2C slave
• Tentacle X: rp2040 as I2C master - This is NOT DUT, it is the reference to test the DUTs.
• Tentacle Y: PicoScope 2205A - take scope shots and fail the test if the I2C analyzer detects a problem.

Testprocedure

• Repeat for DUT A (pyboard), DUT B (rp2040) and DUT C (esp32):
  • Install firmware 1.23-preview (Software under test) on DUT A (pyboard).
  • Install firmware 1.18 (reference software) to tentacle X.
  • Load the python test code to DUT A und tentacle X.
  • Inkbus: Connect I2C on tentacles A, X and Y (close corresponding photo couplers).
  • Run tests and collect results. Take scope shots with Tentacle Y

Questions about the tentacle design in this early conceptional phase:

• What is required to allow firmware updates of all kind of micropython boards?
• How may we connect the tentacles A, B, C, X, Y to be able to run above test? Reed relais, photo couplers?
• What could go wrong? How can we ensure that microOctopus runs without becoming stuck? How may we detect errors of the infrastructure, for example leaking photo couplers? How to recover micropython boards or gadgets?

Start small, get it working, grow over time.

[mattytrentini]

I would suggest starting as simple as possible; don't try to connect the 'tentacles'. Deploy firmware to the DUT (latest build) and ATE (last stable build) from the connected computer using the normal tools for the devices (esptool, dfu etc). Everything should be automated, there should be no human in the loop. Drop other external devices, they can be added later.

Just use a micro to observe another micro - it's the best bang-for-buck though we won't be able to test everything. Aim to test something like GPIO, SPI, I2C, PWM, UART first, Timers, ADC, WDT, Networking next. Add others as we run into issues (like the recent rp2 thread issue) or as motivation dictates.

I'd also drop the opto's and just directly connect GPIO's.

The fallback should be to reset the DUT and ATE in a clean state.

100% agree with starting small. :)

[hmaerki]

Hi @mattytrentini

I just had a look at https://pybricks.com. Very cool! Congratulations!

Matth: I would suggest starting as simple as possible; don't try to connect the 'tentacles'.
...
Deploy firmware to the DUT (latest build) and ATE (last stable build) from the connected computer using the normal tools for the devices (esptool, dfu etc).
...
I'd also drop the opto's and just directly connect GPIO's.

I completely agree. To be able to automatically deploy firmware, we already have to design our ATE hardware.

Requirements are

• USB with automated power on/off: Some USB hubs have power buttons - cool! However, they use quite complex chipsets! To complex for microOctopus!
• One microOctopus should support up to 20 tentacles. This requires dozens USB cables. This is not acceptable for microOctopus.

Why not just take 1 USB connection which connects to the currently active DUT?

My current approach is to provide USB on the ribbon wire bus (the inkbus which connects the tentacles). Only one tentacle at the time is allowed to connect to these USB wires.
This simplifies the test setup a lot! No USB hubs, no USB cables, just the inkbus! But derived requirements arise:

• inkbus: USB over ribbon wire only works on USB 1.0 Low Speed 1.5 Mbit/s. This is easy to implement with 2 resistors.
• tentacle: To switch the 5V power we may use a FET.
• tentacle: FSUSB3 allows to switch the USB lines.

But do all micropython devices work with USB 1.0?

Exactly this would be my starting point: Use a long ribbon wire and USB 1.0 and start how you suggested: Deploy firmware to the DUT from the connected computer using the normal tools for the devices (esptool, dfu etc).

What is your opinion?

• Would this USB 1.0/inkbus infrastructure work on all micropython boards?
• Are we limited in our test scenarios when just USB 1.0 is available?

[patfrench]

Hi,
I've had a similar project at work.

The difference was that I had to test only one device (with a lot of evolution).

The choice was the pynqZ2 FPGA development board.

Inside the FPGA there's an ARM9 (Zynq PS) and inside the ARM9 there's a python notebook jupyter that can dynamically modify the Logic part of the FPGA (Zynq PL)

But there's also a ready-to-use tool: Logic tools

https://pynq.readthedocs.io/en/v2.3/pynq_overlays/pynqz2/pynqz2_logictools_overlay.html

Unfortunately, I couldn't go any further than just trying it out, as other tasks were more pressing.

The board costs 230€ (it's expensive) and the Vivado development tool (when you need to make in-depth modifications) isn't easy to understand (for me), but it's a bit like micropython: you can do everything in python (it's fun. It's just to share, if it's off topic then sorry)

[hmaerki]

Hi @patfrench! The pynq board is very cool! I also have one: From the Jupyter notebook, you may compile C code and run it on the board, load bitstreams and test it. Based on this experience, I wrote a pytest testenvironment which allows to run integration tests for the FPGA in https://www.zhinst.com/ch/en/products/shfsg-signal-generator.

[patfrench]

wow ! I don't have the same level

[Josverl]

Only one tentacle at the time is allowed to connect to these USB wires.

And only usb 1.0?
Then for my scenarios (host-MCU) it will not work. Nor for multi device tests, or only with limitations.
One could always just add a usb data-only cable...., and control the power through the tentacle connect

I still see validity in MCU-device testing

[hmaerki]

IIUYC , a tentacle will provide the capability to selectively (per tentacle):

power the DUT on a tentacle up or down
mimic handling reset / boot buttons
connecting to / disconnecting from the USB 1.0 ribbon-bus

Yes, this is the plan for the first prototype. The goal is to be able to do reproducible firmware update to all available micropython devices over USB low speed. This is a microOctopus infrastructure feature which will be the tear up to every test.

In your use case, USB is not a infrastructure feature which will be accessed in the tear up - it is the DUT itself! Do you agree?

If that works , that would be good enough for me and keep the rest of the design simple

Let's continue this discussion when I build the first real tentacles! microOctopus should not support your USB2 requirements. However, microOctopus should be flexible enough to be extended with your requirements. So lets have a try then!

My assumption is that it is possible to run pytest in an environment where I can access USB devices.

I would prefer to do as much as possible in docker. In your case, I agree, that testing usb in docker may have very unpleasant side effects. You will need anyhow a runner for every operating system: Different runners accessing the same microOctopus hardware!

[Josverl]

In your use case, USB is not a infrastructure feature which will be accessed in the tear up - it is the DUT itself! Do you agree?

I do not agreed that testing the Host - USB - MCU the USB the DUT itself,
Its the ability to communicate between the host and the MCU , similar as the ability to communicate between the MCU and a LED / I2C.
and based on the different PRs that Ive needed to create and test , not something that we should take for granted.
Therefore its clear to me that there is a requirement to be able to test that automatically.
But its no problem at all if you choose to not implement that requirement. (see below experiment)
not quite sure what you mean with tear up, do you mean the test setup ?

Data Only USB with external power control

Ive done a quit and dirty test based on an esp32 board with a damaged connector.

USB connected only with GND , D+ and D-
power applied from an external source

Power On : Device detected on the bus
Power Off: Device disappears

Repl access > Just works
Flashing the device did not work - from what i can see a few other traces in the boot/flash had been torn out.
that also prevented the USB 1.5 MB flash test that i wanted to do next

[andrewleech]

Many of the target devices here use USB as their standard firmware install mechanism, I would not trust that a USB1 style connection would be possible and have this work in a realiable fashion. Also, I'd expect most tests to be run ultimately via mpremote / pyboard.py which would typically also need to use a usb repl for communication.

inkbus: USB over ribbon wire only works on USB 1.0 Low Speed 1.5 Mbit/s. This is easy to implement with 2 resistors.

Can you use resistors to force a device to only connect in USB 1.0 mode? I'm used to devices always attempting to connect in USB2 mode and failing if the cable isn't good enough.

I was thinking that I then could just take a make a simple USB2 adapter female - male and just not wire in the GND, D+ and D- and not connect V-Bus(+5v))

This definitely works; I've got a powered USB hub with a switch on each port; All the switch does is cut eh 5V line on the usb port, all other pins are always connected. This works fine for power cycling devices (manually).

Also, this could help: https://github.com/mvp/uhubctl - if we can find something from that list then we could just have a bunch of them daisy chained together for all DUT's

Or, each of the tentacles could be based on this which includes both usb hub and debugger "inline" with the single usb port: https://www.hackster.io/news/negimeister-s-clever-usb-debug-probe-built-using-an-rp2040-compute-module-simplifies-swd-2e1f5a74d5d7

[Josverl]

Can you use resistors to force a device to only connect in USB 1.0 mode?

Yes, with the (two) resistors that Hans mentioned above; based on that I found this ib Speed Identification

A USB device must indicate its speed by pulling either the D+ or D- line high to 3.3 volts. A full speed device, pictured below will use a pull up resistor attached to D+ to specify itself as a full speed device. These pull up resistors at the device end will also be used by the host or hub to detect the presence of a device connected to its port. Without a pull up resistor, USB assumes there is nothing connected to the bus. Some devices have this resistor built into its silicon, which can be turned on and off under firmware control, others require an external resistor.

But it seems to me that that same pull-up resistor will then also signal the USB connection to the host as D- will be High, even if the device is powered off. so the Pull-up would need to be connected to the MCU/DUT Vcc to avoid false detections.

Update:
ESP32 - CP210x - with power separated:

trying to force to low speed with resistor :

Host : Windows 11 x64

MCU	interface	USB.10 low speed , Pull Up on D-
esp32	cp210x	Device Descriptor Request Failed
esp8266	CH340	Device Descriptor Request Failed
Microbitv1	USB composite	Device Descriptor Request Failed
Pybv1.1	USB composite	Device Descriptor Request Failed
esp32 S3 - uart	cp210x	Device Descriptor Request Failed
esp32 S3 - USB	USB composite	Device Descriptor Request Failed
samd	USB composite	Device Descriptor Request Failed
Pico Lipo	USB composite	Device Descriptor Request Failed
Pico	USB composite	No Device Detected or Device Descriptor Request Failed
Pico-W	USB composite	No Device Detected or Device Descriptor Request Failed

Detailed info from USB tree viewer

        ---------------- Connection Information ---------------
Connection Index         : 0x04 (4)
Connection Status        : 0x02 (DeviceFailedEnumeration)
Current Config Value     : 0x00
Device Address           : 0x00 (0)
Is Hub                   : 0x00 (no)
Device Bus Speed         : 0x00 (Low-Speed)
Number Of Open Pipes     : 0x00 (0 pipes to data endpoints)
        --------------- Connection Information V2 -------------
Connection Index         : 0x04 (4)
Length                   : 0x10 (16 bytes)
SupportedUsbProtocols    : 0x03
 Usb110                  : 1 (yes)
 Usb200                  : 1 (yes)
 Usb300                  : 0 (no)
 ReservedMBZ             : 0x00
Flags                    : 0x00
 DevIsOpAtSsOrHigher     : 0 (Is not operating at SuperSpeed or higher)
 DevIsSsCapOrHigher      : 0 (Is not SuperSpeed capable or higher)
 DevIsOpAtSsPlusOrHigher : 0 (Is not operating at SuperSpeedPlus or higher)
 DevIsSsPlusCapOrHigher  : 0 (Is not SuperSpeedPlus capable or higher)
 ReservedMBZ             : 0x0

Note that the device was indeed detected as a low speed device, so that part works.
but I used 1 resistor , and @hmaerki mentioned two, not sure how / where to connect though

This just a Q&D test , and perhaps I misunderstood how to force the USB to Low speed or where you intend to apply it.
or it may be that the host now expects low speed , and the MCUs all keep sending high speed.

As the goal is to flash the firmware to the devices over USB 1 Low speed , then I would suggest that some more prototyping should be done before settling on a hardware design, and at least get something that works on a breadboard.

[hmaerki]

@andrewleech
Also, this could help: https://github.com/mvp/uhubctl

Hi Andrew, uhubctl is awesome. I will try this!

I will postpone my prototype with USB low speed!