Remote debug/test bench

[glitchcore]

We discuss integration tests and automated remote test bench here.

We have flipper board and control board (for example, Raspberry PI). How Rpi can interact with Flipper to perform tests? Also remote debugging tool will be heplful.

@lomalkin promise to write down his vision by end of this week.

[glitchcore]

Smoke testing, I could write a python or ruby script fairly quickly to interact and parse through standard i/o for at least rudimentary smoke testing. This process could be integrated with automated CI to smoke test firmware on pull requests.

Integration/Hardware testing, I suggesting using a control board, my preferred would be Arduino, to connect to and interact with the hardware pins with the firmware in a "testing" mode so extra debug messages and user/sensor input simulations could be communicated with the control board, and through serial into a docker based testing service. Integrations between this and the front end of a hardware emulator would be awesome to get the simulated input polished.

@officialdarksheao What difference between Smoke and Integration/HW testing?

[glitchcore]

@lomalkin can we use your testing board for making test bench? Maybe you can place a test bench (RPi + Flipper + test board + optional webcam and SDR) at home or in B4CKSP4CE?

[Disasm]

For running unit tests (with Rust) in simple environments, we can also use avatar-rs. It allows to build code for x86, but execute all peripheral access operations on the device via SWD. Also FerrousSystems started developing something for unit-testing on the device itself.

[glitchcore]

avatar-rs require svd2rust?

[Disasm]

@glitchcore it uses pac crates which are built with svd2rust, but doesn't use svd2rust directly iirc

[mbelt]

Is it possible to use JTAG instead of a control board to simulate button pushes and user I/O? I'm assuming the test host already has an ICD/programmer so it can flash the test builds. That would simplify the required hardware for setting up integration test hosts.

[glitchcore]

Is it possible to use JTAG instead of a control board to simulate button pushes and user I/O?

I have no experience with using JTAG on STM32 as boundary scan and I found nothing in google about it.

[mbelt]

I have no experience with using JTAG on STM32 as boundary scan and I found nothing in google about it.

OpenOCD implements BS for STM32 and works with the cheap ST-Link debuggers. Writing to GPIO pins is possible.

OpenOCD runs as a GDB server, but it also has a socket for accepting tcl commands. I'm thinking along the lines of a program that implements the same interface as the hardware emulator and spits tcl at openocd.

The question I think is whether or not we can safely use JTAG BS without either upsetting some watchdog, throwing off timing, or upsetting OpenRTOS somehow.

If the interface for the hardware emulator is written down somewhere I could go off and experiment.

[glitchcore]

What do you mean "hardware emulator"? Now we have some stubs for handling GPIO, UART, etc. operations.

Big question — how to emulate interrupts. For example, could we change pin level from OpenOCD and trigger EXTI?

[mbelt]

What do you mean "hardware emulator"? Now we have some stubs for handling GPIO, UART, etc. operations.

"Hardware emulator" was the phrase used above, as the software sim that app developers can use if they don't have real hardware. It has some UI that sends commands (button presses) and gets back a display from some kind of re-hosted or re-targeted version of the device software.

The API that the simulator would expose to this front end UI would be very similar to the API exposed by a "device adapter" used by integration tests. If the interface were made common, then you could theoretically run and debug integration tests without hardware, then swap out the hardware emulator with this openocd bridge that would run those same tests on actual hardware.

This all kind of assumes integration tests are black-box style, where the surface area the tests have to work with is only what the user can do while operating the device. Again, based on quoted comment from @officialdarksheao above.

One more benefit of using JTAG instead of a control board is that black-box becomes gray-box because your tests can now read/write static symbols in the code running on the device, in addition to being able to push its buttons and get serial output.

Big question — how to emulate interrupts. For example, could we change pin level from OpenOCD and trigger EXTI?

Since EXTI pin is GPIO, then I think it could. It's worth trying.

[glitchcore]

you could theoretically run and debug integration tests without hardware, then swap out the hardware emulator with this openocd bridge that would run those same tests on actual hardware.

Sounds cool!

My point: using openocd instead of driving/scanning real signal is good if this method guarantee that we see and reproduce real state of device.

[glitchcore]

Today I learned https://robotframework.org/ for IoT testing, and Renode for STM emulating

[mbelt]

I'll look into ROS, but from what I remember it would be a heavy dependency.

pyOCD is almost exactly what I think we want here. The problem is that it only supports the ARM debug TAP, not the STM32 TAP which would let you observe/drive the MCU's pads without involving the CPU.

Today I'm going to figure out if it's possible to add support for additional JTAG TAPs to pyOCD.

[glitchcore]

@mbelt pay attention to my comment in emulator issue thread: #22 (comment)

[glitchcore]

I see that #26 and #22 have very close subject and maybe we should refactor this discussion

[lomalkin]

Hello, guys!

I can create by myself testing tool for remote hardware debugging with following features (so here is my suggestions):

• 24*7 connected target Flipper device and accessible via Internet. Raspberry PI or some Linux single-board PC can be used as basic high-level control board.
• Tool can push/click each user buttons by hardware by "control board" (low level). Usage of optocouples/reed-switch relays is fine for that.
• Connect other Flipper peripherals to target:
  • 433 door bell/barrier controller, to read it status and it's remote control (to sniff it signal by flipper).
  • Some iButtons and it's reader can be also connected to target.
  • RFID reader under the target, to paste readed keys from it to out UART.
  • RFID cards with different IDs. Can changed by servo or carousel from CD-changer (I have this part).
  • IR-transmitter/receiver.
  • ...all the peripherals, that we'll can realize...
• "Hardware" USB peripherals (SWD programmer tool, etc?) reconnection and control of PWR, RST, BOOTx internal service lines, etc also need. This can be made by some software hacks or by relays.
• Image from target display will be translated by webcam to web page (see next). That's simplest way for start.
• WEB page, accessed to all developers (or maybe for everyone to view is better?) with these things:
  • Target status area, Flipper control buttons, "connected" to real target. Power and service line switches.
  • Ability to upload custom firmware binary to target. Take firmware build from CI/CD. Button to flash.
  • Test tool UART with Flipper peripherals status and target device UART log also should be here.
  • List can be expanded with other good ideas.
• OpenOCD connection from target (can be accessed from developers around the world by VPN or just port forwarding with IP access lists). So this feature allows deep remote debug.
• Test tool must be as autonomous, as possible. So from WEB page we can reboot everything, if it need, without local operator help. I have some experience in that.
• Planned life time for the tool is around 6 months. It's enough?

I guess, that main control part of tool (e.g. "control board") must be independent and accessed only via UART with simple single-line protocol, and connected to target with real hardware (wires, w/o JTAG/etc. But this do not excluded JTAG as addition). That's main idea.
The tool should be build as semi-portable construction, in order to move it to host in other place. I can start to build it and host it by myself for start.

Some another abilities can be also realized:

• For test automation we can use RobotDemo (from @glitchcore suggestion) or simple expect tool/python scripts/etc.
  • Flash firmware (from CI/CD or custom)
  • Apply test cases and submit its results.
• Maybe run tests with CI/CD (I'll need help with that)
• (Development results from this stage can be easily used to build Mass-production testing tool in future)

I suggest to make functional prototype, and after it maybe we'll want to built a few upgraded tools like this.

Questions:

• Sharing control. Tool WEB page can be accessible to view for all, but ideologically should have single-access to control (modify).
  • My idea is to make some schedule page, where developers can submit request to single use for specified time. But I'm not sure, that it's good solution.

[dontsovcmc]

Remark for RFID test. We can use relay and simulate card touching. https://vostnod.livejournal.com/75405.html if needed.

We should have "clean firmware" and queue for developers tests (checks) for CI. I think "online Flipper" can be overkill feature, cause we don't know a number of active developers. Developers - first, brainstorming - next.

But building firmware and unit test are needed.

[glitchcore]

Remark for RFID test. We can use relay and simulate card touching. https://vostnod.livejournal.com/75405.html if needed.

Cool example) I'm afraid it not be applicable to NFC card due to high working frequency.
Also how to emulate multiple cards in this setup?

I think "online Flipper" can be overkill feature

It is very useful to debug code on real hardware if you have not developer kit.

[dontsovcmc]

I didn't use NFC.

The main idea not to do unuseful features. Read "Getting real" book. So, lets solve only main problems. Online Flipper can be only "hipe" feature. I dont know development shedulers... I see supercomputer queue. I think its uncompatible with your goals. The simplest way to send development Flippers or use development boards.

[glitchcore]

The simplest way to send development Flippers or use development boards.

We don't have enough kits for everyone, and new contributors may want to work right now and don't wait when devboard come to them

I see supercomputer queue

Yes it is.

I think its uncompatible with your goals.

Really? I think such remote bench will be funny.

[Mr-Dwight-Schrute]

what dev board do we need to purchase?

[glitchcore]

what dev board do we need to purchase?

To some developers we will send F2B0C1.1 board for debugging

[glitchcore]

Replaced by #96