Tests fail

[ToddG]

PART 1

Top level cargo test fails

HASH

commit ad6fedd35a9a17639edf44a59fab6a14b2707cce (HEAD -> master, origin/master, origin/HEAD)
Author: Rahix <rahix@rahix.de>
Date:   Mon Sep 7 23:06:02 2020 +0200

    generic: delay: Fix a compiler warning
    
    Remove unnecessary parentheses.
    
    Signed-off-by: Rahix <rahix@rahix.de>

REPRO

git clone https://github.com/Rahix/avr-hal
cd avr-hal
rustup override set nightly
cargo build
cargo test

ERROR

   Compiling arduino-uno v0.1.0 (/home/todd/temp/foo/avr-hal/boards/arduino-uno)
   Compiling arduino-mega2560 v0.1.0 (/home/todd/temp/foo/avr-hal/boards/arduino-mega2560)
   Compiling sparkfun-pro-micro v0.1.0 (/home/todd/temp/foo/avr-hal/boards/sparkfun-pro-micro)
   Compiling arduino-leonardo v0.1.0 (/home/todd/temp/foo/avr-hal/boards/arduino-leonardo)
error: Ensure that you are using an AVR target! You may need to change directories or pass a --target flag to cargo. See
       https://github.com/Rahix/avr-device/pull/41 for more details.
  --> boards/arduino-uno/examples/uno-panic.rs:27:1
   |
27 | #[arduino_uno::entry]
   | ^^^^^^^^^^^^^^^^^^^^^
   |
   = note: this error originates in an attribute macro (in Nightly builds, run with -Z macro-backtrace for more info)

EXPECTED

I suspect this is an issue with the use of workspaces. As shown in PART 2 below, running cargo test within one of the crates fails for a different reason there.

Full file attached.
cargo-test-out.txt

QUESTION

Why aren't the tests being run as part of the travis build? I only see cargo build being run.

PART 2

Device cargo test fails, too, but with different errors.

REPRO

cd boards/arduino-uno
cargo build (succeeds)
cargo clean
cargo test

ERROR

   Compiling nb v1.0.0
   Compiling void v1.0.2
   Compiling ufmt-write v0.1.0 (https://github.com/Rahix/ufmt.git?rev=12225dc1678e42fecb0e8635bf80f501e24817d9#12225dc1)
   Compiling cfg-if v0.1.10
   Compiling vcell v0.1.2
   Compiling panic-halt v0.2.0
   Compiling bare-metal v0.2.5
error: duplicate lang item in crate `core` (which `rustc_std_workspace_core` depends on): `bool`.
  |
  = note: the lang item is first defined in crate `core` (which `vcell` depends on)
  = note: first definition in `core` loaded from /home/todd/temp/foo/avr-hal/target/avr-atmega328p/debug/deps/libcore-77c7fc359a8bad07.rmeta
  = note: second definition in `core` loaded from /home/todd/temp/foo/avr-hal/target/avr-atmega328p/debug/deps/libcore-6d69c1e64d8513e0.rmeta
...

EXPECTED

Why don't the tests pass? Why is there this issue with duplicate definitions? Is this the real problem or is this masking some other, known, issue?

[ToddG]

Posted a related bug/question on SO here:

https://stackoverflow.com/questions/63961435/rust-cargo-test-fails-for-arduino-targets-with-duplicate-lang-item-in-crate

[Rahix]

There's a few things here ...

First of all, what do you expect cargo test to do? The crates in this repo target a microcontroller, it's hard to test those without actual hardware ;) So it doesn't really make sense to run cargo test, there isn't much it could test and no way to run AVR machine code on your computer without an emulator anyway ...

If you take a look at the code base, there aren't any #[test]s defined anywhere to run. The tests are (more or less) the examples for each board as that's the best thing we have to test functionality at this point.

In general, cargo test doesn't support testing on embedded targets yet. There are frameworks like utest which try to make it work but that's all really experimental at this point.

That said, regarding the errors you've seen:

error: Ensure that you are using an AVR target! You may need to change directories or pass a --target flag to cargo. See [...]

Well, the message says it all: You didn't specify an AVR target with --target so rust tried to build avr-device for your computer which doesn't make sense - hence the error. When building inside a board directory, the .cargo/config.toml file specifies the target for you; that's why you didn't get the above error there ...

error: duplicate lang item in crate `core` (which `rustc_std_workspace_core` depends on): `bool`.

I don't know for certain, but I would assume this error is a symptom of the facts that A) cargo test does not support bare-metal targets and B) std aware cargo (i.e. building custom libcore) is very experimental and not yet fully implemented.

[ToddG]

Thank you for the detailed message. This is my first brush with rust, so here we go. I'm using avr-hal as a base case for understanding how the cargo commands should work.

I was expecting that since workspace commands at the top level such as cargo clean and cargo build work by iterating over the member crates, then cargo test would, too. With no #[test] blocks, I'd expect the workspace command to iterate through the member crates and report back zero tests. At any rate, I would not expect to get a bunch of build errors.

I think the AVR target message is a red herring. The top level cargo commands invoke cargo commands in the child crates, and those crates have the necessary .cargo/config.toml files.

Q: How do people test embedded targets like the AVR in rust? After spending a full day trying to get cargo test running (nowhere in the documentation have I seen mention that this is unsupported)...I've decided to just write my code and tests in a standard workspace, and then copy and paste the code (without the tests) into the .rs files within a crate that targets the AVR.

Thanks again.

[Rahix]

I'm using avr-hal as a base case for understanding how the cargo commands should work.

Hm, you might want to look at something else first, avr-hal and similar projects are quite special and probably not the best way to understand cargo ... Especially avr-hal because it uses unstable cargo features.

I was expecting that since workspace commands at the top level such as cargo clean and cargo build work by iterating over the member crates, then cargo test would, too. With no #[test] blocks, I'd expect the workspace command to iterate through the member crates and report back zero tests. At any rate, I would not expect to get a bunch of build errors.

The build errors were because it is fundamentally impossible to run AVR machine code on your computer and because cargo test does not support bare-metal targets. I'd say for avr-hal there is not other sensible outcome from running cargo test than a build error; at least at the moment ...

I think the AVR target message is a red herring. The top level cargo commands invoke cargo commands in the child crates, and those crates have the necessary .cargo/config.toml files.

No, .cargo/config.toml is only pulled in from the directory you're running the command from (and any directory above). So cargo ... in the top level directory will not have any target or other options set. That's the reason the cargo config is separate from Cargo.toml, because it contains the parts that should not be crate specific.

Q: How do people test embedded targets like the AVR in rust? After spending a full day trying to get cargo test running (nowhere in the documentation have I seen mention that this is unsupported)...I've decided to just write my code and tests in a standard workspace, and then copy and paste the code (without the tests) into the .rs files within a crate that targets the AVR.

It really depends on what you're trying to test: For unit-testing isolated, machine-independent, parts people usually put those in a separate library crate that they can then run tests on on their development machine. For anything machine specific, things you can only really test on the target device, it's really project-specific which methods make sense: You could add a feature-flag to include selftest routines in the firmware build, have a second binary crate that calls into the machine-specific functionality to test it, try using crates like utest as I mentioned before, etc. There isn't really a canonical way and, to be quite honest, automated testing is not (yet) that far spread in the embedded (not just Rust) community in general ...

[Rahix]

Is this issue resolved for you now?

[ToddG]

Yes, this is resolved. Thank you.

…

-------- Original Message --------

On Oct 2, 2020, 2:45 AM, Rahix wrote: Is this issue resolved for you now? — You are receiving this because you authored the thread. Reply to this email directly, [view it on GitHub](#71 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAAZQYNRYIDJWNZZ44LHXE3SIWOMTANCNFSM4RSFWICQ).

[koutoftimer]

@Rahix

The crates in this repo target a microcontroller, it's hard to test those without actual hardware ;) So it doesn't really make sense to run cargo test, there isn't much it could test and no way to run AVR machine code on your computer without an emulator anyway ...

I guess, this is not quiet true anymore.

μtest's README says:

For a stable approach you can use a custom Cargo runner and a procedural macro as it's done in the defmt-test crate.

If you'd like to know more about testing embedded Rust firmware in general I recommend this series of blog posts.

Looks like things have been changed a bit.

Also, looks like QEMU can emulate avr, which can be used to perform testing

[Rahix]

Yes, testing in an emulator is possible (although I'd look at simavr more than QEMU) and you can build a no_std test-harness similar to defmt-test. But I think there is not much to gain here: avr-hal concerns itself mostly with hardware peripheral drivers. To test those properly, the emulator would need to have full support for the peripherals in question and this support must work exactly like the real hardware. Anything less and the tests are mostly useless because they don't tell you anything about whether the code works on real hardware.

As an example: simavr doesn't care whether you set the correct baudrate configuration for UART, it will happily output data even if our baudrate calculation logic is completely wrong. In contrast, with a HIL (hardware in the loop) testing setup, such cases are easily catched because the receiver won't decode UART data correctly if we miscalculate the baudrate register values...

Similarly, I2C pull-up resistor configuration is probably also not checked by an emulator (because it can't know whether external pull-ups exist) but it is integral that avr-hal configures them correctly.

So all in all, I think the effort of setting up emulator testing is not worth it when considering how little of the possible kinds of bugs it can catch. I'd rather build a HIL testsetup, maybe not as part of CI but as something to manually run before releasing new versions.

However that's going to be quite a task: We need to cover as many supported devices as possible and for each one, test as many peripherals as possible. For each of those, we need an "other end" which is also controlled by the test-runner to verify the interactions initiated by the DUT. If you know of any existing (AVR) setups like this, do let me know - I'd rather not reinvent the wheel here...