arm-embedded GCC breaks Zephyr RTOS on nRF51 (Cortex-M0)

[lopsided98]

Issue description

pkgsCross.arm-embedded.buildPackages.gcc9 causes Zephyr RTOS to immediately crash on boot for all non-trivial applications. This also occurs with GCC 7, but I did most of my testing with GCC 9 for easier comparison with Arch Linux's arm-none-eabi-gcc package, which has GCC 9.1 and works correctly.

I have reproduced this on the Nordic nRF51822, which has an ARM Cortex-M0 CPU. I have no other hardware available to test.

The output of diffoscope --tool-prefix-binutils arm-none-eabi- zephyr.nix.strip.elf zephyr.arch.strip.elf can be seen here.

Most of the differences seem to be caused by different offsets (which is unexpected, but doesn't seem to be the root cause of the problem), but I did notice this:

│  Hex dump of section '.ARM.attributes':
│    0x00000000 412e0000 00616561 62690001 24000000 A....aeabi..$...
│ -  0x00000010 0541524d 2076364b 00060907 4d080109 .ARM v6K....M...
│ +  0x00000010 05436f72 7465782d 4d300006 0c074d09 .Cortex-M0....M.
│    0x00000020 01120414 01150117 0318011a 011e04   ...............

This makes me suspect that Nix's GCC is getting confused about the architecture and potentially generating code for the wrong CPU.

Steps to reproduce

Create a Zephyr application for the nRF51 that calls k_sleep in an infinite loop. Compile in a nix-shell environment as shown below. Flash the result to the device and observe that it immediately crashes with a hard fault.

shell.nix

let
  pkgs = import <nixpkgs> { overlays = [ (import ../nix) ]; };
in

with pkgs;

let 
  gcc = pkgsCross.arm-embedded.buildPackages.gcc9;
  binutils = pkgsCross.arm-embedded.buildPackages.binutils;
  toolchain = buildEnv {
    name = "arm-toolchain";
    paths = [ gcc binutils ];
  };
in mkShell {
  name = "water-level-monitor-env";
  buildInputs = with python3Packages; [
    which
    git
    gcc
    binutils
    cmake
    dtc
    gperf
    west
    wheel
    breathe
    sphinx
    docutils
    sphinx_rtd_theme
    sphinxcontrib-svg2pdfconverter
    junit2html
    pyyaml
    ply
    gitlint
    pyelftools
    pyocd
    pyserial
    pykwalify
    colorama
    pillow
    intelhex
    pytest
    gcovr
  ];
  
  GNUARMEMB_TOOLCHAIN_PATH = toolchain;
}

This environment is what I use for development, but it cannot be easily reproduced because many of the packages are not in upstream nixpkgs. I can publish these packages somewhere if someone wants to reproduce this environment.

Technical details

• system: "x86_64-linux"
• host os: Linux 5.1.7-arch1-1-ARCH, Arch Linux, noversion
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.2.2
• channels(root): "nixpkgs"
• nixpkgs: /home/ben/Documents/Projects/nixpkgs

[matthewbauer]

You should make sure the right gcc & binutils are being called. mkShell awkwardly has its own ld and cc. This can be done by putting them in "nativeBuildInputs". Note sure if this is causing the issue, just one thing to make sure is happening correctly. strip can be especially bad at breaking these things.

[lopsided98]

Yes, its definitely calling the right toolchain. Zephyr uses GNUARMEMB_TOOLCHAIN_PATH to find GCC and binutils. I have looked at make's verbose output to confirm this. I also think the build would fail quickly if it tried to use an x86_64 toolchain.

[pkral78]

It looks like intermixing both toolchain paths, what is type of the crash, UsageFault UNDEFINSTR ? How about use gcc-arm-embedded instead of pkgsCross.arm-embedded.buildPackages.gcc ?

Reference #51907

Also I'm also thinking about usage of Zephyr SDK directly (wrapped with build FHS) as it is used for Shippable CI in Zephyr PR testing.

[stale]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[lopsided98]

I'm pretty sure I've discovered the cause of this problem, although now I'm working on building ArduPilot with a fully Nix built toolchain, and I haven't gone back and tested my Zephyr project.

The key difference between pkgsCross.arm-embedded and gcc-arm-embedded (and Arch's arm-none-eabi-gcc package, which is basically equivalent) is that the latter includes multilib versions of all the target libraries (e.g. newlib, libgcc, libstdc++). These libraries are built for a variety of -march targets, and the compiler links against the appropriate version. The full list of variants and corresponding -march flags can be found here: https://github.com/gcc-mirror/gcc/blob/1fab05a885a308c19cf42b72fd36805ddf27fdc8/gcc/config/arm/t-rmprofile#L30-L31

With pkgsCross.arm-embedded, no -march flag is passed, so you get target libraries for whatever the default gcc architecture is. In this case the default appears to be -march=armv4t, which is not going to work on your standard Cortex-M based microcontroller.

The fix is to manually specify the proper -march and other flags in the Nix platform config. Unfortunately, this currently results in rebuilding two copies of gcc (stage-static and stage-final) and also means that pkgsCross.arm-embedded is basically useless. #132340 is fixed, this solution should no longer involve rebuilding gcc.

For example, to build ArduPilot for Cortex-M4 and M7 processors, I use the following platform, which is equivalent to the thumb/v7e-m+fp/hard multilib variant:

{
  config = "arm-none-eabi";
  libc = "newlib-nano";
  gcc = {
    arch = "armv7e-m+fp";
    float-abi = "hard";
  };
}

Additionally, newlib-nano is built with -O2 instead of -Os as in gcc-arm-embedded. I've worked around this using withCFlags [ "-Os" ], but this should probably be fixed in the package to match other distributions.

These limitations are the reason there are currently only two usages of pkgsCross.arm-embedded in nixpkgs:

• fusee-launcher: works because it targets a Cortex-A processor compatible with -march=armv4t and also doesn't use any of the standard libraries.
• arm-trusted-firmware: built with -nostdlib