Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

workaround esp-idf version detection #39

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

madmo
Copy link

@madmo madmo commented Feb 20, 2024

If version.txt is missing some part of the cmake toolchain tries to read refs/heads/fetchgit in the idf repo which failed.

This is not a perfect solution, but works nicely when using cargo, and fixes some of the issues in #5

@@ -1,5 +1,5 @@
{ rev ? "v5.1.2"
, sha256 ? "sha256-uEf3/3NPH+E39VgQ02AbxTG7nmG5bQlhwk/WcTeAUfg="
, sha256 ? "sha256-qjZ438LUQLzyQDKkeo4WMO0jxfkFh6dReuLq/6rftpw="

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change intentional? I'm trying to test using your fork but this appears to be breaking it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, it was working when I made the change. If I rebuild now it also stopped working for me. I will update the hash, and we will see if this fixes the issue for now.

@madmo madmo force-pushed the master branch 2 times, most recently from aa89fb4 to b96f3c2 Compare March 6, 2024 15:32
@madmo madmo marked this pull request as draft March 6, 2024 16:19
@madmo
Copy link
Author

madmo commented Mar 6, 2024

Ok, back to the drawing board. The issue is that fetchgit with leaveDotGit is not always reproducible: NixOS/nixpkgs#8567.

git init .
git config user.email "nixbld@localhost"
git config user.name "nixbld"
git commit --date="1970-01-01 00:00:00" --allow-empty -m "make idf happy"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems quite clever. Is it working on your end?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting this error now:

To add an exception for this directory, call:

        git config --global --add safe.directory /nix/store/mfmmywqch5m4bshhmh5lb01np9yjdlgk-esp-idf-v5.1.2```

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried fiddling with this for a few hours but didn't crack it. You can see the things I tried here:

https://github.com/jonahbron/nixpkgs-esp-dev/commits/madmo/?author=jonahbron

Hoping you have more luck!

@jonahbron
Copy link

Seeing other comments around this affecting non-ESP related stuff, might have some useful resources

NixOS/nixpkgs#169193

@madmo
Copy link
Author

madmo commented Mar 22, 2024

For me it is working now. Had a issue with the installPhase script writing to the source dir, which in turn caused some hash issue.

@madmo madmo marked this pull request as ready for review March 22, 2024 08:37
@jonahbron
Copy link

jonahbron commented Mar 23, 2024

I can now get into a shell with nix develop with your fork. However, I'm still having trouble pulling it into another flake for building.

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    nixpkgs-esp-dev.url = "github:madmo/nixpkgs-esp-dev";
  };

  outputs = { self, nixpkgs, nixpkgs-esp-dev }: let
    forAllSystems = function:
      nixpkgs.lib.genAttrs [
        "x86_64-linux"
      ] (system: function (import nixpkgs {inherit system;}));
  in {
    packages = forAllSystems (pkgs: let
      berkeley-db-1_xx = pkgs.fetchFromGitHub {
        owner = "pfalcon";
        repo = "berkeley-db-1.xx";
        rev = "35aaec4418ad78628a3b935885dd189d41ce779b";
        sha256 = "sha256-XItxmpXXPgv11LcnL7dty6uq1JctGokHCU8UGG9ic04=";
      };
      micropython-src = pkgs.fetchFromGitHub {
        owner = "micropython";
        repo = "micropython";
        rev = "v1.22.2";
        sha256 = "sha256-67CAR34VrMOzvNkukDeGRnUfoOLO66R37wsrRHjpp5E=";
      };
    in {
      esp32genericc3 = pkgs.stdenv.mkDerivation {
        name = "esp32genericc3";
        nativeBuildInputs = [
          pkgs.git
          nixpkgs-esp-dev.packages.${pkgs.system}.esp-idf-esp32c3
        ];

        src = micropython-src;
        phases = ["unpackPhase" "patchPhase" "buildPhase"];

        patchPhase = ''
          rmdir lib/berkeley-db-1.xx
          ln -s ${berkeley-db-1_xx} lib/berkeley-db-1.xx
        '';
        
        buildPhase = ''
          make V=1 -C mpy-cross
          make V=1 -C ports/esp32 BOARD=ESP32_GENERIC_C3
        '';
      };
    });
  };
}

This is incomplete but I can't even get past the buildPhase before it hits the Git error

...
make: Leaving directory '/build/source/mpy-cross'
make: Entering directory '/build/source/ports/esp32'
idf.py -D MICROPY_BOARD=ESP32_GENERIC_C3 -D MICROPY_BOARD_DIR=/build/source/ports/esp32/boards/ESP32_GENERIC_C3  -B build-ESP32_GENERIC_C3 build || (echo -e "See \033[1;31mhttps://github.com/micropython/micropython/wiki/Build-Troubleshooting\033[0>
Executing action: all (aliases: build)
Running cmake in directory /build/source/ports/esp32/build-ESP32_GENERIC_C3
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DPYTHON=/nix/store/41253d4jjd23jjdaiidgnb8q2qambh1a-python3-3.11.7-env/bin/python3.11 -DESP_PLATFORM=1 -DMICROPY_BOARD=ESP32_GENERIC_C3 -DMICROPY_BOARD_DIR=/build/source/ports/esp32/boards/ESP32_G>
-- Found Git: /nix/store/bd0dcc9qzpfz5xfz7fm6cwf7q64cyfhi-git-2.44.0/bin/git (found version "2.44.0")
-- git rev-parse returned 'fatal: detected dubious ownership in repository at '/nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2'

Any idea why the issue is persisting when run in a derivation?

@madmo
Copy link
Author

madmo commented Mar 23, 2024

@jonahbron This is related to the safe.directory setting of git. I completely disabled this security feature on my machine.

@jonahbron
Copy link

After hours of fiddling I got it to almost work. But now, it looks like the compiler is trying to download files over the network!

Dependencies lock doesn't exist, solving dependencies.
CMake Error at /nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2/tools/cmake/build.cmake:540 (message):
  ERROR: Solver failed processing dependency "espressif/mdns" from the
  manifest file "/build/source/ports/esp32/main_esp32c3/idf_component.yml".

  HTTPSConnectionPool(host='components-file.espressif.com', port=443): Max
  retries exceeded with url: /components/espressif/mdns.json (Caused by
  NameResolutionError("<urllib3.connection.HTTPSConnection object at
  0x7ffff5c2b190>: Failed to resolve 'components-file.espressif.com' ([Errno
  -3] Temporary failure in name resolution)"))

Call Stack (most recent call first):
  /nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2/tools/cmake/project.cmake:547 (idf_build_process)
  CMakeLists.txt:53 (project)


-- Configuring incomplete, errors occurred!
cmake failed with exit code 1, output of the command is in the /build/source/ports/esp32/build-ESP32_GENERIC_C3/log/idf_py_stderr_output_1601 and /build/source/ports/esp32/build-ESP32_GENERIC_C3/log/idf_py>
See https://github.com/micropython/micropython/wiki/Build-Troubleshooting
make: *** [Makefile:66: all] Error 1
make: Leaving directory '/build/source/ports/esp32'
CMake Error at /nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2/tools/cmake/build.cmake:540 (message):
  ERROR: Solver failed processing dependency "espressif/mdns" from the
  manifest file "/build/source/ports/esp32/main_esp32c3/idf_component.yml".

  HTTPSConnectionPool(host='components-file.espressif.com', port=443): Max
  retries exceeded with url: /components/espressif/mdns.json (Caused by
  NameResolutionError("<urllib3.connection.HTTPSConnection object at
  0x7ffff5c2b190>: Failed to resolve 'components-file.espressif.com' ([Errno
  -3] Temporary failure in name resolution)"))

Call Stack (most recent call first):
  /nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2/tools/cmake/project.cmake:547 (idf_build_process)
  CMakeLists.txt:53 (project)

This might be make building purely in a flake a no-go 😢

@jonahbron
Copy link

jonahbron commented Mar 23, 2024

It's not my favorite, but the solution (plus a bunch of other little fixes) was to make it into a known-output derivation; adding a outputHash prop etc. so that the derivation could make the network calls to Espressif for pulling in dependency components.

Here is my flake for anyone interested in how it works.

https://git.sr.ht/~jonahbron/doorbell/tree/963d4cc33cbba10d555237a4b4e2ef8df6f24776/item/mpy-flake/flake.nix

Take specific note of the workaround I applied to the Git problem, specifically marking only the correct directory safe, and using the GIT_CONFIG_GLOBAL env var to apply that config.

I'd love to see this PR submitted, it's helping me immensely!

@jonahbron
Copy link

Settled onto what I think is quite a nice Nix expression for this purpose.

https://git.sr.ht/~jonahbron/doorbell/tree/83e24a1477a65b0d3902d2492baab623cfc7baed/item/firmware.nix

Hopefully this can be a useful reference so nobody has to burn as much time figuring this out as I have 😄

@jonahbron
Copy link

Actually, because I necessarily have to have the Git fix in my project too, this PR has no efffect. Removing it my build still works.

https://git.sr.ht/~jonahbron/doorbell/tree/701ca6b16861b7183e72719be843a34308d2cc5b/item/flake.nix

@madmo At this point the issue might only be coming down to the "dubious ownership" issue, which I've solved with this code:

https://git.sr.ht/~jonahbron/doorbell/tree/701ca6b16861b7183e72719be843a34308d2cc5b/item/firmware.nix#L133

If version.txt is missing some part of the cmake toolchain tries to read
refs/heads/fetchgit in the idf repo which failed.

This is not a perfect solution, but works nicely when using cargo.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants