kernels fail to build on rpi5

[Daholli]

When trying to upgrade my system, non of the realeases, nor the master branch let me build/grab the kernel from cachix. then when trying to build the kernel, which has worked previously on a debian rpi it just fails.

here is the full nix log
build.log

I really don't know how to proceed from here

┏━ 2 Errors: 
 ⋮ 
┃        > QUESTION: Runtime Testing, NAME: RUNTIME_TESTING_MENU, ALTS: N/y/?, ANSWER: n
┃        > GOT: n
┃        > QUESTION: Memtest, NAME: MEMTEST, ALTS: Y/n/?, ANSWER: y
┃        > GOT: y
┃        > GOT: *
┃        > GOT: * Rust hacking
┃        > GOT: *
┃        > GOT: #
┃        > GOT: # configuration written to .config
┃        > GOT: #
┃        > GOT: make[1]: Leaving directory '/build/source/build'
┃        > GOT: make: Leaving directory '/build/source'
┃        > warning: unused option: ACPI_APEI
┃        > warning: unused option: ACPI_APEI_GHES
┃        > warning: unused option: ACPI_FPDT
┃        > warning: unused option: ACPI_HMAT
┃        > error: unused option: ACPI_HOTPLUG_CPU
┃        > error: unused option: ACPI_HOTPLUG_MEMORY
┃        > error: unused option: CGROUP_HUGETLB
┃        > error: unused option: CHROMEOS_TBMC
┃        > warning: unused option: DRM_VBOXVIDEO
┃        > error: unused option: FSL_MC_UAPI_SUPPORT
┃        > error: unused option: HOTPLUG_PCI_ACPI
┃        > error: unused option: HOTPLUG_PCI_PCIE
┃        > warning: unused option: KEXEC_JUMP
┃        > error: unused option: MOUSE_ELAN_I2C_SMBUS
┃        > error: unused option: MOUSE_PS2_ELANTECH
┃        > error: unused option: MT798X_WMAC
┃        > error: unused option: NET_VENDOR_MEDIATEK
┃        > warning: unused option: PARAVIRT_SPINLOCKS
┃        > error: unused option: PCI_TEGRA
┃        > warning: unused option: PCI_XEN
┃        > warning: unused option: PERF_EVENTS_AMD_BRS
┃        > error: unused option: PINCTRL_AMD
┃        > error: unused option: PM_WAKELOCKS
┃        > error: unused option: SCHED_CORE
┃        > error: unused option: SUN8I_DE2_CCU
┃        > error: unused option: USB_XHCI_TEGRA
┃        > warning: unused option: VBOXGUEST
┃        > warning: unused option: XEN_HAVE_PVMMU
┃        > warning: unused option: XEN_MCE_LOG
┃        > warning: unused option: XEN_PVH
┃        > warning: unused option: XEN_PVHVM
┃        > warning: unused option: XEN_SAVE_RESTORE
┃        For full logs, run 'nix log /nix/store/vif15wghxn3l0kf3iapmrkrhykdcyf5y-linux-config-6.6.51.drv'.
┣━ Dependency Graph:
┃ ┌─ ⚠ linux-config-6.6.51-aarch64-linux failed with exit code 1 after ⏱ 1m1s in buildPhase
┃ ⏸ linux-rpi-6.6.51-aarch64-linux

[Daholli]

ok I managed to at least pinpoint it a bit more, it fails

when i change the flake input to unstable
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
(this by itself works)
and
raspberry-pi-nix.board = "bcm2712";

[osbm]

Yes i can confirm my rpi kernel also fails to build

[Daholli]

I was git bisecting nixpkgs for a bit yesterday, but some cache misses cost a lot of time

[Daholli]

git bisect start
status: waiting for both good and bad commits
❯ git bisect good cbd8ec4de4469333c82ff40d057350c30e9f7d36
status: waiting for bad commit, 1 good commit known
❯ git bisect bad 852ff1d9e153d8875a83602e03fdef8a63f0ecf8
Bisecting: a merge base must be tested
[1d95cb5fa7a38717172e090d54338ca8daef8a86] nixos/duckdns: init module (#294489)
❯ git bisect good
Bisecting: 17907 revisions left to test after this (roughly 14 steps)
[6c96a50b6046f921bcd52a4c25d1fad972d74b0e] libvlcpp: init 0.1.0-unstable-2024-02-04 (#348562)
❯ git bisect good
Bisecting: 8961 revisions left to test after this (roughly 13 steps)
[bd5fadf56b280e26429f2c0b06fd136cb862ba3f] home-assistant-custom-components.xiaomi_miot: 1.0.7 -> 1.0.8 (#371894)
❯  git bisect good
Bisecting: 4483 revisions left to test after this (roughly 12 steps)
[cf7c9d7ff0d814c1da5bced71d21e03b40a91183] obs-studio-plugins.obs-shaderfilter: 2.3.2 -> 2.4.0 (#374425)
❯ git bisect good
Bisecting: 2242 revisions left to test after this (roughly 11 steps)
[0359308417056722c885bffd29523d7b3dd2a734] terraform-providers.huaweicloud: 1.72.0 -> 1.72.1 (#375240)
❯ git bisect bad
Bisecting: 1129 revisions left to test after this (roughly 10 steps)
[51e4b3bf169e8f63179eaaa064d835da9fd6fe0b] python313Packages.icalevents: 0.1.29 -> 0.2.0
❯ git bisect bad
Bisecting: 555 revisions left to test after this (roughly 9 steps)
[aa5f70e60572c2ecde8030b45599080177004d7d] libdeflate: enable tests (#368784)
❯ git bisect bad
Bisecting: 276 revisions left to test after this (roughly 8 steps)
[e2c01893f18b0a46730a6681517ceddff36fd2d7] libcap: ship the optional 'captree' component (#332399)

this is how far i got so far, but there are qutie a few cache misses after this and the raspberry takes a while to compile all that, if someone has a stronger aarch64 machine some help would be greatly appreciated
I am just letting it build until

this one succedes (or not)

[JamieMagee]

I'm hitting the same issue.

the raspberry takes a while to compile all that, if someone has a stronger aarch64 machine some help would be greatly appreciated

It might be possible to run it on GitHub actions? Use runs-on: ubuntu-24.04-arm. Check out https://github.blog/changelog/2025-01-16-linux-arm64-hosted-runners-now-available-for-free-in-public-repositories-public-preview/

[Daholli]

I guess that could work, but right now I am just letting it run over night. Got myself a pretty nasty influenca, if you wanna continue with the git bisect this is how far i got so far

❯ git bisect start
status: waiting for both good and bad commits
❯ git bisect good cbd8ec4de4469333c82ff40d057350c30e9f7d36
status: waiting for bad commit, 1 good commit known
❯ git bisect bad 852ff1d9e153d8875a83602e03fdef8a63f0ecf8
Bisecting: a merge base must be tested
[1d95cb5fa7a38717172e090d54338ca8daef8a86] nixos/duckdns: init module (#294489)
❯ git bisect good
Bisecting: 17907 revisions left to test after this (roughly 14 steps)
[6c96a50b6046f921bcd52a4c25d1fad972d74b0e] libvlcpp: init 0.1.0-unstable-2024-02-04 (#348562)
❯ git bisect good
Bisecting: 8961 revisions left to test after this (roughly 13 steps)
[bd5fadf56b280e26429f2c0b06fd136cb862ba3f] home-assistant-custom-components.xiaomi_miot: 1.0.7 -> 1.0.8 (#371894)
❯  git bisect good
Bisecting: 4483 revisions left to test after this (roughly 12 steps)
[cf7c9d7ff0d814c1da5bced71d21e03b40a91183] obs-studio-plugins.obs-shaderfilter: 2.3.2 -> 2.4.0 (#374425)
❯ git bisect good
Bisecting: 2242 revisions left to test after this (roughly 11 steps)
[0359308417056722c885bffd29523d7b3dd2a734] terraform-providers.huaweicloud: 1.72.0 -> 1.72.1 (#375240)
❯ git bisect bad
Bisecting: 1129 revisions left to test after this (roughly 10 steps)
[51e4b3bf169e8f63179eaaa064d835da9fd6fe0b] python313Packages.icalevents: 0.1.29 -> 0.2.0
❯ git bisect bad
Bisecting: 555 revisions left to test after this (roughly 9 steps)
[aa5f70e60572c2ecde8030b45599080177004d7d] libdeflate: enable tests (#368784)
❯ git bisect bad
Bisecting: 276 revisions left to test after this (roughly 8 steps)
[e2c01893f18b0a46730a6681517ceddff36fd2d7] libcap: ship the optional 'captree' component (#332399)
❯ git bisect good
Bisecting: 131 revisions left to test after this (roughly 7 steps)
[4c523a595a70d76e18037fb9243563b0f4f6b509] haskellPackages: update stackage and hackage (#354270)

quickly looking over some of the revisions that are left, there were some changes regarding arm kernel builds that were changed. I presume it is one of those that ended up breaking it

[JamieMagee]

This is the commit that broke it: NixOS/nixpkgs@37789d9

The workaround is to add ignoreConfigErrors = true when building the kernel. The fix is to use a correct kernel config for the kernel we're building. There's more detail in the nixpkgs PR NixOS/nixpkgs#366004

I'll create a PR with the workaround for now.

[tstat]

I think this issue doesn't really belong here. The kernels successfully build with the nixpkgs pinned by this repo (as guaranteed by CI) and if you attempt to build that exact kernel then a cached copy is available in the community cache. Surely it is not a goal of this repo to successfully build the kernel with any version of nixpkgs that the user might be using.

So, the challenge is ensuring you are trying to build the same kernel derivation, which means you must supply all the same inputs. There is an option to pin the inputs to match this repo.

I have had a cache miss before, and when I looked at my lock file I saw that the (transitively introduced) kernel dependency didn't match the lock file in this repo. I don't really know how this can happen, and suspect it is some sort of nix bug. I expect my transitive dependencies to be pinned according to the lockfile that introduced the transitive dependency.

In any case, using nix flake update to recreate the lockfile then results in the pinned kernel in my local lockfile aligning with the pinned kernel in the repo's lockfile.

So, I would only consider a cache miss or kernel compilation failure to be a bug in this repo if it is the same derivation as a kernel built by this repo.

[JamieMagee]

I think it's fair to say that users shouldn't be overriding the nixpkgs pinning, but we're going to hit the same issue when this repository's pin of nixpkgs is updated to 25.05. The issue will need to be fixed at some point.