Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ccache doesn't work for linux kernel #153343

Open
WesleyAC opened this issue Jan 3, 2022 · 10 comments
Open

ccache doesn't work for linux kernel #153343

WesleyAC opened this issue Jan 3, 2022 · 10 comments

Comments

@WesleyAC
Copy link

WesleyAC commented Jan 3, 2022

Describe the bug

I am trying to use ccache to compile the linux kernel with the following config:

  programs.ccache.enable = true;
  programs.ccache.packageNames = [ "linuxPackages_latest" ];
  nix.sandboxPaths = [ config.programs.ccache.cacheDir ];
  boot.kernelPackages = pkgs.linuxPackages_latest;

But I get the following error:

error: attribute 'override' missing, at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/programs/ccache.nix:55:54

Steps To Reproduce

  1. Use the four lines above in /etc/nixos/configuration.nix
  2. Run nixos-rebuild switch

Expected behavior

Kernel is built using ccache.

Additional context

#62121 may be related?

Notify maintainers

Not sure who this belongs to :(

Metadata

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.81, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.16`
 - channels(root): `"nixos-21.11.334247.573095944e7"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`
@aviallon
Copy link
Contributor

Easy workaround:

nix.sandboxPaths = [ (toString config.programs.ccache.cacheDir) ];

@WesleyAC
Copy link
Author

@aviallon do you have more info on that? I still get the error error: attribute 'override' missing, at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/programs/ccache.nix:55:54 when I try it.

@milahu
Copy link
Contributor

milahu commented Jan 19, 2022

context of ccache.nix:55

    # target configuration
    (mkIf (cfg.packageNames != []) {
      nixpkgs.overlays = [
        (self: super: genAttrs cfg.packageNames (pn: super.${pn}.override { stdenv = builtins.trace "with ccache: ${pn}" self.ccacheStdenv; }))

so the nixpkgs.linuxPackages_latest.override attribute is missing

but nixpkgs.linuxPackages_latest.kernel.override exists:

$ nix repl '<nixpkgs>'
nix-repl> pkgs.linuxPackages_latest.kernel.override

probably this will not work, cos not implemented

programs.ccache.packageNames = [ "linuxPackages_latest.kernel" ];

@WesleyAC
Copy link
Author

I tried setting programs.ccache.packageNames = [ "linuxPackages_latest.kernel" ]; — that compiles, but unfortunately does not seem to actually cache the build artifacts :(

@milahu
Copy link
Contributor

milahu commented Jan 20, 2022

{
      nixpkgs.overlays = [
        (self: super: {
          linuxPackages_latest.kernel = super.linuxPackages_latest.kernel.override {
            stdenv = self.ccacheStdenv;
          };
        })
      ];

?

you probably also need

nix.sandboxPaths = [ (toString config.programs.ccache.cacheDir) ];

then you can

sudo watch ccache --dir /nix/var/cache/ccache --show-stats

(not sure about the cache path)

@WesleyAC
Copy link
Author

@milahu Thanks so much for the help! I just tried that, and unfortunately it doesn't seem to work — I get:

error: attribute 'extend' missing, at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/kernel.nix:40:31

--show-trace gives me:

error: while evaluating the attribute 'config.system.build.toplevel' at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix:302:5:
while evaluating 'foldr' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:52:20, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/activation/top-level.nix:133:12:
while evaluating 'fold'' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:55:15, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/lists.nix:59:8:
while evaluating the attribute 'assertions' at undefined position:
while evaluating 'g' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:301:19, called from undefined position:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:171:72, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:304:20:
while evaluating the attribute 'value' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:585:9:
while evaluating the option `assertions':
while evaluating the attribute 'mergedValue' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:617:5:
while evaluating the attribute 'values' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:611:9:
while evaluating the attribute 'values' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:710:7:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:597:28, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:597:17:
while evaluating definitions from `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/kernel.nix':
while evaluating 'dischargeProperties' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:669:25, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:598:137:
while evaluating 'dischargeProperties' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:669:25, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:675:11:
while evaluating the attribute 'content' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:753:14:
while evaluating the attribute 'boot.kernelPackages.kernel' at undefined position:
while evaluating 'g' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:301:19, called from undefined position:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:171:72, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/attrsets.nix:304:20:
while evaluating the attribute 'value' at /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:585:9:
while evaluating the option `boot.kernelPackages':
while evaluating 'apply' at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/kernel.nix:40:15, called from /nix/var/nix/profiles/per-user/root/channels/nixos/lib/modules.nix:578:35:
attribute 'extend' missing, at /nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/system/boot/kernel.nix:40:31

Any idea what's up?

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 31, 2022
@cyplo cyplo removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Aug 19, 2022
@patryk4815
Copy link
Contributor

patryk4815 commented Sep 22, 2022

workaround:

  programs.ccache.enable = true;
  programs.ccache.cacheDir = "/opt/ccache";
  programs.ccache.packageNames = [ "linuxPackages_5_19.kernel" ];
  nix.settings.extra-sandbox-paths = [ "/opt/ccache" ];

  nixpkgs.overlays = [
    (self: prev: {
      kernel_cache = (prev.linuxPackages_5_19.kernel.override {
        stdenv = self.ccacheStdenv;
        buildPackages = prev.buildPackages // {
          stdenv = self.ccacheStdenv;
        };
      }).overrideDerivation (attrs: {
        preConfigure = ''
          export NIX_CFLAGS_COMPILE="$(echo "$NIX_CFLAGS_COMPILE" | sed -e "s/-frandom-seed=[^-]*//")"
        '';
      });
    })
  ];

  boot = {
    kernelPackages =  pkgs.linuxPackagesFor pkgs.kernel_cache;
    kernelPatches = [
      {
        name = "fix-something";
        patch = /opt/linux_5_19_9/linux.patch;
        extraConfig = ''
        '';
      }
    ];
  };

programs.ccache.packageNames don't override buildPackages.stdenv ;(
And next problem is frandom-seed soo we need trim it - #109033 ccache/ccache#1002

@aviallon
Copy link
Contributor

aviallon commented Oct 5, 2022

@patryk4815
Fortunately, ccache's master has just been updated to work around the -frandom-seed problems.
ccache/ccache#1168

@jordanisaacs
Copy link
Member

jordanisaacs commented Jan 11, 2023

Curious if someone has an example of it working. It seems to be breaking because the ccache wrapper isn't outputting the right version information? Not entirely sure how the ccache wrapper works and do not want to go down the rabbit hole of figuring it out if it isn't necessary. The exact same build works with normal stdenv. This is the kernel build error:

make[1]: Entering directory '/build/linux-6.1/build'
  GEN     Makefile
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/confdata.o
  HOSTCC  scripts/kconfig/expr.o
  LEX     scripts/kconfig/lexer.lex.c
  YACC    scripts/kconfig/parser.tab.[ch]
  HOSTCC  scripts/kconfig/lexer.lex.o
  HOSTCC  scripts/kconfig/menu.o
  HOSTCC  scripts/kconfig/parser.tab.o
  HOSTCC  scripts/kconfig/preprocess.o
  HOSTCC  scripts/kconfig/symbol.o
  HOSTCC  scripts/kconfig/util.o
  HOSTLD  scripts/kconfig/conf
/nix/store/4g4vvpli8w1c7j4f6r6px1w9k1m46aj6-ccache-links-wrapper-/bin/cc: unknown assembler invoked
scripts/Kconfig.include:50: Sorry, this assembler is not supported.
make[2]: *** [../scripts/kconfig/Makefile:77: oldconfig] Error 1
make[1]: *** [/build/linux-6.1/Makefile:697: oldconfig] Error 2
make[1]: Leaving directory '/build/linux-6.1/build'
make: *** [Makefile:231: __sub-make] Error 2

This is the kernel script that causes the error and what is being invoked.

Interestingly calling the script manually with ./as-version.sh $(ccache_cc) outputs GNU 23900 with no errors. Seems to be something with the CLANG_FLAGS then?

This is an excerpt of my kernel derivation (full source is here):

    ((callPackage "${nixpkgs}/pkgs/os-specific/linux/kernel/manual-config.nix" {})
      {
        inherit src modDirVersion version kernelPatches configfile;
        inherit lib;

        stdenv = ccacheStdenv;
        ....
      })
    .overrideAttrs (old: {
      preConfigure = lib.optionalString ccache.enable ''
        export CCACHE_DIR=${ccache.dir}
        export CCACHE_UMASK=007
      '';
      ...
    })

@minhuw
Copy link

minhuw commented Feb 22, 2024

I encountered a similar issue as @jordanisaacs, and after some investigation, I realized that I had forgotten to update the extra-sandbox-paths setting in my nix.conf file. Once I added extra-sandbox-paths = /nix/var/cache/ccache to my nix.conf, the problem was resolved.

Initially, I wasn't receiving any error messages from ccache because its output was being redirected to /dev/null in the kernel compilation script. However, after removing these redirects with a simple patch, I was able to view the raw ccache output and identify the underlying problem.

I hope this information helps others who might encounter a similar issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants