Unable to patch executable produced by `deno compile` and use it in Nix. REPL starts instead of program.

[0x241F31]

I'm trying to build application with deno compile to single executable with Nix and run it in nix container:

packages.api = pkgs.stdenv.mkDerivation rec {
  name = "api";
  src = ./api;
  __noChroot = true;
  nativeBuildInputs = with pkgs; [
    deno
    unzip
    autoPatchelfHook
  ];
  buildInputs = with pkgs; [
    stdenv.cc.cc
  ];
  buildPhase = ''
    runHook preBuild
    export DENO_DIR=./.deno
    mkdir $DENO_DIR
    deno cache --reload --lock ./deno.lock --lock-write main.ts
    mkdir -p $out/bin
    deno compile -A --check --lock=deno.lock ./main.ts --cached-only --output $out/bin/${name} --target=x86_64-unknown-linux-gnu
    runHook postBuild
  '';
};

Patched libraries:

foxpro:~/craft/sferadel/next$ /nix/store/1x4ijm9r1a88qk7zcmbbfza324gx1aac-glibc-2.37-8/lib/ld-linux-x86-64.so.2 --list /nix/store/agib1vzv0i6br7cy7kdl1ymcm01w5wjx-api/bin/api                  19:22 27.07
        linux-vdso.so.1 (0x00007fffa8f36000)
        libdl.so.2 => /nix/store/1x4ijm9r1a88qk7zcmbbfza324gx1aac-glibc-2.37-8/lib/libdl.so.2 (0x00007ff7134a0000)
        libgcc_s.so.1 => /nix/store/m6kphh9rh424vbw5wq84jcg0r4is4sc5-gcc-12.3.0-libgcc/lib/libgcc_s.so.1 (0x00007ff71347f000)
        libpthread.so.0 => /nix/store/1x4ijm9r1a88qk7zcmbbfza324gx1aac-glibc-2.37-8/lib/libpthread.so.0 (0x00007ff71347a000)
        libm.so.6 => /nix/store/1x4ijm9r1a88qk7zcmbbfza324gx1aac-glibc-2.37-8/lib/libm.so.6 (0x00007ff71339a000)
        libc.so.6 => /nix/store/1x4ijm9r1a88qk7zcmbbfza324gx1aac-glibc-2.37-8/lib/libc.so.6 (0x00007ff70d21a000)
        /nix/store/1x4ijm9r1a88qk7zcmbbfza324gx1aac-glibc-2.37-8/lib/ld-linux-x86-64.so.2 (0x00007ff7134a7000)

Starting produced binary:

foxpro:~/craft/sferadel/next$ with-env {LD_DEBUG: files} { /nix/store/agib1vzv0i6br7cy7kdl1ymcm01w5wjx-api/bin/api }                                                                            19:23 27.07
    505344:
    505344:     file=libdl.so.2 [0];  needed by /nix/store/agib1vzv0i6br7cy7kdl1ymcm01w5wjx-api/bin/api [0]
    505344:     file=libdl.so.2 [0];  generating link map
    505344:       dynamic: 0x00007f26e71e3da8  base: 0x00007f26e71e0000   size: 0x0000000000004010
    505344:         entry: 0x00007f26e71e0000  phdr: 0x00007f26e71e0040  phnum:                 11
    505344:
    505344:
    505344:     file=libgcc_s.so.1 [0];  needed by /nix/store/agib1vzv0i6br7cy7kdl1ymcm01w5wjx-api/bin/api [0]
    505344:     file=libgcc_s.so.1 [0];  generating link map
    505344:       dynamic: 0x00007f26e71dec00  base: 0x00007f26e71bf000   size: 0x0000000000020148
    505344:         entry: 0x00007f26e71bf000  phdr: 0x00007f26e71bf040  phnum:                 10
    505344:
    505344:
    505344:     file=libpthread.so.0 [0];  needed by /nix/store/agib1vzv0i6br7cy7kdl1ymcm01w5wjx-api/bin/api [0]
    505344:     file=libpthread.so.0 [0];  generating link map
    505344:       dynamic: 0x00007f26e71bdda8  base: 0x00007f26e71ba000   size: 0x0000000000004010
    505344:         entry: 0x00007f26e71ba000  phdr: 0x00007f26e71ba040  phnum:                 11
    505344:
    505344:
    505344:     file=libm.so.6 [0];  needed by /nix/store/agib1vzv0i6br7cy7kdl1ymcm01w5wjx-api/bin/api [0]
    505344:     file=libm.so.6 [0];  generating link map
    505344:       dynamic: 0x00007f26e71b8c58  base: 0x00007f26e70da000   size: 0x00000000000df018
    505344:         entry: 0x00007f26e70da000  phdr: 0x00007f26e70da040  phnum:                 11
    505344:
    505344:
    505344:     file=libc.so.6 [0];  needed by /nix/store/agib1vzv0i6br7cy7kdl1ymcm01w5wjx-api/bin/api [0]
    505344:     file=libc.so.6 [0];  generating link map
    505344:       dynamic: 0x00007f26e70ca980  base: 0x00007f26e6ef4000   size: 0x00000000001e5d70
    505344:         entry: 0x00007f26e6f17c90  phdr: 0x00007f26e6ef4040  phnum:                 14
    505344:
    505344:
    505344:     calling init: /nix/store/1x4ijm9r1a88qk7zcmbbfza324gx1aac-glibc-2.37-8/lib/ld-linux-x86-64.so.2
    505344:
    505344:
    505344:     calling init: /nix/store/1x4ijm9r1a88qk7zcmbbfza324gx1aac-glibc-2.37-8/lib/libc.so.6
    505344:
    505344:
    505344:     calling init: /nix/store/1x4ijm9r1a88qk7zcmbbfza324gx1aac-glibc-2.37-8/lib/libm.so.6
    505344:
    505344:
    505344:     calling init: /nix/store/1x4ijm9r1a88qk7zcmbbfza324gx1aac-glibc-2.37-8/lib/libpthread.so.0
    505344:
    505344:
    505344:     calling init: /nix/store/m6kphh9rh424vbw5wq84jcg0r4is4sc5-gcc-12.3.0-libgcc/lib/libgcc_s.so.1
    505344:
    505344:
    505344:     calling init: /nix/store/1x4ijm9r1a88qk7zcmbbfza324gx1aac-glibc-2.37-8/lib/libdl.so.2
    505344:
    505344:
    505344:     initialize program: /nix/store/agib1vzv0i6br7cy7kdl1ymcm01w5wjx-api/bin/api
    505344:
    505344:
    505344:     transferring control: /nix/store/agib1vzv0i6br7cy7kdl1ymcm01w5wjx-api/bin/api
    505344:
Deno 1.35.2
exit using ctrl+d, ctrl+c, or close()
REPL is running with all permissions allowed.
To specify permissions, run `deno repl` with allow flags.
> 

Related: #12780

[0x241F31]

I was able to build it like this:

packages.api = let
  target = "x86_64-unknown-linux-gnu";
  denoCache = pkgs.stdenv.mkDerivation {
    name = "api-deno-cache";
    src = ./api;
    nativeBuildInputs = with pkgs; [
      deno
      unzip
    ];
    configurePhase = ''
      echo "Configuring denoCache"
    '';
    buildPhase = ''
      export DENO_DIR=./.deno; mkdir $DENO_DIR
      deno cache --reload main.ts
    '';
    installPhase = ''
      mkdir $out
      cp -r .deno/deps $out
      cp -r .deno/npm $out
    '';
    outputHashMode = "recursive";
    outputHashAlgo = "sha256";
    outputHash = "sha256-8FzGaIEGM5RIJF0qAi9qKYtZyG+pPw+2kwIOQb171UE=";
    # outputHash = lib.fakeSha256;
  };
  compilier = pkgs.fetchurl {
    url = "https://dl.deno.land/release/v${pkgs.deno.version}/deno-${target}.zip";
    sha256 = "sha256-Q2AdVMRh5+NFhOhFDZKjII5Ol3D/3ebND4LP64HEqks=";
  };
in
  pkgs.stdenv.mkDerivation rec {
    name = "api";
    src = ./api;
    nativeBuildInputs = with pkgs; [
      deno
      unzip
      autoPatchelfHook
    ];
    buildInputs = with pkgs; [
      stdenv.cc.cc
    ];
    dontFixup = true;
    # dontStrip = true;

    configurePhase = ''
      echo "Built cache is on: ${denoCache}"
      export DENO_DIR=.deno
      mkdir $DENO_DIR
      ln -s ${denoCache}/.deno/deps $DENO_DIR/deps
      ln -s ${denoCache}/.deno/npm $DENO_DIR/npm

      mkdir -p $DENO_DIR/dl/release/v${pkgs.deno.version}
      ln -s ${compilier} $DENO_DIR/dl/release/v${pkgs.deno.version}/deno-${target}.zip
      ls -la $DENO_DIR/dl/release/v${pkgs.deno.version}/
    '';
    buildPhase = ''
      mkdir -p $out/bin
      deno compile --unstable -A --check --cached-only --lock deno.lock ./main.ts --output $out/bin/${name} --target=${target}
    '';
  };

[0x241F31]

But it doesn't work this way in Nix Containers. So this is related then: #18764

I was able to bypass it with following hack:

packages.oci-api = nix2c.buildImage {
  name = "oci-api";
  config = {
    Cmd = ["${getExe self'.packages.api}"];
    WorkingDir = "/";
    Env = [
      "DENO_DIR=${self'.packages.api}/.deno"
      "DENO_INSTALL_ROOT=${pkgs.deno}"
      "DENO_VERSION=${pkgs.deno.version}"
      "LD_LIBRARY_PATH=${pkgs.stdenv.cc.libc_lib}/lib"
    ];
    ExposedPorts = {
      "${toString 3001}/tcp" = {};
    };
  };
  copyToRoot = [
    (pkgs.buildEnv {
      name = "root";
      paths = [
        pkgs.deno
        self'.packages.api
        pkgs.stdenv.cc.libc_lib
      ];
      pathsToLink = "/";
    })
  ];
};

This is still not a solution.

[0x241F31]

@06kellyjac

[0x241F31]

Reproduction:

[0x241F31]

Ok, with help of discord community I can show this:

pkgs.stdenv.mkDerivation rec {
  pname = "api";
  version = "0.0.0-rc1";
  src = ./api;
  __noChroot = true;
  compilier = pkgs.fetchurl {
    url = "https://dl.deno.land/release/v${pkgs.deno.version}/deno-${target}.zip";
    sha256 = "sha256-Q2AdVMRh5+NFhOhFDZKjII5Ol3D/3ebND4LP64HEqks=";
  };
  buildInputs = with pkgs; [stdenv.cc.cc];
  nativeBuildInputs = with pkgs; [deno unzip autoPatchelfHook unixtools.xxd hexdump];
  dontAutoPatchelf = true;
  dontPatchELF = true;
  dontStrip = true;
  configurePhase = ''
    export DENO_DIR=.deno; mkdir $DENO_DIR
    deno cache --unstable --lock=deno.lock --lock-write --reload main.ts
    mkdir -p $DENO_DIR/dl/release/v${pkgs.deno.version}
    ln -s ${compilier} $DENO_DIR/dl/release/v${pkgs.deno.version}/deno-${target}.zip
  '';
  buildPhase = ''
    mkdir -p $out/bin
    export OUTPUT=$out/bin/${pname}
    deno compile --unstable -A --lock=deno.lock --check --cached-only ./main.ts --output $OUTPUT --target=${target}
    TRAILER=$(tail -c 40 "$OUTPUT" | hexdump -v -e '1/1 "%02x "')
    autoPatchelf $OUTPUT
    echo "$TRAILER" | xxd -r -p >>"$OUTPUT"
  '';
};

[StarLederer]

One way to work around this would be to bundle the JS app and run it using Deno from Nix store, in fact this might even be a better solution altogether, unfortunately there are currently no feature-complete bundlers for Deno... deno bundle is deprecated and doesn't support NPM, deno_emit just doesn't work sometimes, esbuild and rollup are mentioned in the docs but neither have Deno plugins that resolve all kinds of dependencies Deno supports nor a bootstrap configuration for them.

I understand the drive to build a highly performant deno_emit but I think they should have built a solid Rollup plugin first to establish a good standard for feature completeness and compile-time customization and then focus on building a highly performant competitor from scratch

[CMCDragonkai]

@MrFoxPro you found the right trick. The autoPatchelf hook shouldn't be used that way though.

Instead inside my nix-shell with patchelf as a native build input:

deno compile --output y ./src/main.ts
TRAILER=$(tail -c 40 ./y | hexdump -v -e '1/1 "%02x "')
patchelf --set-interpreter $(cat $NIX_CC/nix-support/dynamic-linker) ./y
echo "$TRAILER" | xxd -r -p >>"./y"
./y
Hello World

Now works.

How did you find out what the last 40 bytes was?

[CMCDragonkai]

It appears the last 40 bytes is some sort of deno binary signature.

64 33 6e 30 6c 34 6e 64 00 00 00 00 04 d9 45 70 00 00 00 00 04 d9 47 15 00 00 00 00 04 d9 4a 95 00 00 00 00 04 d9 4a 99

Where there's an initial:

64 33 6e 30 6c 34 6e 64 00 00 00 00
d3 n0 l4 nd .... (followed by null bytes)

Leet speak d3nol4nd.

I wonder why this is missing in the deno compiled binary?

[06kellyjac]

I'm with @StarLederer on this

unfortunately there are currently no feature-complete bundlers for Deno

And some of that nix seems excessively complex at first glance but obviously I'd need to play with it (ideally with some sample code for ./api) to confirm

[CMCDragonkai]

Well in this case I believe this issue is focused on the compiled binary itself.

[CMCDragonkai]

@MrFoxPro how did you know that you needed exactly 40 bytes?