nixpkgs_package is corrupting binary files

[guibou]

Describe the bug

One of my nixpkgs_package rules loads (in the nix file) a binary file. Which ends corrupted when evaluated by nix in the context of nixpkgs_package (i.e. after the copy).

To Reproduce

Have a binary file directly used in your nix file, something such as:

    src = ./abinaryfile.tar.gz;

This file must appear in the nix_file_deps or your nixpkgs repository. Observes the failure when uncompressing.

Expected behavior

It should not corrupt binaries files.

Environment

• OS name + version:
• Version of the code:

Additional context

I tracked the issue to:

https://github.com/tweag/rules_nixpkgs/blob/master/nixpkgs/nixpkgs.bzl#L677

which I extended with the following:

+    print(repository_ctx.execute([repository_ctx.which("sha256sum"), repository_ctx.path(src)]).stdout)
+    print(repository_ctx.execute([repository_ctx.which("sha256sum"), repository_ctx.path(dest)]).stdout)

And indeed observed that the checksum are not the same.

Aside, I'm not convinced by the executable=False flag of the template call. It seems to me that a file which is considered executable must keep its bit. I don't know if that's important for nix however.

It seems that template does not work well with binary files. Unfortunately there is no such things as a simple copy function in repository context.

I tried the following:
- with an execute(["cp", ...]) but I ended with a lot of problems due to how cp behaves with directory versus files or already existing files
- with an symlink, but issues with existing files.

[guibou]

I also tried:

    # cp may be called many time, for each rules. So before the symlink,
    # we ensure that the destination does not exists.
    repository_ctx.delete(dest)
    repository_ctx.symlink(src, dest)

It build perfectly fine, however it seems to disable the check for nix_file_deps entirely.

[mboes]

The fact that repository_ctx.template is not binary-friendly is a nasty gotcha that should be documented upstream. It furthermore strengthens the case for adding a repository_ctx.cp() function.

[mboes]

@guibou could you open the two tickets upstream as discussed?

Meanwhile, the symlink workaround rendering nix_file_deps is ineffective because Nix might lookup paths inside a .nix file relative to the target of the symlink. In which case we'd be stuck with using the cp command (which we'll have to assume is provided by the host environment).

[guibou]

Meanwhile, the symlink workaround rendering nix_file_deps is ineffective because Nix might lookup paths inside a .nix file relative to the target of the symlink. In which case we'd be stuck with using the cp command (which we'll have to assume is provided by the host environment).

I'm having issue with cp, it was creating some directories in subdirectories instead of creating them where they need to appear. I tried cp -T but no success.

@guibou could you open the two tickets upstream as discussed?

• feature request: repository_ctx.copy bazelbuild/bazel#11858
• document that repository_ctx.template requires text files bazelbuild/bazel#11857