Docker build rule with deb packages tries to use xzcat which might not exist

[endobson]

My host machine does not have xzcat installed so when I try to use the docker build rule with a deb package as a source I get the following.

endobson@yggdrasil () ~/tmp/bazel-project % bazel build :minimal_deb                                                                                                                                                                   (0)
INFO: Found 1 target...
INFO: From DockerLayer minimal_deb.layer:
sh: xzcat: command not found
cat: stdout: Broken pipe
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_endobson/dd678381b15653ba614ba2d7587f54af/bazel-project/bazel-out/host/bin/tools/build_defs/docker/build_layer.runfiles/tools/build_defs/docker/build_layer.py", line 166, in <module>
    main(FLAGS(sys.argv))
  File "/private/var/tmp/_bazel_endobson/dd678381b15653ba614ba2d7587f54af/bazel-project/bazel-out/host/bin/tools/build_defs/docker/build_layer.runfiles/tools/build_defs/docker/build_layer.py", line 160, in main
    layer.add_deb(deb)
  File "/private/var/tmp/_bazel_endobson/dd678381b15653ba614ba2d7587f54af/bazel-project/bazel-out/host/bin/tools/build_defs/docker/build_layer.runfiles/tools/build_defs/docker/build_layer.py", line 144, in add_deb
    self.add_tar(tmpfile[1])
  File "/private/var/tmp/_bazel_endobson/dd678381b15653ba614ba2d7587f54af/bazel-project/bazel-out/host/bin/tools/build_defs/docker/build_layer.runfiles/tools/build_defs/docker/build_layer.py", line 108, in add_tar
    self.tarfile.add_tar(tar, numeric=True)
  File "/private/var/tmp/_bazel_endobson/dd678381b15653ba614ba2d7587f54af/bazel-project/bazel-out/host/bin/tools/build_defs/docker/build_layer.runfiles/tools/build_defs/docker/archive.py", line 243, in add_tar
    intar = tarfile.open(fileobj=f, mode='r:')
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tarfile.py", line 1685, in open
    return func(name, filemode, fileobj, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tarfile.py", line 1715, in taropen
    return cls(name, mode, fileobj, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tarfile.py", line 1581, in __init__
    self.firstmember = self.next()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tarfile.py", line 2359, in next
    raise ReadError("empty file")
tarfile.ReadError: empty file
ERROR: /Users/endobson/tmp/bazel-project/BUILD:17:1: null failed: build_layer failed: error executing command bazel-out/host/bin/tools/build_defs/docker/build_layer '--output=bazel-out/local_darwin-fastbuild/bin/minimal_deb.layer' '--directory=/' '--mode=0555' ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Target //:minimal_deb failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.881s, Critical Path: 0.10s

[damienmg]

Yes we should output a clearer message. Unfortunately LZMA compression is not supported in python 2.7 so we have to use that hack :(

[endobson]

What systems does this hack work on? To get it to work on my osx machine I had to install xzcat from macports and then symlink it in /usr/bin because bazel runs 'env -' before commands and doesn't inherit my customized path. And while this is fine for my local work, it doesn't seem all that scaleable for using in a shared package.

Would the eventual solution be to include xzcat (or an equivalent binary) as a tool in the genrule?

[damienmg]

This works on my Darwin machine, and on my Linux machine.
At head it should use your customized environment. This is a bug with earlier version of Bazel.

Finally, the correct solution is to use a python LZMA library but I haven't found a good one and currentlyt don't have time to do it.

[endobson]

I installed bazel from head and am not seeing it use my customized environment.

endobson@yggdrasil () ~/tmp/bazel-project % bazel version                                                                                                                                                                              (1)
Build label: head (@50bf666)
Build target: bazel-out/local_darwin-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel-main_deploy.jar
Build time: Tue Nov 17 08:50:54 2015 (1447750254)
Build timestamp: 1447750254
Build timestamp as int: 1447750254

It still seems that it is running with the empty environment

(cd /private/var/tmp/_bazel_endobson/dd678381b15653ba614ba2d7587f54af/bazel-project && \
  exec env - \
  bazel-out/host/bin/tools/build_defs/pkg/build_tar '--output=bazel-out/local_darwin-fastbuild/bin/minimal_deb.layer' '--directory=/' '--mode=0555' '--tar=lib64-sym.tar' '--deb=bazel-out/local_darwin-fastbuild/genfiles/libc.deb')

[damienmg]

oops...

[damienmg]

It has been rolled-back

[endobson]

This seems to have been rolled forward again and is broken for me on OS X.

The issue that I am seeing is

Repo: https://github.com/endobson/minimal-racket/tree/148608c8a3d4eba496bfe29999ea832fe6af5e99

endobson@yggdrasil () ~/proj/racket/minimal-racket % bazel build -s //racket/...
INFO: Found 2 targets...
>>>>> # //racket:minimal_racket [action 'DockerLayer racket/minimal_racket.layer']
(cd /private/var/tmp/_bazel_endobson/e36ff165f90c34e5ec103dfa2b4ef8e1/minimal-racket && \
  exec env - \
  bazel-out/host/bin/tools/build_defs/pkg/build_tar '--output=bazel-out/local_darwin-fastbuild/bin/racket/minimal_racket.layer' '--directory=/racket' '--mode=0555' '--tar=bazel-out/local_darwin-fastbuild/genfiles/racket/racket-src.tar.gz')
>>>>> # //debian-base:minimal_debian [action 'DockerLayer debian-base/minimal_debian.layer']
(cd /private/var/tmp/_bazel_endobson/e36ff165f90c34e5ec103dfa2b4ef8e1/minimal-racket && \
  exec env - \
  bazel-out/host/bin/tools/build_defs/pkg/build_tar '--output=bazel-out/local_darwin-fastbuild/bin/debian-base/minimal_debian.layer' '--directory=/' '--mode=0555' '--tar=debian-base/lib64-sym.tar' '--deb=bazel-out/local_darwin-fastbuild/genfiles/debian-base/libc.deb')
INFO: From DockerLayer debian-base/minimal_debian.layer:
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_endobson/e36ff165f90c34e5ec103dfa2b4ef8e1/minimal-racket/bazel-out/host/bin/tools/build_defs/pkg/build_tar.runfiles/tools/build_defs/pkg/build_tar.py", line 180, in <module>
    main(FLAGS(sys.argv))
  File "/private/var/tmp/_bazel_endobson/e36ff165f90c34e5ec103dfa2b4ef8e1/minimal-racket/bazel-out/host/bin/tools/build_defs/pkg/build_tar.runfiles/tools/build_defs/pkg/build_tar.py", line 173, in main
    output.add_deb(deb)
  File "/private/var/tmp/_bazel_endobson/e36ff165f90c34e5ec103dfa2b4ef8e1/minimal-racket/bazel-out/host/bin/tools/build_defs/pkg/build_tar.runfiles/tools/build_defs/pkg/build_tar.py", line 141, in add_deb
    self.add_tar(tmpfile[1])
  File "/private/var/tmp/_bazel_endobson/e36ff165f90c34e5ec103dfa2b4ef8e1/minimal-racket/bazel-out/host/bin/tools/build_defs/pkg/build_tar.runfiles/tools/build_defs/pkg/build_tar.py", line 108, in add_tar
    self.tarfile.add_tar(tar, numeric=True, root=root)
  File "/private/var/tmp/_bazel_endobson/e36ff165f90c34e5ec103dfa2b4ef8e1/minimal-racket/bazel-out/host/bin/tools/build_defs/pkg/build_tar.runfiles/tools/build_defs/pkg/archive.py", line 322, in add_tar
    raise self.Error('Cannot handle .xz and .lzma compression: '
tools.build_defs.pkg.archive.Error: Cannot handle .xz and .lzma compression: xzcat not found.
ERROR: /Users/endobson/proj/racket/minimal-racket/debian-base/BUILD:3:1: null failed: build_tar failed: error executing command bazel-out/host/bin/tools/build_defs/pkg/build_tar '--output=bazel-out/local_darwin-fastbuild/bin/debian-base/minimal_debian.layer' '--directory=/' '--mode=0555' '--tar=debian-base/lib64-sym.tar' ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
INFO: Elapsed time: 0.986s, Critical Path: 0.90s

It seems to still be clearing the environment as use_default_shell_environment is only set for the RewriteJson action not the BuildLayer action. This clears the environment (importantly PATH), then when python runs which xzcat it cannot find any executable in the path and thus the error is presented to the user. It looks like that which's behavior is different than that of bash in the case where PATH is not set. Note that I only see this behavior on OS X, in linux running which in the empty environment seems to work as expected using the 'default' path.

[simonhorlick]

I've made a fix to the issue @endobson described above.

[damienmg]

@simonhorlick I would be glad to review it :)

[simonhorlick]

@damienmg I've rebased against the most recent commit and pushed to Gerrit if you'd like to review there: https://bazel-review.googlesource.com/3492

[greggdonovan]

I suggest we re-open this. We've been bit by the dependency on system xzcat and I think we should find a better option than the current implementation, which seems like a hermeticity violation.

E.g. on macOS with --experimental_strict_action_env=true, xzcat will not be found:

• xzcat is not installed by default on macOS.
• With --experimental_strict_action_env=true, the macOS PATH is just /bin:/usr/bin. Both of these directories are under System Integrity Protection, so a user cannot move/symlink a homebrew xzcat to a directory that Bazel can see. We have to add --action_env=PATH=/usr/local/bin:/bin:/usr/bin on macOS to make xzcat visible.

cc: @katre @laszlocsomor

[laszlocsomor]

#4319 is related.