testing/e2e.sh fails on Mac OS X

[Capstan]

This quickly fails due to a missing /bin/true on line 42 of testing/e2e.sh. Swapping that out for /usr/bin/true goes much further but still fails:

____Loading complete.  Analyzing...
____Loading package: @rust_default_toolchains//
____Loading package: @cc_image_base//image
____Found 1 target...
____Building...
Target //testdata:cc_image up-to-date:
  bazel-bin/testdata/cc_image-layer.tar
____Elapsed time: 8.804s, Critical Path: 1.76s

____Running command line: bazel-bin/testdata/cc_image
standard_init_linux.go:185: exec user process caused "exec format error"
ERROR: Non-zero return code '1' from command: Process exited with status 1.
Checking Loaded image ID: sha256:09069941bff078c779c8c5cf1dc7576f35bf032f6b6b945e4e041fe29ea6a7b2
Tagging 09069941bff078c779c8c5cf1dc7576f35bf032f6b6b945e4e041fe29ea6a7b2 as bazel/testdata:cc_image contains Hello World
FAILURE: Expected 'Hello World' not found in 'Loaded image ID: sha256:09069941bff078c779c8c5cf1dc7576f35bf032f6b6b945e4e041fe29ea6a7b2
Tagging 09069941bff078c779c8c5cf1dc7576f35bf032f6b6b945e4e041fe29ea6a7b2 as bazel/testdata:cc_image'

System configuration:

• Mac OS X Sierra 10.12.6 (16G29)
• xCode 8.3.3 (8E3004b)
• Docker Version 17.09.0-ce-mac35 (19611) stable a98b7c1b7c
• bazel release 0.7.0-homebrew
• homebrew config:

HOMEBREW_VERSION: 1.3.6
ORIGIN: https://github.com/Homebrew/brew.git
HEAD: 270b752f5d9d218bfbed6fe85b6974fa653fb25f
Last commit: 3 weeks ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 9f0714bc010f55a59b07174dd5fcd6704ae88c13
Core tap last commit: 23 hours ago
HOMEBREW_PREFIX: /Users/nherring/homebrew
HOMEBREW_REPOSITORY: /Users/nherring/homebrew
HOMEBREW_CELLAR: /Users/nherring/homebrew/Cellar
HOMEBREW_BOTTLE_DOMAIN: https://homebrew.bintray.com
CPU: quad-core 64-bit haswell
Homebrew Ruby: 2.3.3 => /Users/nherring/homebrew/Library/Homebrew/vendor/portable-ruby/2.3.3/bin/ruby
Clang: 8.1 build 802
Git: 2.14.0 => /usr/local/git/current/bin/git
Perl: /usr/bin/perl
Python: /usr/bin/python
Ruby: /usr/bin/ruby => /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/bin/ruby
Java: 1.8.0_91, 1.8.0_72, 1.8.0_60, 1.8.0_45, 1.7.0_79, 1.7.0_76, 1.7.0_71, 1.7.0_67, 1.7.0_65, 1.7.0_60, 1.7.0_55, 1.7.0_51
macOS: 10.12.6-x86_64
Xcode: 8.3.3
CLT: N/A
X11: 2.7.11 => /opt/X11

[Capstan]

bazel run //testdata:cc_image is failing here.

[Capstan]

I'll bet Docker is expecting linux executables, and Mac OS X is building local Darwin, which then is the wrong format. Perhaps we'd need some CROSSTOOL support a la https://github.com/bazelbuild/bazel/wiki/Building-with-a-custom-toolchain

[Capstan]

https://stackoverflow.com/questions/47120292/how-do-i-install-linux-crosstool-compilers-for-bazel-via-brew

[mattmoor]

I think that if we do something in the near-term it should be to make this fail-fast saying the platform is unsupported to avoid confusion.

This test wasn't intended to cover cross-compilation, which is harder to a variety of reasons (including broad upstream toolchain support).

[Capstan]

That seems like a fine interim solution.

It does seem to limit development though, and perhaps we should ensure we have the right issues filed with bazel proper to solve this and other similar requests. I can only imagine that this is an issue that is going to affect the larger docker community who wants to use bazel.

[Capstan]

How hard would it be to build a Linux compiler toolchain inside a Docker image? Since the e2e test already has a dependency on local Docker, we could include a crosstool that knew how to run the compiler from within that image (rather than having to figure out local cross toolchain support for every development host)?

[mattmoor]

Good point, we could just have Bazel inside of a Docker image :)

The e2e test could use: gcr.io/cloud-builders/bazel instead of the host's Bazel.

[nictuku]

Note that Bazel builds very slowly when running inside docker on OSX because volume IO has to go through Linux virtualization for OSX.. and it's very painful.

As a rules_k8s user I would love having a convenient way to build, say, a nodeJS image that runs on Linux, but built from OSX.

How can we support that in a more robust way? I could do the legwork if I had some guidance from the Bazel/rules_docker team.

[mattmoor]

@nictuku You can do this for Go (and probably Java / C++), but it gets challenging for Python / Node.js because they delegate to pip and npm, which don't (AFAIK) have meaningful cross-install stories.

e.g. I can't say npm install --pretend-im-on-linux, which is sort of what we want.

[mattmoor]

For Go, we should add the following to a .bazelrc file in this repo's root:

run --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64

It likely requires an update to the version of rules_go we're using.

[nlopezgi]

is this still an issue? please reopen if so.