can't use docker spawn strategy on mac

[talya]

Description of the problem / feature request:

Trying to use docker spawn strategy on Mac for java projects, results in standard_init_linux.go:190: exec user process caused "exec format error"

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Using this sample repo

• on a debian machine, running bazel test --spawn_strategy=docker --experimental_docker_image=ubuntu:latest --experimental_docker_privileged=true --experimental_enable_docker_sandbox=true --experimental_docker_verbose=true //... succeeds
• on my mac, running a locally built version of bazel (since I need this fix which was not released in 14.0.1 ) running the same command results in ERROR: /private/var/tmp/_bazel_talyas/42151487a684bdbd20cae185f8059e47/external/junit_junit/jar/BUILD.bazel:2:1: Extracting interface @junit_junit//jar:jar failed (Exit 1) standard_init_linux.go:190: exec user process caused "exec format error"
• try to run with an arm architecture docker image (resin/armhf-alpine) - results in this error - ERROR: /private/var/tmp/_bazel_talyas/42151487a684bdbd20cae185f8059e47/external/junit_junit/jar/BUILD.bazel:2:1: Extracting interface @junit_junit//jar:jar failed (Exit 1) [FATAL tini (29)] exec external/bazel_tools/tools/jdk/ijar/ijar failed: Exec format error

What operating system are you running Bazel on?

• OSx High Sierra
• Docker (Edge) - Version 18.05.0-ce-mac67 (25042)

What's the output of bazel info release?

development version, since I need the PR mentioned above in order to avoid the The command 'chown -R -1:-1 /execroot/__main__' error

Have you found anything relevant by searching the web?

• seems related to the toolchain discussion mentioned in the fix for allowing docker strategy to work on OSx
• also seems closely related to Please allow remote execution to have its own Platform, separate from host and target. #5309

[nlopezgi]

You are running into issues related to having the host environment (where bazel runs) different to the execution environment (the docker container).
In the case of your mac build, the issue is that your rules are using @junit_junit//jar:jar which likely maps to the local_jdk:jar. This target corresponds to the jar binary installed in your mac. The error is occurring when this binary is being executed in the docker container. This use case is not supported with local docker (i.e., you cant run any builds that use rules that depend on local_jdk from a mac->linux container: https://docs.bazel.build/versions/master/remote-execution-rules.html#managing-platform-dependent-binaries)
In the case of your arm docker image build you are using the ijar target that maps to a binary compiled for the host machine (which I'm assuming its not arm). This issue, you might be able to circumvent via adding this flag:
--define=EXECUTOR=remote
This flag enables bazel to use a source version of ijar that will compile in the docker container.
For more info about these issues please see:
https://docs.bazel.build/versions/master/remote-execution-rules.html
https://docs.bazel.build/versions/master/remote-execution-sandbox.html

[laszlocsomor]

@nlopezgi : Thanks for the detailed analysis! Is there a bug we need to fix or can we close the issue?

[ittaiz]

Thanks Nick! Is there a way for us to track down the which of the rules we use depend on existing binaries (for example the jar utility) and are there docs/examples showing how one can refactor such rules?

…

On Fri, 15 Jun 2018 at 10:14 László Csomor ***@***.***> wrote: @nlopezgi <https://github.com/nlopezgi> : Thanks for the detailed analysis! Is there a bug we need to fix or can we close the issue? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5397 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF7HWt2MYOYtsggdBn6B7B9akm2Huks5t817NgaJpZM4Un-IM> .

[nlopezgi]

@laszlocsomor Perhaps it would be good to note in the documentation for the sandbox feature that it does not trivially support cross platform builds and point to the new docs we published (particularly https://docs.bazel.build/versions/master/remote-execution-sandbox.html#troubleshooting-in-a-docker-container) other than that I think we could close.

@ittaiz unfortunately, there is virtually no way to automate / make the task of tracking down which binaries your rules are using. For example, it would be extremely hard to track down that, e.g., you have a rule that invokes some shell script, and that script invokes some tool and that tool indirectly invokes a binary. Nevertheless, you can use some heuristics:

• For access to java tools, the usual culprits are rules using the @local_jdk targets (you should be able to construct a bazel query to try to track that?)
• After that I would search rules that use scripts and templates for occurrences of java or java_home or strings of that sort
• For tools that are not required to run Bazel itself, you can catch if your rules are using them by running a build inside a container as advised here (https://docs.bazel.build/versions/master/remote-execution-sandbox.html#troubleshooting-in-a-docker-container) using a container that has the bare minimum set of tools installed.

In terms of refactoring, the response we have is even more precarious (particularly for java tools), as there is no way (afaik) to refactor uses of java tools currently accessed via the local_jdk to properly enable runs from a mac. The only viable fix (imo) is to request Bazel team to deprecate the local_jdk and replace it with a refactored version of java_toolchain that includes the missing tools.

[ittaiz]

Thanks Nick! I’ll add two more (somewhat side) question: Given we run with docker sand boxing and remote caching: 1. Will Bazel theoretically be able to utilize the remote cache, since the docker sandbox is the same platform, or is the Mac in middle cause this to be a non starter? 2. If the answer to the above is yes do you know what gaps we have for this to be true practically?

…

On Fri, 15 Jun 2018 at 19:20 Nicolas Lopez ***@***.***> wrote: @laszlocsomor <https://github.com/laszlocsomor> Perhaps it would be good to note in the documentation for the sandbox feature that it does not trivially support cross platform builds and point to the new docs we published (particularly https://docs.bazel.build/versions/master/remote-execution-sandbox.html#troubleshooting-in-a-docker-container) other than that I think we could close. @ittaiz <https://github.com/ittaiz> unfortunately, there is virtually no way to automate / make the task of tracking down which binaries your rules are using. For example, it would be extremely hard to track down that, e.g., you have a rule that invokes some shell script, and that script invokes some tool and that tool indirectly invokes a binary. Nevertheless, you can use some heuristics: - For access to java tools, the usual culprits are rules using the @local_jdk targets (you should be able to construct a bazel query to try to track that?) - After that I would search rules that use scripts and templates for occurrences of java or java_home or strings of that sort - For tools that are not required to run Bazel itself, you can catch if your rules are using them by running a build inside a container as advised here ( https://docs.bazel.build/versions/master/remote-execution-sandbox.html#troubleshooting-in-a-docker-container) using a container that has the bare minimum set of tools installed. In terms of refactoring, the response we have is even more precarious (particularly for java tools), as there is no way (afaik) to refactor uses of java tools currently accessed via the local_jdk to properly enable runs from a mac. The only viable fix (imo) is to request Bazel team to deprecate the local_jdk and replace it with a refactored version of java_toolchain that includes the missing tools. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5397 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF8bhcQVEfFG6SBUXPqyRzQ8PESu5ks5t897NgaJpZM4Un-IM> .

[talya]

@nlopezgi - possibly this is unrealted, but I thought you should know, that by simply adding the --define=EXECUTOR=remote flag alone I get a similar error but on C++: /private/var/tmp/_bazel_talyas/42151487a684bdbd20cae185f8059e47/external/bazel_tools/src/main/cpp/util/BUILD:34:1: C++ compilation of rule '@bazel_tools//src/main/cpp/util:filesystem' failed (Exit 1) standard_init_linux.go:190: exec user process caused "exec format error" AND it causes the docker daemon to crash!

see screenshot with 2 consecutive runs, only difference is the flag. reproduces multiple times.

[nlopezgi]

@talya the --define=EXECUTOR=remote flag allows Bazel to use a src version of ijar and singlejar (meaning that the ijar and singlejar binary get compiled in the container). In your build, the compilation of the filesystem target (https://github.com/bazelbuild/bazel/blob/master/src/main/cpp/util/BUILD#L34) is failing. This is probably one of the first c targets that is executed as part of your build. The error you are seeing indicates that you are trying to execute a binary that is not compatible with the docker container (see https://docs.bazel.build/versions/master/remote-execution-rules.html#managing-platform-dependent-binaries). Given the error executes while running a c action, my suspicion is that you are not selecting the c toolchain (via the crosstool flag) properly. The first thing you should do, is start using --verbose_failures so that you can see what command actually tries to run inside the container, from then you will need to track down if the binaries that get executed and are resulting in the error are the right ones to actually execute inside the container.

[nlopezgi]

@ittaiz

1. Will Bazel theoretically be able to utilize the remote cache, since the
   docker sandbox is the same platform, or is the Mac in middle cause this to
   be a non starter?

If you are talking about having a mac host (where bazel is running) launch builds both with local docker sandbox and with a remote execution service (either backed by a local or remote cache), then yes, theoretically this should work.
However, in practice, if you have any rules that are using any java tools (which is practically impossible to avoid) then you will run into issues that you can only avoid if you have the bazel host be a linux machine. So, until the issues with java tools are resolved you would need to run bazel inside a docker container (even with use of the local docker sandbox) - this doc explains how to do this: https://docs.bazel.build/versions/master/remote-execution-sandbox.html#troubleshooting-in-a-docker-container

2. If the answer to the above is yes do you know what gaps we have for this
   to be true practically?

The java tools (i.e., the local_jdk) is the main gap afaik (i.e., the last time I tried to get a build with local docker/re working from a mac host, this was the only blocker that I could not trivially get around)

[laszlocsomor]

@laszlocsomor Perhaps it would be good to note in the documentation for the sandbox feature that it does not trivially support cross platform builds and point to the new docs we published

@nlopezgi , thanks! Would you mind doing that?

[purkhusid]

@nlopezgi @laszlocsomor Has anything changed since this issue was created? Is it possible to run the docker spawn strategy on mac?

[nlopezgi]

There have been no changes here afaik

[laszlocsomor]

Same from me, not aware of any changes.

[jmmv]

I believe this belongs in the configurability / platforms bucket, as this is about getting the host platform decoupled from the execution platform.

[katre]

This should hopefully be fixed by the work on toolchain transitions (#10523) and execution platforms vs strategies (#11432).