Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bazel 0.21.0 #53412

Merged
merged 2 commits into from Jan 10, 2019

Conversation

Projects
None yet
3 participants
@Profpatsch
Copy link
Member

commented Jan 4, 2019

First commit is a patch created by @dtzWill to improve the substituteAll stuff.

Second commit is my progress of getting 0.21 to run so far. 0.21 removed the bundled openjdk-distribution. Instead, tries to fetch the “right” distribution on-the-fly when building. So we need to provide our own openjdk.

According to bazelbuild/bazel#6865 (comment) we should set --host_javabase if we want to do that.

We copy the buildJdk nix path to a temporary directory, make it look like a bazel repository (init BUILD and WORKSPACE) and point to it via local_repository.
As it turns out, exposing the files via filegroup is not enough, the bazel build needs it to expose a JavaRuntimeInfo provider:

ERROR: /tmp/.bazel-1000/bazel_HPQZBqno/out/external/bazel_tools/tools/jdk/BUILD:19:1:
in :alias attribute of java_host_runtime_alias rule
@bazel_tools//tools/jdk:current_host_java_runtime:
'@nix_jdk//:all-files' does not have mandatory providers:
'JavaRuntimeInfo'

I am unsure how to achieve that.

This commit also replaces the line-number-based sed invocations with
something more stable.


I actually tried to follow the upstream suggestion first and use --host_javabase='@local_jdk//:jdk', but that did not work at all:

ERROR: /build/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/BUILD:82:1: error executing shell command: '/nix/store/l04m8mvir9ljd9dngnkxxgmw83wrxzs2-bash/bin/bash -c set -e;rm -rf bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output;mkd...' failed (Exit 127): bash failed: error executing command
  (cd /tmp/.bazel-1000/bazel_18rkBpuk/out/execroot/io_bazel && \
  exec env - \
    PATH=/nix/store/can00lfiynqkbsdkkmgp6qg8p8w92cxa-bash-4.4-p23/bin:/nix/store/0q4i5ll9gxs6giq7kqkniww934j9j8dk-coreutils-8.30/bin:/nix/store/izhcvddpv3n052wp82zm5s8zcg514kr3-findutils-4.6.0/bin:/nix/store/zzzq8a9af192wfsi7lvf0mndpc8ykp4q-gnugrep-3.1/bin:/nix/store/4q0b6gz1yvb4bdzfcbyicz3vmlq05nxa-gnused-4.5/bin:/nix/store/c2czqxjf05amp81si5lvjxswx2bd7j6a-which-2.21/bin \
  /nix/store/l04m8mvir9ljd9dngnkxxgmw83wrxzs2-bash/bin/bash -c 'set -e;rm -rf bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output;mkdir bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/libbootstrap.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/libskylark-deps.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn third_party/auto/auto-value-1.5.4.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn third_party/error_prone/error_prone_annotations-2.2.0.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn third_party/guava/guava-25.1-jre.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn third_party/jcip_annotations/jcip-annotations-1.0-1.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn third_party/jsr305/jsr-305.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/shell/libshell-skylark.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
external/local_jdk/bin/jar cmf bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_MANIFEST.MF bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar -C bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output .
touch bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
')
Execution platform: @bazel_tools//platforms:host_platform

@local_jdk looks like a magic repository, from what I’ve seen it also has special support in bazel source code. Maybe it’s still the way to go.

@mboes @uri-canva @guibou, maybe you can find a way to get this to work.

bazel: greatly reduce time spent substituting, be smart which files
Files inspected: 2756 -> 40
Total size of inspected files: 20M -> 1016K
@Profpatsch

This comment has been minimized.

Copy link
Member Author

commented Jan 8, 2019

It was brought to my attention that there exists an undocumented built-in java_runtime rule that should provide the JavaRuntimeInfo provider and thus hopefully the last missing piece to get this working.

https://github.com/bazelbuild/bazel/blob/master/src/main/tools/jdk.BUILD#L166-L174

bazel: 0.20.0 -> 0.21.0
0.21 removed the bundled openjdk-distribution. Instead, tries to fetch
the “right” distribution on-the-fly when building.
So we need to provide our own openjdk.

According to
bazelbuild/bazel#6865 (comment)
we should set `--host_javabase="@local_jdk//:jdk` if we want to do
that. This uses the jdk that is currently in the environment, which is
openjdk 8 in our case. 0.21 defaulted to a toolchain for JDK9, which
we don’t package in nixpkgs, so we use the JDK8 toolchain.

This commit also replaces the line-number-based sed invocations with
something more stable.

@Profpatsch Profpatsch force-pushed the Profpatsch:bazel-0.21 branch from 5409eec to 581bc7f Jan 10, 2019

@Profpatsch

This comment has been minimized.

Copy link
Member Author

commented Jan 10, 2019

Adding java_runtime does work, but has the same effect as just using @local_jdk//:jdk, leading back to the same build failure. The solution was to add unzip to defaultShellPath in addition to nativeBuildInputs.

@GrahamcOfBorg build bazel

@Profpatsch Profpatsch changed the title WIP: Bazel 0.21.0 Bazel 0.21.0 Jan 10, 2019

@Profpatsch Profpatsch merged commit 9e9fec6 into NixOS:master Jan 10, 2019

12 of 13 checks passed

nix-build -A bazel --argstr system aarch64-linux Build Results
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
nix-build -A bazel --argstr system x86_64-darwin Build Results
Details
nix-build -A bazel --argstr system x86_64-linux Build Results
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.