Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bazel_git and protobuf-java packages #101292

Closed
wants to merge 2 commits into from
Closed

Conversation

@divanorama
Copy link
Contributor

@divanorama divanorama commented Oct 21, 2020

bazel_git is built from fixed recent git sources.

There are several bootstrap options
https://docs.bazel.build/versions/master/install-compile-source.html

  • use release dist sources and compile without bazel installed
  • use git sources and compile with some bazel version
  • use git sources and supply extra bootstrap jars&tools

We could compile with stable versions but this can cause more
expensive rebuilds and we can't expand same compile method to
all bazel versions in nixpkgs.

Few extra changes that were needed to make bootstrap compilation work:

  • provide protobuf-java package
  • change update-srcDeps to depend on "bazel query" output as newer
    versions have more complex dependencies setup with quite some Starlark
    involved
  • don't rewrite src-deps.json on update-srcDeps.py errors

Notes:

  • bazel version is set to 3.9999, which makes sense because there's
    no single version yet, similar to say 3.7.0-SNAPSHOT but keeping
    the numeric format. This may affect .bazelversion or version_check()
  • hopefully we can bump bazel_3 to 3.8.0 once that release is
    available. Versions 3.4 to 3.7.0 have issues with dependencies
    without sha256 or even @master dependencies
  • we can also switch release versions compilation to same bootstrap
    process not relying on dist archives. Potentially may introduce
    differences from binary releases, but we'd be testing full bootstrap
    and rely on git sources.
  • "bazel version" reports
    Build label: 3.9999 ~=3.7.0 (@259930228cfe505c015055b0cec62a406b4881910)
    bazel_3 output is
    Build label: 3.3.1- (@non-git)
Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
bazel_git is built from fixed recent git sources.

There are several bootstrap options
https://docs.bazel.build/versions/master/install-compile-source.html
- use release dist sources and compile without bazel installed
- use git sources and compile with some bazel version
- use git sources and supply extra bootstrap jars&tools

We could compile with stable versions but this can cause more
expensive rebuilds and we can't expand same compile method to
all bazel versions in nixpkgs.

Few extra changes that were needed to make bootstrap compilation work:
- provide protobuf-java package
- change update-srcDeps to depend on "bazel query" output as newer
  versions have more complex dependencies setup with quite some Starlark
  involved
- don't rewrite src-deps.json on update-srcDeps.py errors

Notes:
- bazel version is set to 3.9999, which makes sense because there's
  no single version yet, similar to say 3.7.0-SNAPSHOT but keeping
  the numeric format. This may affect .bazelversion or version_check()
- hopefully we can bump bazel_3 to 3.8.0 once that release is
  available. Versions 3.4 to 3.7.0 have issues with dependencies
  without sha256 or even @master dependencies
- we can also switch release versions compilation to same bootstrap
  process not relying on dist archives. Potentially may introduce
  differences from binary releases, but we'd be testing full bootstrap
  and rely on git sources.
- "bazel version" reports
  Build label: 3.9999 ~=3.7.0 (@259930228cfe505c015055b0cec62a406b4881910)
  bazel_3 output is
  Build label: 3.3.1- (@non-git)
@divanorama divanorama requested a review from Profpatsch as a code owner Oct 21, 2020
buildPhase = ''
cd java
mkdir $TMPDIR/.m2
mvn -Dmaven.repo.local=$TMPDIR/.m2 -DskipTests -Dprotoc2=${protobuf}/bin/protoc package -pl core,util,lite
Copy link
Contributor Author

@divanorama divanorama Oct 21, 2020

looks like this one fails in sandbox, need to somehow put maven dependencies in

local version="$(grep -m 1 '## Release' bazel_src/CHANGELOG.md \
| sed -E 's|.*Release (.*) \(.*\)|\1|')"
export EMBED_LABEL="${version} ~=$version (@${commit})"
${customBash}/bin/bash ./bazel_src/compile.sh
Copy link
Contributor Author

@divanorama divanorama Oct 21, 2020

also fails in sandbox

Building Bazel from scratch......
Building Bazel with Bazel.
...
ERROR: no such package '@bazel_toolchains//rules': java.io.IOException: Error downloading [https://mirror.bazel.build/github.com/bazelbuild/bazel-toolchains/releases/download/3.1.0/bazel-toolchains-3.1.0.tar.gz, https://github.com/bazelbuild/bazel-toolchains/releases/download/3.1.0/bazel-toolchains-3.1.0.tar.gz] to /build/bazel_AFAZt0Fc/out/external/bazel_toolchains/temp4351923723686256938/bazel-toolchains-3.1.0.tar.gz: Unknown host: github.com

probably 2nd phase doesn't use our distdir bazelrc setting because bazel-toolchains-3.1.0.tar.gz is present in src-deps.json

@Profpatsch
Copy link
Member

@Profpatsch Profpatsch commented Oct 22, 2020

Bazel has a pretty tight release cycle (every 2 months or so), is there a particular reason we want a master version? I don’t see it being updated regularly to be honest, and if you need it for your downstream project it should be possible to just copy these expressions there.

@divanorama
Copy link
Contributor Author

@divanorama divanorama commented Oct 22, 2020

I don't have a strong need for git version, but we'll anyway need some of those changes to build 3.8.0+ once it gets released. Can keep this in my fork and prepare 3.8.0 PR later though

@Profpatsch Profpatsch closed this Oct 22, 2020
@Profpatsch
Copy link
Member

@Profpatsch Profpatsch commented Oct 22, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants