Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bazel: update to 3.5.0 #97497

Closed
epigramengineer opened this issue Sep 9, 2020 · 12 comments
Closed

bazel: update to 3.5.0 #97497

epigramengineer opened this issue Sep 9, 2020 · 12 comments
Labels
0.kind: packaging request Request for a new package to be added

Comments

@epigramengineer
Copy link

When attempting to follow the instructions in pkgs/development/tools/build-managers/bazel/bazel_3/default.nix I ran into an issue with eval $(nix-build -A bazel.updater):

❯ eval $(nix-build -A bazel.updater)
these derivations will be built:
  /nix/store/m8275a3529g9252jd4h66rdawp3lmw3c-bazel-deps.json.drv
  /nix/store/3yf5bgp4jr1h98m0gajn0zmjfzw07683-update-bazel-deps.sh.drv
building '/nix/store/m8275a3529g9252jd4h66rdawp3lmw3c-bazel-deps.json.drv'...
Archive:  /nix/store/z6whc45g0yhcgf8595whhk47ww7f8inq-bazel-3.5.0-dist.zip
  inflating: WORKSPACE
Traceback (most recent call last):
  File "/nix/store/d5h0g8zr573ksxgs31vg3fi7sdbn1pp1-update-srcDeps.py", line 54, in <module>
    exec(f.read())
  File "<string>", line 1050, in <module>
NameError: name 'grpc_deps' is not defined

I'm unsure how to fix that myself; apologies.

See: https://github.com/bazelbuild/bazel/releases
@epigramengineer epigramengineer added the 0.kind: packaging request Request for a new package to be added label Sep 9, 2020
@doronbehar
Copy link
Contributor

cc @mboes

@dmivankov
Copy link
Contributor

`Does 3.4.1 work? It seems bazel is making some breaking changes among 3.x versions, like bazelbuild/rules_docker#1620

@dmivankov
Copy link
Contributor

the real issue is https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/build-managers/bazel/update-srcDeps.py stubbing functions to extract dependencies, it doesn't work anymore with more complex bazel WORKSPACE
https://github.com/bazelbuild/bazel/blob/7cfe416c5b702967b63cb2d9e2a2a2aefb8d2cac/WORKSPACE#L638

@dmivankov
Copy link
Contributor

dmivankov commented Oct 6, 2020 •
edited
Loading

Using bazel itself (so, older but recent enough version or one from a binary distribution)

bazel query 'kind(http_archive, //external:all) + kind(http_file, //external:all) + kind(distdir_tar, //external:all) + kind(git_repository, //external:all)' --output build

will produce the list of dependencies without fancy language features

Notes:

  • not sure whether we need to handle git_repository's
  • there's also xml output option, but build format looks easier to handle
  • bazel query will download multiple GBs the first time, including j2obc, jdk and other things

@dmivankov
Copy link
Contributor

dmivankov commented Oct 6, 2020 •
edited
Loading

Further complications include

@dmivankov
Copy link
Contributor

trying to fix non-checksummed dependencies on bazel side bazelbuild/bazel#12226

@dmivankov
Copy link
Contributor

bazel 3.6.1 should be compatible, but not released yet

@divanorama
Copy link
Contributor

Actually 3.7.0 is still incompatible, waiting for 3.8.0 (November)
Alternatively, I managed to compile it from recent git sources (git sources don't include some generated bootstrap sources, so we need to change compilation process a little), will publish a PR later.

@divanorama
Copy link
Contributor

3.7.0+ / 3.9999 / bazel_git PR #101292

@lambdadog
Copy link
Contributor

Do we have any reasonable solution to just using a bazel version higher than 3.3.1 even if it's not included into nixpkgs? I need bazel>=3.4.0 for the dependency of a project I'm working on and I'm trying to riddle out how I can manage that using nix.

@divanorama
Copy link
Contributor

divanorama commented Nov 4, 2020 •
edited
Loading

I have WIP branch for bazel_git with 3.7.0+ https://github.com/divanorama/nixpkgs/commits/bazel_git
bazel 3.8.0 should only happen in ~1month, we should be able to use that version in nixpkgs then.
It may be possible to make 3.4..3.7 compilable with nix, hopefully it amounts to just manually pinning boringssl and skylib dependencies, but still some work to do.
nixpkgs.bazelisk isn't functional
There also are some docker images, which may be an option
https://console.cloud.google.com/gcr/images/cloud-marketplace/GLOBAL/google/bazel?gcrImageListsize=30
or
https://github.com/GoogleCloudPlatform/cloud-builders/tree/master/bazel

@divanorama divanorama mentioned this issue Dec 15, 2020
Merged
divanorama added a commit to divanorama/nixpkgs that referenced this issue Jan 27, 2021
@divanorama
bazel_4: init at 4.0.0
8bd857d 
Bazel 4 is going to be a long term support release.

Latest version in NixPkgs so far was 3.3.1
There's a need for more recent version
NixOS#97497

All versions from 3.5.0 to 3.7.1 had some reproducibility issues
as noted in issue above, but there also seems to be
a working PR for 3.7.1 now at
NixOS#105439

Notable changes from bazel_3 setup:
- put python to default bash path

  For autodetecting python toolchain
  with strict action_env on and without this change
  bazel would fail to autodetect host python.

  There are some repos that define hermetic python
  toolchains, but they aren't easy to use yet. Also
  telling python paths to bazel isn't a 1-liner it
  seems:
  - action_env=PATH would affect cache
  - declaring toolchain via BUILD&WORKSPACE files
    is not per-user but more like per-repo and
    affects cache too

  Using python from nixpkgs shouldn't be too bad
  in the lack of simpler hermetic python toolchain
  options

- bazel_4.updater is bazel on `bazel query` to support
  new constructs in WORKSPACE (load of vars, transitive
  load etc). This is more robust but requires bazel
  to run the updater, using bazel_3 for now. This is
  only needed to bump package version, doesn't introduce
  bazel_4 build dependency on bazel_3

https://blog.bazel.build/2020/11/10/bazel-4.0-announce.html
https://blog.bazel.build/2020/11/10/long-term-support-release.html
bazelbuild/bazel#12455
https://github.com/bazelbuild/bazel/releases/tag/4.0.0
https://blog.bazel.build/2021/01/19/bazel-4-0.html
divanorama added a commit to divanorama/nixpkgs that referenced this issue Feb 4, 2021
@divanorama
bazel_4: init at 4.0.0
0f5783f 
Bazel 4 is going to be a long term support release.

Latest version in NixPkgs so far was 3.3.1
There's a need for more recent version
NixOS#97497

All versions from 3.5.0 to 3.7.1 had some reproducibility issues
as noted in issue above, but there also seems to be
a working PR for 3.7.1 now at
NixOS#105439

Notable changes from bazel_3 setup:
- put python to default bash path

  For autodetecting python toolchain
  with strict action_env on and without this change
  bazel would fail to autodetect host python.

  There are some repos that define hermetic python
  toolchains, but they aren't easy to use yet. Also
  telling python paths to bazel isn't a 1-liner it
  seems:
  - action_env=PATH would affect cache
  - declaring toolchain via BUILD&WORKSPACE files
    is not per-user but more like per-repo and
    affects cache too

  Using python from nixpkgs shouldn't be too bad
  in the lack of simpler hermetic python toolchain
  options

- bazel_4.updater is bazel on `bazel query` to support
  new constructs in WORKSPACE (load of vars, transitive
  load etc). This is more robust but requires bazel
  to run the updater, using bazel_3 for now. This is
  only needed to bump package version, doesn't introduce
  bazel_4 build dependency on bazel_3

https://blog.bazel.build/2020/11/10/bazel-4.0-announce.html
https://blog.bazel.build/2020/11/10/long-term-support-release.html
bazelbuild/bazel#12455
https://github.com/bazelbuild/bazel/releases/tag/4.0.0
https://blog.bazel.build/2021/01/19/bazel-4-0.html
@divanorama
Copy link
Contributor

bazel_3 is now at 3.7.2 and bazel_4 is at 4.0.0 (bump to 4.1.0 pending), so I think this issue can be closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
0.kind: packaging request Request for a new package to be added
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@divanorama @doronbehar @lambdadog @dmivankov @Artturin @epigramengineer