Vendor registry files #22566

meteorcloudy · 2024-05-28T13:48:07Z

Vendor registry files needed for Bazel module resolution to achieve
offline build with vendor mode.
Also refactored bazel_vendor_test to avoid vendoring dependencies of
bazel_tools, which speeds up the test significantly.

src/test/py/bazel/bzlmod/bazel_vendor_test.py

src/main/java/com/google/devtools/build/lib/bazel/bzlmod/IndexRegistry.java

fmeum · 2024-06-04T11:06:33Z

src/main/java/com/google/devtools/build/lib/bazel/bzlmod/VendorUtil.java

+      vendorDirectory.createDirectoryAndParents();
+    }
+
+    for (RepositoryName repo : reposToVendor) {


This could probably be parallelized across all repos.

Do we need to worry about atomicity, e.g. if users interrupt the vendor command? Maybe we should atomically move the marker files into their final location at the end?

Indeed, I left a TODO for parallelizing.

But I think copying one marker file after vendoring one repo source is fine? If the vendoring is interrupted, the user probably doesn't want to re-vendor what's already been vendored?

If the copying of the marker file was atomic then I would agree, but what if the marker file is only written partially and then happens to become identical to an up-to-date one even if the contents aren't?

I see! Updated the code to copy the marker file to a tmp file first and then rename it.

there's technically still potential for races unless we actually have a proper locking scheme (eg. multiple Bazels can be working in the same workspace). but we can leave that for future PRs.

then external/A/../B will point to ~/workspace/vendor/B

This only happens when the symlink is resolved somehow, right? Do people actually use this pattern to locate another repo? I would consider this is a hack?

I have used something like this in the past to point --repo_env=JAVA_HOME to the embedded JDK for r_j_e's coursier and I could see some repo rules doing this, but I do agree it's a hack :-) I just wanted to point out that we should have such breakages on our radar when we make this optimization (I think we still should).

Yeah moving + leaving a symlink sounds good to me.

I ended up not doing this. It could cause problem in this situation:

vendor a repo foo: now external/foo -> vendor_dir/foo

Remove --vendor_dir

Think vendor mode is off, and I can change vendor_dir/foo freely (e.g. delete it)

Build breaks because external/foo still points to the modified source

did you actually run into this scenario? I'd expect step 2 to cause RepositoryDelegatorFunction to re-run and realize that --vendor_dir is now off.

Yeah, then it will check the marker file, consider it's up-to-date and won't do anything. I guess we can remove the marker file under external/ to make sure the repo is re-refetched when vendor mode is off. I can explore in a next PR.

src/main/java/com/google/devtools/build/lib/bazel/bzlmod/VendorUtil.java

src/main/java/com/google/devtools/build/lib/bazel/commands/VendorCommand.java

src/main/java/com/google/devtools/build/lib/bazel/BazelRepositoryModule.java

src/main/java/com/google/devtools/build/lib/bazel/commands/VendorCommand.java

src/main/java/com/google/devtools/build/lib/bazel/bzlmod/VendorUtil.java

Wyverald · 2024-06-05T02:54:16Z

src/main/java/com/google/devtools/build/lib/bazel/bzlmod/VendorUtil.java

+      vendorDirectory.createDirectoryAndParents();
+    }
+
+    for (RepositoryName repo : reposToVendor) {


there's technically still potential for races unless we actually have a proper locking scheme (eg. multiple Bazels can be working in the same workspace). but we can leave that for future PRs.

Wyverald · 2024-06-06T00:39:00Z

src/main/java/com/google/devtools/build/lib/bazel/bzlmod/VendorUtil.java

+      vendorDirectory.createDirectoryAndParents();
+    }
+
+    for (RepositoryName repo : reposToVendor) {


Yeah moving + leaving a symlink sounds good to me. In fact I think that was my expectation in the beginning, before we settled on copying (can't remember why now).

src/main/java/com/google/devtools/build/lib/bazel/bzlmod/VendorUtil.java

- Vendor registry files needed for Bazel module resolution to achieve offline build with vendor mode. - Also refactored bazel_vendor_test to avoid vendoring dependencies of bazel_tools, which speeds up the test significantly. Fixes bazelbuild#22554

malt3 · 2024-06-07T19:43:05Z

Hello,
If I understand this change correctly, it would also be super useful for building Bazel itself in an air gapped environment (like the nix sandbox).
Is there any chance that this PR can be backported to the 7.x stream?
It would greatly simplify the packaging of Bazel (and potentially other projects using Bazel) in nixpkgs.

meteorcloudy · 2024-06-10T08:40:44Z

@malt3 Yes, we plan to back port the full vendor feature into 7.3, as 7.2 will probably be released today.

- Vendor registry files needed for Bazel module resolution to achieve offline build with vendor mode. - Also refactored bazel_vendor_test to avoid vendoring dependencies of bazel_tools, which speeds up the test significantly. Fixes bazelbuild#22554 Closes bazelbuild#22566. PiperOrigin-RevId: 641246903 Change-Id: I01a78612ad7ca454df2c70dc5dcc38ec2fbfb71c

meteorcloudy requested a review from Wyverald as a code owner May 28, 2024 13:48

meteorcloudy marked this pull request as draft May 28, 2024 13:48

meteorcloudy removed the request for review from Wyverald May 28, 2024 13:48

github-actions bot added team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. awaiting-review PR is awaiting review from an assigned reviewer labels May 28, 2024

meteorcloudy force-pushed the vendor_registry branch 3 times, most recently from 0747210 to 9448d68 Compare June 3, 2024 15:16

meteorcloudy marked this pull request as ready for review June 3, 2024 15:21

meteorcloudy requested a review from lberki as a code owner June 3, 2024 15:21

meteorcloudy requested review from Wyverald and fmeum and removed request for lberki June 3, 2024 15:21

meteorcloudy commented Jun 3, 2024

View reviewed changes

src/test/py/bazel/bzlmod/bazel_vendor_test.py Show resolved Hide resolved

meteorcloudy requested review from gregestren and fweikert as code owners June 3, 2024 15:49

meteorcloudy removed request for gregestren and fweikert June 3, 2024 15:50

fmeum reviewed Jun 4, 2024

View reviewed changes

meteorcloudy force-pushed the vendor_registry branch from 2ae5fd0 to b618862 Compare June 4, 2024 13:10

fmeum approved these changes Jun 4, 2024

View reviewed changes

Wyverald reviewed Jun 5, 2024

View reviewed changes

meteorcloudy force-pushed the vendor_registry branch from a979e42 to 3e1c4a3 Compare June 5, 2024 12:44

Wyverald approved these changes Jun 6, 2024

View reviewed changes

meteorcloudy added 6 commits June 6, 2024 17:38

Vendor registry files

9f00d63

- Vendor registry files needed for Bazel module resolution to achieve offline build with vendor mode. - Also refactored bazel_vendor_test to avoid vendoring dependencies of bazel_tools, which speeds up the test significantly. Fixes bazelbuild#22554

Add doc about offline

dfba2d9

internal cleanups

70eb000

Polish messages

205a170

Refactor VendorUtil

a7882a1

Add comment for parallelizing vendoring

7ae8e0a

meteorcloudy added 8 commits June 6, 2024 17:38

Refactor marker file name

bc00d2b

Fix vendorDirectory init

2a01092

Fix init VendorCommand.java

25fb936

Make marker file copy atomic

766a6fe

Address reviewer comments

5ab2144

clean up before vendoring

f205946

Add test case to check re-vendor

dd3e0a5

Rename VendorUtil to VendorManager

076de9f

meteorcloudy force-pushed the vendor_registry branch from 4ad573d to 076de9f Compare June 6, 2024 16:03

copybara-service bot closed this in 96b90ba Jun 7, 2024

github-actions bot removed the awaiting-review PR is awaiting review from an assigned reviewer label Jun 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vendor registry files #22566

Vendor registry files #22566

meteorcloudy commented May 28, 2024 •

edited

Loading

fmeum Jun 4, 2024

meteorcloudy Jun 4, 2024

fmeum Jun 4, 2024

meteorcloudy Jun 4, 2024

Wyverald Jun 5, 2024

meteorcloudy Jun 6, 2024

fmeum Jun 6, 2024

meteorcloudy Jun 6, 2024 •

edited

Loading

Wyverald Jun 6, 2024

meteorcloudy Jun 6, 2024

Wyverald Jun 5, 2024

Wyverald Jun 6, 2024

malt3 commented Jun 7, 2024

meteorcloudy commented Jun 10, 2024

Vendor registry files #22566

Vendor registry files #22566

Conversation

meteorcloudy commented May 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meteorcloudy Jun 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

malt3 commented Jun 7, 2024

meteorcloudy commented Jun 10, 2024

meteorcloudy commented May 28, 2024 •

edited

Loading

meteorcloudy Jun 6, 2024 •

edited

Loading