Modularize core Bazel dependencies

[meteorcloudy]

Dependencies of the Bazel project itself should be added into the Bazel Central Registry:

• bazel_skylib
• bazel_skydoc
• com_google_protobuf
• platforms
• rules_cc
• rules_java
• rules_proto
• rules_pkg
• rules_python
• rules_nodejs
• rules_sass
• grpc
• upb
• c-ares
• zlib
• abseil-cpp
• re2
• boringssl

Blockers:

• There should be periodic named releases. rules_cc#91
• Have a new release for stardoc stardoc#109

Notes:

• Currently we are writing MODULE.bazel files for those project and checking them in the BCR, that is fine because Bzlmod don't read MODULE.bazel from the source archive. But ideally they should be upstreamed.

[meteorcloudy]

Blocker for this issue: rules_cc don't have official releases, therefore we cannot check in a specific version.

bazelbuild/rules_cc#91

[meteorcloudy]

stardoc also hasn't have new releases for a while and Bazel is using a commit that is newer than its latest release.

[Wyverald]

Can't we just download any commit from github as a zip? Like https://github.com/bazelbuild/rules_cc/archive/6c5c8e9b41e6427033dd011f8ce52a1f2532abdc.zip, and then just make it version "0.0.0-20210924-6c5c8e9b" or something

[meteorcloudy]

We can certainly do that as a workaround, but ideally projects rules_cc should have official releases.

[Wyverald]

another way to go about this is to have source.json support git commits too; so instead of specifying an archive from a URL, you could alternatively specify a git repo and a commit/tag.

We can certainly do that as a workaround, but ideally projects rules_cc should have official releases.

Agreed.

[meteorcloudy]

another way to go about this is to have source.json support git commits too; so instead of specifying an archive from a URL, you could alternatively specify a git repo and a commit/tag.

But you still need a fake comparable version for that right?

[meteorcloudy]

In generally, I would like people to check in well-tested release versions of their project in to BCR, instead of a random commit.

[Wyverald]

But you still need a fake comparable version for that right?

Yeah, the only difference is that a URL like GitHub's to download an arbitrary commit as a zip isn't always available (for example, for your company's own git repos).

In generally, I would like people to check in well-tested release versions of their project in to BCR, instead of a random commit.

That's fair -- I think we should make more effort to push projects such as rules_cc to make a release then (potentially doing it ourselves). I just don't want us to be stuck in a limbo where rules_cc is reluctant to make a release, so we're just blocked on them and not making progress.

[meteorcloudy]

I just don't want us to be stuck in a limbo where rules_cc is reluctant to make a release, so we're just blocked on them and not making progress.

Oh, of course, making a new release shouldn't be too hard (hopefully) and we have many other dependencies to check in.

[alexeagle]

Dependencies of the Bazel project itself should be added into the Bazel Central Registry

Wait, is this registry for Bazel's use or the community? If zlib goes here, does that mean other C++ libraries can be contributed by users? You'll have hundreds of "modules" here very quickly.

[Wyverald]

This is for use by the community. Other C++ libraries can indeed be contributed by users.

You'll have hundreds of "modules" here very quickly.

Scalability has always been a concern of mine, but similar projects (notably Homebrew, and to some extent Debian) have functioned well with this pattern, so I think we should be able to manage.

[alexeagle]

only difference is that a URL like GitHub's to download an arbitrary commit as a zip isn't always available

there is another difference which is that integrity hashes of github's archive downloads are not stable. They sometimes rollout upgrades to their frontend nodes where the tar or zip command run and those have small variation. The registry should probably strongly advise and/or enforce only release/ URLs are used, where the integrity matches what the rule maintainer uploaded

[meteorcloudy]

there is another difference which is that integrity hashes of github's archive downloads are not stable. They sometimes rollout upgrades to their frontend nodes where the tar or zip command run and those have small variation.

That's a good point. To mitigate this, we also mirror all archive URLs to a GCS bucket.
See https://buildkite.com/bazel/bcr-postsubmit/builds/7#eb1a86ce-5b1a-461c-9f87-cf2f1b20a78c

[alexeagle]

oh this is a place to automate the mirroring of new releases, that's a great improvement. of course if GitHub changes, you'll have one correct mirror and one incorrect one.