The system for specifying Bazel's distribution dependencies is too complex

[aiuto]

The scheme we use is needlessly complex, which makes it error prone. If you update a distribution dependency you might have to change a file name in 10 places. Using pkg_name for the archive name, and pkg_sha for the sha256 of it, we have.

distdir_tar(
  name = "additional_distfiles",
  archive = [...  pkg_name.tgz, ...]
  sha256 = { ..., pkg_name.tgz: pkg_sha },
  urls = {... pkg_name.tgz: [  http://mirror/....pkg_name.tgz, http://canonical_host/.../pkg_name.tgz ], ...},
)

Then, sometimes we repeat it again in distdir_tar(name = "test_WORKSPACE_files", ...)
or we use the name, sha256, and urls directly in an http_archive rule.

We could make this easier to update by consolidating the information into a single structure, something like

archives = {
   archive_1: { file_name: pkg_name.tgz, sha256: sha, url: [urls] }
   archive_2: ...
}

distdir_tar would change to take a list of keys from the archive rather than 3 aligned attributes (archive, sha26, urls)
instead of http_archive, we would use a wrapper method that just takes the archive name.

Inspired by #12077

[aiuto]

Current plan:

1. Make create a macro that does an http_archive for a repository from DIST_DEPS Add dist_http_archive() to make it easier to synchronize distdir with http_archive #12722
   This reduces error-prone boilerplate from the top level WORKSPACE
2. Generate embedded *.WORKSPACE files which must be in sync with dist_deps. Generate rules/cpp/cc_configure.WORKSPACE from distdir_deps.bzl rather than keeping it in sync by hand #12743
   Example: src/main/java/com/google/devtools/build/lib/bazel/rules/cpp/cc_configure.WORKSPACE
   Add a rule that can create WORKSPACE stanzas from a template, using references to repositories in DIST_DEPS.
   • Replace the hand made *.WORKSPACE file 'srcs' for the package with the generated file.
   • Note that in the PRs that do this swap we must temporarily exclude the static file from srcs. See the TODO in src/main/java/com/google/devtools/build/lib/bazel/rules/cpp/BUILD in Generate rules/cpp/cc_configure.WORKSPACE from distdir_deps.bzl rather than keeping it in sync by hand #12743
3. Iterate on dependencies until all elements //:additional_distfiles are moved to the DIST_DEPS list.

[aiuto]

Status update: I figured out the pattern for generating files like cc_configure.WORKSPACE. Now that that PR is in place, I can continue for Java.

[aiuto]

Next round: #12779

[aiuto]

Status update

• most dependencies required in the distribution tarball have been converted to the new system.
• Java JDKs remain to be done, I am not an expert,so I am handing that to @comius

[aiuto]

I am closing this out as done enough, based on the status update from 2021-01-21. We can move the few individual JDKS we need as we encounter them, but it is likely we can just leapfrog to bzlmod based deps.