chore: Migrate to rules_bazel_integration_test

[philsc]

This patch deletes the old copy of bazel_integration_test that we
had vendored into the repo. It's replaced with the maintained
rules_bazel_integration_test.

Summary of changes:

• Increases minimum tested bazel version to 6.4.0.
  rules_bazel_integration_test depends on some features in 6.4
• Moves several CI jobs to be BIT tests. This free ups
  about 10 CI slots.
• Runs these tests under a separate CI job. This is so that feedback
  from the regular test jobs is faster.

Notes about these bazel-in-bazel integration tests:

• The tests are very heavy and easily overwhelm a system. Unfortunately,
  there doesn't appear to be a way to cap their parallelism; only
  disable it entirely using the exclusive tag. Some light
  testing shows there is some speedup to be gained on CI
  if we can, in the future, limit them to 2 or 4 concurrent
  executions.
• A special version named "self" is created that re-uses
  whatever the outer Bazel program is. This is mainly so that
  Bazel's "at head" testing pipelines (the one that runs tests
  with Bazel built from head) are able to affect the integration tests.
  It also comes in handy when locally testing a custom Bazel build.
• The globbing of child workspace files can be somewhat prone to
  following bazel-* symlinks, so its important the .bazelignore
  and deleted packages configs are up-to-date. Otherwise the
  globbing can turn into 30,000+ files and consume a system-freezing
  level of memory and CPU.

Fixes #1209

[rickeylev]

Looking at the CI failures, most errors mention gazelle. Probably the gazelle files aren't inputs to the test? IIRC, gazelle is a separate distribution.

[rickeylev]

Why do we need it to break the sandbox?

[philsc]

If I don't have it break the sandbox, then the tests will fail:

[philsc]

INFO: Repository bazel_features~1.1.1 instantiated at:
  <builtin>: in <toplevel>
Repository rule http_archive defined at:
  /tmp/bazel-working-directory/rules_python/_tmp/2d3bceb7df44f5c60fb62c99b332cef6/_bazel_phil/99580b3b5ce42da1b00cf311bc78b2b8/external/bazel_tools/tools/build_defs/repo/http.bzl:384:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'bazel_features~1.1.1':
   Traceback (most recent call last):
        File "/tmp/bazel-working-directory/rules_python/_tmp/2d3bceb7df44f5c60fb62c99b332cef6/_bazel_phil/99580b3b5ce42da1b00cf311bc78b2b8/external/bazel_tools/tools/build_defs/repo/http.bzl", line 145, column 45, in _http_archive_impl
                download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: error copying file: couldn't delete destination: /home/users/phil/.cache/bazel_repository/content_addressable/sha256/62c26e427e5cbc751024446927622e398a9dcdf32c64325238815709d11c11a8/tmp-44772af6-b733-44fd-b1a2-ea49aa2d0bcf (Read-only file system)
ERROR: <builtin>: fetching http_archive rule //:bazel_features~1.1.1: Traceback (most recent call last):
        File "/tmp/bazel-working-directory/rules_python/_tmp/2d3bceb7df44f5c60fb62c99b332cef6/_bazel_phil/99580b3b5ce42da1b00cf311bc78b2b8/external/bazel_tools/tools/build_defs/repo/http.bzl", line 145, column 45, in _http_archive_impl
                download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: error copying file: couldn't delete destination: /home/users/phil/.cache/bazel_repository/content_addressable/sha256/62c26e427e5cbc751024446927622e398a9dcdf32c64325238815709d11c11a8/tmp-44772af6-b733-44fd-b1a2-ea49aa2d0bcf (Read-only file system)

It uses the user's home directory by default. Not sure if there's a way to fix that. But that's outside the scope of this PR I think.

[rickeylev]

I get the impression the inner bazel is trying to download bazel_features? Strictly speaking, it shouldn't need to because the outer bazel would have (or could have) already done that, but the inner bazel probably isn't aware of where the outer bazel downloaded things to. In any case, yeah, I agree its out of scope for this PR.

Bazel's own CI does something to optimize this, so we can ask them how they make it work (I recall something about they have a special target to eagerly fetch things, then somehow wire the inner bazel to read from there).

[rickeylev]

FYI: I found that all the integration tests in the examples directory were broken. And CI was skipping them (as part of the main repo tests; it ran them as separate jobs in separate workspace directories).

You might want to check if the other existing integration tests were working at head or not. If not, just delete them. We've been going awhile without them, so I don't think they were adding anything. You might be able to start with a clean slate and not have to worry about trying to convert any of the existing ones over.

[rickeylev]

@philsc i rebased this to head and did some cleanup and pushed it to your branch. they were merges, so it should pull just fine for you. hope so anyways! never done such extensive changes to another person's branch

[aignas]

I would not expect the bazel symlinks to be included here. Or at leastthey should be included for all examples.

[rickeylev]

Yeah, I've been confused by this. It's not very clear to me why the symlinks get picked up sometimes and not others. Sometimes adding to this ignore list helps, sometimes I have to delete the symlinks. It seems like some interaction between the outer bazel, deleted packages, the ignore file, and some sort of caching.

[rickeylev]

Ok, I'm going to merge this. It sets up the basics we need to have better test coverage of stuff that happens during earlier phases (repo rules, modules, etc). I think we'll need some custom test runners to fully support those cases, but we at least have the basics to start.