secrets/vault: stop depending on Vault root module in test

[zombiezen]

Describe the bug

Our integration test spins up a Vault instance by linking in the code and running it in-process AFAICT. Vault recently split their repository into submodules: github.com/hashicorp/vault, github.com/hashicorp/vault/api, and github.com/hashicorp/vault/sdk. However, they have not released a version of the vault top-level module that excludes the api packages. This causes ambiguous import errors during builds.

To Reproduce

1. Run go mod tidy in gocloud.dev/
2. Run internal/testing/listdeps.sh

Expected behavior

A list of dependencies is printed to stdout.

Actual behavior

The following error message occurs:

build gocloud.dev/samples/gocdk-secrets: cannot load github.com/hashicorp/vault/api: ambiguous import: found github.com/hashicorp/vault/api in multiple modules:
        github.com/hashicorp/vault v1.1.2 (/home/light/pkg/mod/github.com/hashicorp/vault@v1.1.2/api)
        github.com/hashicorp/vault/api v1.0.1 (/home/light/pkg/mod/github.com/hashicorp/vault/api@v1.0.1)

Version

gocloud.dev master (a5a281f)

Additional context

We have been near-missing this because up to this point, we haven't imported anything that specified a module requirement on github.com/hashicorp/vault/api. Running go mod tidy appears to be grabbing everything from Vault tests, which then includes github.com/hashicorp/vault/helper/builtinplugins, which seems like a big offender of bringing in more stuff.

This is a known issue reported (and closed as WAI) upstream: hashicorp/vault#6634

[eliben]

Yikes. This seems to interfere with work on #886 because I'll need to run the equivalent of go mod tidy in the "new" contrib/secrets/vault place.

[zombiezen]

Yeah, this isn't great. I think we should fix it sooner rather than later, because it feels like a landmine.

[eliben]

Any ideas for a fix? In a separate experiment it seems that go mod gets confused about usage of vault in general now for code that both imports the main path and api

[zombiezen]

My proposed solution is to run Vault from a Docker container like how we're handling other dependencies rather than trying to link it in (i.e. remove the main Vault module dependency).

[shantuo]

There also needs to be some change on the makeHarness of the vault tests as we'll need to do some more setup to use the vault binary instead of the test server IIRC. I can take a look at this one.