Add support for docker testclusters #20247

ncabatoff · 2023-04-19T12:40:14Z

No description provided.

…dev or -dev-three-node modes.

…e go test case.

…e go test case, take 2

…it, so instead skip each step. Yuck.

…er-node. All that's missing is for the "current" vault binary to be used.

…he current vault binary.

…of cleanup needed but the concept is sound.

vault/external_tests/raft/raft_test.go

When using the podman service runner (creating a socket equivalent to Docker's), tests fail with a nil pointer exception since the bridge network is named "podman", not "bridge". Allow single networked containers and use whatever name the container runner assigns to it. Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>

cipherboy · 2023-04-24T12:59:25Z

sdk/helper/docker/testhelpers.go

+	} else if d.RunOptions.LogConsumer != nil {
+		consumeLogs = true
+		logStdout = &LogConsumerWriter{d.RunOptions.LogConsumer}
+		logStderr = &LogConsumerWriter{d.RunOptions.LogConsumer}


Logging changes will be great to have for the PKI tests, to avoid having to use the tests.OnError(...) manual logging :-)

sdk/helper/testcluster/docker/environment.go

sdk/helper/docker/testhelpers.go

cipherboy · 2023-04-24T13:23:48Z

sdk/helper/testcluster/docker/environment.go

+func (dc *DockerCluster) addNode(ctx context.Context, opts *DockerClusterOptions) error {
+	i := len(dc.ClusterNodes)
+	nodeID := fmt.Sprintf("core-%d", i)
+	node := &dockerClusterNode{


When adding the first node, and similar to the above comment about saving the network, I wonder if it'd be valuable to re-add the first node's network here?

In particular, we want to ensure that we always land nodes on the same network so they can communicate to each other; I think this only happens if someone is actively playing around with default networks though maybe not if we don't allow choosing the first container's network via DockerCluster?

I don't know that it matters much any more, but the reason why we saved network and re-added it manually to subsequent containers in the pkiext/nginx_test.go suite was that CircleCI's remote docker setup would sometimes land us on different networks, which was the oddest thing ever given the other networks still seemed to exist.

What do you mean by re-adding a network? I looked in nginx_test but couldn't figure out what you're referring to?

Ah, sorry in RunNginxRootTest we call ..., networkName, ... := buildNginxContainer which creates a container on an arbitrary network and subsequent calls which create containers which are supposed to be on the same network take that in e.g., CheckWithClients(t, networkName, ...) -- to ensure they're created on the same network as the nginx instance.

Here, since we're exposing this in sdk as a generic test dependency, we can't know how the container would get created, so it'd be best to ensure that the subsequent nodes land on the same network (via forcing it to) as the first node.

cipherboy · 2023-04-24T13:37:15Z

sdk/helper/testcluster/docker/environment.go

+			"SKIP_SETCAP=true",
+			"VAULT_LOG_FORMAT=json",
+		},
+		Ports:           []string{"8200/tcp", "8201/tcp"},


It might be cool in a future PR to add the ability to use a non-TLS enabled port here.

OCSP and CRL fetching should (theoretically) use a non-TLS listener and I've occasionally had some issues with cert auth method talking to localhost over TLS.

I don't think I need this immediately, perhaps I could add this in the future when I do though. But curious if you have thoughts on how this would look, it doesn't look like we get that much influence over listeners but I might've missed something.

I haven't yet added any support for configuring listeners in HCL, but I don't recall any major obstacles to doing so, just didn't need it yet.

cipherboy

Sorry, didn't mean to request changes, meant to just submit review lol.

# Conflicts: # go.sum # sdk/go.mod # sdk/go.sum

sdk/helper/testcluster/docker/environment.go

miagilepner · 2023-04-24T15:34:59Z

sdk/helper/docker/testhelpers.go

+			break
+		}
+	} else {
+		realIP = inspect.NetworkSettings.Networks[d.RunOptions.NetworkName].IPAddress


is there any chance of the network with that name not being found, and this panicking?

Apparently the answer is yes:

acme_test.go:95: [0] Cluster Node Test_ACMERSAPure-core-0 - podman / 10.88.0.2 acme_test.go:95: [1] Cluster Node Test_ACMERSAPure-core-1 - podman / 10.88.0.3 acme_test.go:95: [2] Cluster Node Test_ACMERSAPure-core-2 - podman / 10.88.0.4 acme_test.go:27: creating on network: podman 2023-04-24T11:49:41.586-0400 [DEBUG] Test_ACMERSAPure.core-2.core.cluster-listener: creating rpc dialer: address=10.88.0.2:8201 alpn=req_fw_sb-act_v1 host=fw-50bb4d28-5e69-6a80-b974-6d3562e805b0 timeout=19.999984975s --- FAIL: Test_ACMERSAPure (36.18s) panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0xb7fcc8] goroutine 14 [running]: testing.tRunner.func1.2({0xd69d00, 0x156da90}) /usr/local/go/src/testing/testing.go:1526 +0x24e testing.tRunner.func1() /usr/local/go/src/testing/testing.go:1529 +0x39f panic({0xd69d00, 0x156da90}) /usr/local/go/src/runtime/panic.go:884 +0x213 github.com/hashicorp/vault/sdk/helper/docker.(*Runner).Start(0xc0222c8780, {0xfe59e8, 0xc00003a0d8}, 0x6?, 0x0) /home/cipherboy/GitHub/cipherboy/vault/sdk/helper/docker/testhelpers.go:458 +0x10a8 github.com/hashicorp/vault/builtin/logical/pkiext.CheckCertBot(0xc000582ea0, {0xc000312770, 0x6}, {0xc02207b720, 0x1d}) /home/cipherboy/GitHub/cipherboy/vault/builtin/logical/pkiext/acme_test.go:39 +0x265 github.com/hashicorp/vault/builtin/logical/pkiext.RunACMERootTest(0xc000582ea0, {0xe7961b, 0x3}, 0x52f785?, 0x0, {0xe7961b, 0x3}, 0xc000083f60?, 0x0) /home/cipherboy/GitHub/cipherboy/vault/builtin/logical/pkiext/acme_test.go:173 +0x19d3 github.com/hashicorp/vault/builtin/logical/pkiext.Test_ACMERSAPure(0x0?) /home/cipherboy/GitHub/cipherboy/vault/builtin/logical/pkiext/acme_test.go:177 +0x38 testing.tRunner(0xc000582ea0, 0xef4488) /usr/local/go/src/testing/testing.go:1576 +0x10b created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1629 +0x3ea FAIL github.com/hashicorp/vault/builtin/logical/pkiext 36.192s FAIL

Ah, that's on me. I passed in networkID rather than NetworkName... Hence why it was confused. 😆

After that, no, it does indeed have that network correctly specified.

miagilepner · 2023-04-24T15:51:45Z

sdk/helper/testcluster/docker/environment.go

+}
+
+func (dc *DockerCluster) GetBarrierOrRecoveryKeys() [][]byte {
+	return dc.GetBarrierKeys()


This comment isn't necessarily related to your changes since I know this implementation is the same as the current TestCluster, but should this function ever return recovery keys instead of barrier keys?

Only once we support non-shamir seals.

@ncabatoff This thread reminds me, I think it'd be really cool if we could support other base layers for the Vault container, happy to open a PR for that.

Not quite sure how it interacts with AddNodes tbh, but what I was thinking is it'd be cool to drop in a container file like:

FROM hashicorp/vault:latest RUN apk install softhsm # ... plus more configuration

and then use PKCS#11 auto-unseal/managed keys/... testing in Vault Ent with SoftHSM.

I think we might need some hooks to let us specify custom callbacks at points in time (startup, cluster join, &c ...) so that we can inject data from other containers (initial node will setup auto-unseal keys which we'll need to replicate into other machines unless we start a PKCS#11 network proxy client on the Vault node and transit all requests to a single, common PKCS#11 network proxy server on another node).

This is definitely enough work to be a completely separate PR though.

sdk/helper/testcluster/docker/environment.go

cipherboy

This worked and was sufficient for my tests in #20320!

cipherboy · 2023-04-24T18:10:26Z

sdk/helper/docker/testhelpers.go

-	container, hostIPs, containerID, err := d.Start(context.Background(), addSuffix, forceLocalAddr)
+	if d.RunOptions.PreDelete {
+		name := d.RunOptions.ContainerName
+		matches, err := d.DockerAPI.ContainerList(ctx, types.ContainerListOptions{


Am I correct in understanding this is exclusive of t.Parallel()? If we specify addSuffix to allow tests to execute in parallel, here we'd kill all uuid-appended variants as well as the main one?

A docs follow-up should be sufficient if so.

I don't think so? It seems to me we're using exact match rather than prefix match in the filter.

The main use case for the pre-delete is for local testing where we're not using addSuffix, to prevent a prior failed run from interfering. We should probably do that labels TODO so that in local testing with addSuffix, old containers get cleaned up; they won't interfere, but they'll take up memory.

Ah, I missed the exact match. I think this is good. Yeah, I'm curious what you would put in the label though... perhaps the test name?

cipherboy · 2023-04-24T18:12:55Z

sdk/helper/docker/testhelpers.go

+	// behaviour for HUP is termination.  So the PostStart that NewDockerCluster
+	// passes in (which does all that PKI cert stuff) waits to see output from
+	// Vault on stdout/stderr before it sends the signal, and we don't want to
+	// run the PostStart until we've hooked into the docker logs.


That's slick and really handy!

Haha I'm glad you think it's slick. I agree it's handy but it feels like a messy kludge... still, it works.

ncabatoff added 25 commits April 18, 2023 09:53

First steps towards docker-based tests: tests using vault binary in -…

7d6da7f

…dev or -dev-three-node modes.

Declare setup dep.

a7a6604

Try to build vault the way test-ui is doing it.

115aa3f

Fix missing runs-on for build-vault.

194965c

Add missing fromJSON

4cabf20

Add checkout, setup-go to build-vault

5f54174

Propagate dev vault binary to test-go

bc85429

Improve vault dev binary handling

8a8aa9e

Move vault dev binary building into test-go

4e3f948

Attempt to only run exec tests (and build vault binary) for the simpl…

0bce1b0

…e go test case.

Attempt to only run exec tests (and build vault binary) for the simpl…

027da9f

…e go test case, take 2

Use inputs instead of env

2f936f1

Fix syntax error

f3bfb40

Fix syntax error, again

0815d23

Fix typo

f99e734

Fix shell bug

65b04cd

Add test godoc

f4c8c3e

Fix JSON syntax

f8289b3

Add default to extra-flags input

d8e4330

Add parens around negated expr to workaround a yaml issue.

157b0fb

Exclude fips case as well

516c3d0

We can't actually skip the build-vault job, since test-go depends on …

f531fec

…it, so instead skip each step. Yuck.

Improve CL and godoc.

bd4f3da

Add support for docker testclusters and server option -dev-three-dock…

69d937c

…er-node. All that's missing is for the "current" vault binary to be used.

Allow docker testclusters to build an image dynamically, shoving in t…

b2b1eee

…he current vault binary.

ncabatoff marked this pull request as ready for review April 19, 2023 13:40

ncabatoff requested review from a team, raskchanky and cipherboy April 19, 2023 13:40

Got the JSON log approach from exec working with docker. Still a bit …

6c9bbfa

…of cleanup needed but the concept is sound.

cipherboy reviewed Apr 21, 2023

View reviewed changes

vault/external_tests/raft/raft_test.go Outdated Show resolved Hide resolved

ncabatoff and others added 5 commits April 21, 2023 13:21

Remove hardcoded vault image tag.

de14905

Make APIClient always return a clone

cf69d45

Fix APIClient: clone doesn't copy tokens.

89117bb

Merge branch 'main' into test-exec-vault-dev

e9c6a17

cipherboy mentioned this pull request Apr 24, 2023

Add tests based on vault binary #20224

Merged

ncabatoff added 3 commits April 24, 2023 09:04

Revisit client cloning.

cdc75c3

Merge branch 'test-exec-vault-dev' into test-docker-vault

cdf3dd5

Follow the exec cluster's lead in how to manage vault clients.

4f2f99e

cipherboy suggested changes Apr 24, 2023

View reviewed changes

cipherboy reviewed Apr 24, 2023

View reviewed changes

cipherboy self-requested a review April 24, 2023 13:37

Base automatically changed from test-exec-vault-dev to main April 24, 2023 13:57

ncabatoff requested a review from a team as a code owner April 24, 2023 13:57

ncabatoff added 2 commits April 24, 2023 09:59

Merge branch 'main' into test-docker-vault

0b51ffd

# Conflicts: # go.sum # sdk/go.mod # sdk/go.sum

Merge main, go tidy, document some weirdness.

a6dd92a

github-advanced-security bot found potential problems Apr 24, 2023

View reviewed changes

sdk/helper/testcluster/docker/environment.go Show resolved Hide resolved

Review feedback.

bbefe63

miagilepner reviewed Apr 24, 2023

View reviewed changes

cipherboy reviewed Apr 24, 2023

View reviewed changes

sdk/helper/testcluster/docker/environment.go Outdated Show resolved Hide resolved

cipherboy mentioned this pull request Apr 24, 2023

Start container-based PKI ACME tests #20320

Merged

export Container field

5cd4a84

ncabatoff requested a review from cipherboy April 24, 2023 17:11

Add godoc

9e4deb3

cipherboy approved these changes Apr 24, 2023

View reviewed changes

ncabatoff merged commit 2f0929f into main Apr 24, 2023

ncabatoff deleted the test-docker-vault branch April 24, 2023 18:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for docker testclusters #20247

Add support for docker testclusters #20247

ncabatoff commented Apr 19, 2023 •

edited

Loading

cipherboy Apr 24, 2023

cipherboy Apr 24, 2023

ncabatoff Apr 24, 2023

cipherboy Apr 24, 2023

cipherboy Apr 24, 2023

ncabatoff Apr 24, 2023

cipherboy left a comment

miagilepner Apr 24, 2023

cipherboy Apr 24, 2023 •

edited

Loading

cipherboy Apr 24, 2023

miagilepner Apr 24, 2023

ncabatoff Apr 24, 2023

cipherboy Apr 24, 2023 •

edited

Loading

cipherboy left a comment •

edited

Loading

cipherboy Apr 24, 2023 •

edited

Loading

ncabatoff Apr 24, 2023

cipherboy Apr 24, 2023

cipherboy Apr 24, 2023

ncabatoff Apr 24, 2023

Add support for docker testclusters #20247

Add support for docker testclusters #20247

Conversation

ncabatoff commented Apr 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cipherboy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cipherboy Apr 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cipherboy Apr 24, 2023 • edited Loading

Choose a reason for hiding this comment

cipherboy left a comment • edited Loading

Choose a reason for hiding this comment

cipherboy Apr 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncabatoff commented Apr 19, 2023 •

edited

Loading

cipherboy Apr 24, 2023 •

edited

Loading

cipherboy Apr 24, 2023 •

edited

Loading

cipherboy left a comment •

edited

Loading

cipherboy Apr 24, 2023 •

edited

Loading