Multiport #1012

ndhanushkodi · 2022-02-01T02:08:18Z

Based off of research and POC by @lkysow #754 Thank you!

Note to reviewers:
I recommend reading the acceptance test first, then looking at the implementation. In the acceptance tests, I've changed the signature for CheckStaticServerConnection* functions, so there's a lot of 1 line changes all over the acceptance tests due to that.

Changes proposed in this PR:

Support a workaround for multi port pods by registering a Consul service per port, and by injecting init containers and envoy sidecars per port.

Acceptance test for multiport
- The multiport app is just http-echo listening on different ports in 2 containers
- First, the test deploys static-client and multiport, with static-client having multiport and multiport-admin as upstreams. It checks that static-client can make connections to each service-- multiport, and multiport-admin. This is to test inbound connections for a multiport app.
- Then the test deploys static-server, and checks that multiport can make a connection to static-server. This is to test outbound connections from a multiport app. Note that there is only 1 intention because all upstream connections are configured and go through the 1st service's envoy proxy.
Implementation for multiport
- handler
  - uses the service name annotation to figure out if this is a multiport pod or not
  - inject a connect-init container and an envoy-sidecar container per port/service
  - mount all service account tokens not already on the pod. For any service whose service account is not already attached to the pod, this mounts a volume for that service account token. In container-init, these volumes are added as a container volumeMount so the service account token file can be passed to the connect-init command via -bearer-token-file
- container_init -- update the template to pass in arguments specific to the multiport service
- envoy_sidecar -- allow the bootstrap file path to have the service name in it for multiport services, and add --base-id since there could be multiple envoy sidecars
- endpoints_controller: the listener port should be 20000 for the first service, 20001 for the second, etc.
- connect_init command: when its a multiport app, add a filter to poll by service name, use the service specific bearertoken, acl-token-sink, and proxyid files

How I've tested this PR:
acceptance tests and unit tests

How I expect reviewers to test this PR:
see note above, and code review.

Checklist:

Tests added
CHANGELOG entry added

HashiCorp engineers only, community PRs should not add a changelog entry.
Entries should use present tense (e.g. Add support for...)

ndhanushkodi

Adding some comments for reviewers since theres a lot of files

ndhanushkodi · 2022-02-11T21:31:34Z

acceptance/framework/k8s/deploy.go

@@ -16,8 +16,6 @@ import (
 	"k8s.io/apimachinery/pkg/util/yaml"
 )



Changed the function signature for CheckStaticServer* to allow passing in an app which is the app you're connecting from, and an expectedSuccessOutput which is the expected message back from the server. I needed to be able to make connections from the multiport app and not just static client, and each of the multiport services had a different success output.

Could we add this note to a comment for this function(s)🙏

ndhanushkodi · 2022-02-11T21:31:57Z

acceptance/tests/connect/connect_helper.go

@@ -19,6 +19,7 @@ import (

 const staticClientName = "static-client"


This file is function signature changes

ndhanushkodi · 2022-02-11T21:32:25Z

acceptance/tests/connect/connect_inject_namespaces_test.go

@@ -177,9 +177,9 @@ func TestConnectInjectNamespaces(t *testing.T) {
 			if c.secure {
 				logger.Log(t, "checking that the connection is not successful because there's no intention")
 				if cfg.EnableTransparentProxy {
-					k8s.CheckStaticServerConnectionFailing(t, staticClientOpts, fmt.Sprintf("http://static-server.%s", staticServerNamespace))


function signature changes

ndhanushkodi · 2022-02-11T21:33:04Z

acceptance/tests/connect/connect_inject_test.go

+		k8s.CheckStaticServerConnectionSuccessful(t, ctx.KubectlOptions(t), staticClientName, "http://localhost:1234")
+	}
+}
+


Adds the multiport acceptance test

ndhanushkodi · 2022-02-11T21:35:38Z

acceptance/tests/fixtures/Dockerfile

@@ -0,0 +1,2 @@
+FROM kschoche/http-echo:latest


This adds curl to the http-echo image, I plan to swap out this for the actual http-echo image, so this can be ignored for now in review-- the only change would be to delete this file and swap the image.

Swapped and deleted

ndhanushkodi · 2022-02-11T21:37:50Z

acceptance/tests/partitions/partitions_test.go

@@ -415,11 +415,11 @@ func TestPartitions(t *testing.T) {
 				if c.ACLsAndAutoEncryptEnabled {
 					logger.Log(t, "checking that the connection is not successful because there's no intention")
 					if cfg.EnableTransparentProxy {


Function signature changes

ndhanushkodi · 2022-02-11T21:38:00Z

acceptance/tests/terminating-gateway/terminating_gateway_namespaces_test.go

@@ -121,7 +121,7 @@ func TestTerminatingGatewaySingleNamespace(t *testing.T) {



Function signature changes

ndhanushkodi · 2022-02-11T21:38:07Z

acceptance/tests/terminating-gateway/terminating_gateway_test.go

@@ -93,7 +93,7 @@ func TestTerminatingGateway(t *testing.T) {

 			// Test that we can make a call to the terminating gateway.


Function signature changes

ndhanushkodi · 2022-02-11T21:38:16Z

acceptance/tests/vault/vault_test.go

@@ -15,7 +15,8 @@ import (
 )



Function signature changes

ndhanushkodi · 2022-02-11T21:42:37Z

control-plane/connect-inject/endpoints_controller_test.go

+		})
+	}
+}
+


Kept this TestReconcileCreateEndpoint_MultiportService test separate from the existing create endpoint test because of the changes to the case structure that wouldn't be needed by the non-multiport tests. Also because this is a workaround, it might be nice to keep the code as separate as possible.

I really appreciate the comments that you are leaving in the PR!

+1! Love all the comments!

curtbushko · 2022-02-16T15:55:10Z

acceptance/framework/k8s/deploy.go

@@ -90,19 +88,24 @@ func CheckStaticServerConnection(t *testing.T, options *k8s.KubectlOptions, expe
 // If expectSuccess is true, it will expect connection to succeed,
 // otherwise it will expect failure due to intentions. If multiple failureMessages are provided it will assert
 // on the existence of any of them.
-func CheckStaticServerConnectionMultipleFailureMessages(t *testing.T, options *k8s.KubectlOptions, expectSuccess bool, failureMessages []string, curlArgs ...string) {
+func CheckStaticServerConnectionMultipleFailureMessages(t *testing.T, options *k8s.KubectlOptions, app string, expectSuccess bool, failureMessages []string, expectedSuccessOutput string, curlArgs ...string) {


RE: expectedSuccessOutput. I am not 100% sure if you need to change the function signature here as every call to CheckStaticServerConnectionMultipleFailureMessages is setting "" and using the default. Maybe it is good for future proofing?

Nearly every call is setting "" and using the default, except for CheckStaticServerConnectionSuccessfulWithMessage, which is what we're using in the multiport test

curtbushko · 2022-02-16T19:38:13Z

control-plane/connect-inject/endpoints_controller_test.go

+		})
+	}
+}
+


I really appreciate the comments that you are leaving in the PR!

control-plane/connect-inject/envoy_sidecar.go

ishustava

This looks fantastic!!! Thanks you so much for comments throughout the code and the PR - this made it so easy to review ❤️

I left some comments and questions inline, mostly to clarify my own understanding and correct some minor things.

Great work on this!!

acceptance/framework/k8s/deploy.go

ishustava · 2022-02-16T18:08:07Z

acceptance/framework/k8s/deploy.go

@@ -16,8 +16,6 @@ import (
 	"k8s.io/apimachinery/pkg/util/yaml"
 )



Could we add this note to a comment for this function(s)🙏

acceptance/tests/connect/connect_inject_test.go

control-plane/connect-inject/container_init.go

ishustava · 2022-02-16T21:53:41Z

control-plane/connect-inject/endpoints_controller_test.go

+		})
+	}
+}
+


+1! Love all the comments!

control-plane/connect-inject/endpoints_controller_test.go

control-plane/connect-inject/handler.go

ndhanushkodi · 2022-02-18T18:26:13Z

WANFed tests for vault are failing but it might be due to my image? I'll try rebasing and using a new image for acceptance tests, but since everything else is passing and this doesn't touch vault this is ready for a second round of review cc @ishustava @curtbushko

ishustava

🎉 🎉 🎉

A couple of minor comments, but they're not blocking!

control-plane/connect-inject/container_init.go

ishustava · 2022-02-18T19:04:16Z

control-plane/connect-inject/endpoints_controller.go

+			return i
+		}
+	}
+	return -1


nit: I think we could also return an error here (it feels a bit more idiomatic to go) but definitely not a blocker!

I started making this change, but then I realized that in a case where you may have a non-multiport pod, you need to know to ignore the error (for example, in the processUpstreams function). I like the way it is now, because it implies that it's not an error if you don't find a multiport index in the service list, and rather you should handle the -1 by not following the multiport case.

ishustava · 2022-02-18T19:18:10Z

control-plane/connect-inject/envoy_sidecar.go

@@ -62,7 +68,7 @@ func (h *Handler) envoySidecar(namespace corev1.Namespace, pod corev1.Pod) (core
 		// has only injected init containers so all containers defined in pod.Spec.Containers are from the user.
 		for _, c := range pod.Spec.Containers {
 			// User container and Envoy container cannot have the same UID.
-			if c.SecurityContext != nil && c.SecurityContext.RunAsUser != nil && *c.SecurityContext.RunAsUser == envoyUserAndGroupID {
+			if c.SecurityContext != nil && c.SecurityContext.RunAsUser != nil && *c.SecurityContext.RunAsUser == envoyUserAndGroupID && c.Image != h.ImageEnvoy {


I'm curious why do we need that extra image check at the end?

When we're running multiple sidecars in the pod, they clash with the same uid/gid so this check allows using the same uid/gid for 2 envoy sidecars. Since tproxy isn't supported it seems ok to let them run with the same uid/gid.

control-plane/subcommand/connect-init/command.go

wip: need a test for getServiceName

with multiport10 image - any acceptance test that uses CheckStaticServerConnectionMultipleFailureMessages probably need signature updated - add a retrier so it doesn't keep checking for tokens before they are reconciled

signature for CheckStaticServerConnectionMultipleFailureMessages

This is used in multiport12 image

passes acc tests

…r type apps, fix linter

…nfigured in the proxy service registration

…tiport

ndhanushkodi force-pushed the multiport branch from 3a31667 to 5646627 Compare February 11, 2022 19:27

ndhanushkodi commented Feb 11, 2022

View reviewed changes

ndhanushkodi requested review from a team, curtbushko and ishustava and removed request for a team February 11, 2022 22:48

ndhanushkodi marked this pull request as ready for review February 11, 2022 22:48

curtbushko reviewed Feb 16, 2022

View reviewed changes

ishustava reviewed Feb 16, 2022

View reviewed changes

ndhanushkodi force-pushed the multiport branch from 7f7851e to 3bf896e Compare February 18, 2022 06:33

ndhanushkodi requested review from curtbushko and ishustava February 18, 2022 08:38

ishustava approved these changes Feb 18, 2022

View reviewed changes

ndhanushkodi added 16 commits February 18, 2022 11:59

endpoints controller changes and tests

f1507e9

wip: need a test for getServiceName

this passes the multiport acceptance test locally

f1ac1b1

with multiport10 image - any acceptance test that uses CheckStaticServerConnectionMultipleFailureMessages probably need signature updated - add a retrier so it doesn't keep checking for tokens before they are reconciled

use the new multiport-app and change function invocations to use new

bfa57a9

signature for CheckStaticServerConnectionMultipleFailureMessages

refactor handler and continue to pass acceptance tests

81c04b6

This is used in multiport12 image

handler unit test

b284740

fix some unit tests, this is multiport13

d9cead3

passes acc tests

add envoy sidecar and container init tests, fix connect_init tests

3c37737

fix container_init test and bump image

4f28160

add connect-init tests

48aa143

rebase, and remove unused app and add one more token check

7b5bb98

bump multiport image for testing

1d7cc8c

add app to function signature for checking connection to static-serve…

aca066b

…r type apps, fix linter

fix lint

25cfa36

some cleanups to connect init command that passes acceptance tests

93d6c9c

check for injection in acc test, cleanup comments

13a7f2f

add one more unit test to ep controller, cleanup comment

2f6a5b2

ndhanushkodi added 13 commits February 18, 2022 11:59

bump image for acc tests

16fd773

use new http-echo image, change name of test

262eb2b

rename app to sourceApp

bf51f4f

update comments/text

bb466c6

remove redundant configuration for envoy address-- this should get co…

a5294b9

…nfigured in the proxy service registration

only add upstreams to first service's proxy in a multi port pod

53dcc55

fix lint

865b952

bump test image, and use different check for whether something is mul…

b23bcab

…tiport

fix container init test after removing -address flag

7008dec

update rolebindings for both multiport and multiport-admin service accts

24891c3

bump image and add docs

0421d6f

delete ndhanushkodi/http-echo dockerfile

2bbca24

bump image

eb31efd

ndhanushkodi force-pushed the multiport branch from 656f4e2 to eb31efd Compare February 18, 2022 20:12

ndhanushkodi added 3 commits February 18, 2022 13:32

fix vault wan fed test

61e2008

code review

5c41df5

revert image change in ci

e39ea53

ndhanushkodi merged commit 06ab056 into main Feb 22, 2022

ndhanushkodi deleted the multiport branch February 22, 2022 17:53

This was referenced Feb 23, 2022

Lkysow/multiport #754

Closed

connect-init command breaks workaround for supporting multiple ports in sidecar proxy #578

Closed

ndhanushkodi mentioned this pull request Feb 23, 2022

add multiport docs for K8s hashicorp/consul#12428

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiport #1012

Multiport #1012

ndhanushkodi commented Feb 1, 2022 •

edited

Loading

ndhanushkodi left a comment

ndhanushkodi Feb 11, 2022

ishustava Feb 16, 2022

ndhanushkodi Feb 11, 2022

ndhanushkodi Feb 11, 2022

ndhanushkodi Feb 11, 2022

ndhanushkodi Feb 11, 2022

ndhanushkodi Feb 18, 2022

ndhanushkodi Feb 11, 2022

ndhanushkodi Feb 11, 2022

ndhanushkodi Feb 11, 2022

ndhanushkodi Feb 11, 2022

ndhanushkodi Feb 11, 2022

curtbushko Feb 16, 2022

ishustava Feb 16, 2022

curtbushko Feb 16, 2022

ndhanushkodi Feb 18, 2022

curtbushko Feb 16, 2022

ishustava left a comment

ishustava Feb 16, 2022

ishustava Feb 16, 2022

ndhanushkodi commented Feb 18, 2022 •

edited

Loading

ishustava left a comment

ishustava Feb 18, 2022

ndhanushkodi Feb 18, 2022

ishustava Feb 18, 2022

ndhanushkodi Feb 22, 2022

		@@ -16,8 +16,6 @@ import (
		"k8s.io/apimachinery/pkg/util/yaml"
		)

		@@ -19,6 +19,7 @@ import (

		const staticClientName = "static-client"

		@@ -121,7 +121,7 @@ func TestTerminatingGatewaySingleNamespace(t *testing.T) {

		@@ -93,7 +93,7 @@ func TestTerminatingGateway(t *testing.T) {

		// Test that we can make a call to the terminating gateway.

Multiport #1012

Multiport #1012

Conversation

ndhanushkodi commented Feb 1, 2022 • edited Loading

ndhanushkodi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ishustava left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndhanushkodi commented Feb 18, 2022 • edited Loading

ishustava left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndhanushkodi commented Feb 1, 2022 •

edited

Loading

ndhanushkodi commented Feb 18, 2022 •

edited

Loading