Ignore services in endpoint controller using `consul.hashicorp.com/service-ignore` #858

t-eckert · 2021-11-11T19:45:39Z

Fixes #592

Changes proposed in this PR:

If a service has the label consul.hashicorp.com/service-ignore: "true", it will not be registered by the endpoints controller. If is has been registered before, it will be deregistered.
If the user does not have the correct number of services registered, they will get a log instructing them to add the "connect ignore" label to services which should not be routed.
Some comments have been added where it took me time to understand what processes were doing.
mapAddresses function created as a refactor. I spun off reading this "subprocess" in the Reconcile method and got sidetracked. I think it makes sense to abstract over it. Feel free to disagree.

How I've tested this PR:

Unit tests.
Manual test (detailed below).

How I expect reviewers to test this PR:

Running the unit tests.
The manual testing setup I have been using is here
To test the behavior run the install script which will install Consul onto the cluster; create two services, one with the connect-ignore label; and create a pod which will come online.

Here is a demo of the change, initially the services are deployed without the connect-ignore label on the "headless" service. The pod crashes as consul-connect-inject-init cannot register the service. When the label is added to the "headless" service, the pod comes up in a healthy state:

Screen.Recording.2021-11-19.at.1.02.38.PM.mov

Checklist:

Tests added
CHANGELOG entry added

HashiCorp engineers only, community PRs should not add a changelog entry.
Entries should use present tense (e.g. Add support for...)

control-plane/connect-inject/endpoints_controller.go

thisisnotashwin

hey @t-eckert. I took a quick look and the approach looks sound. Left some comment inline.

control-plane/connect-inject/endpoints_controller_test.go

control-plane/connect-inject/endpoints_controller.go

control-plane/connect-inject/endpoints_controller_test.go

control-plane/connect-inject/endpoints_controller.go

control-plane/connect-inject/endpoints_controller_test.go

control-plane/subcommand/connect-init/command.go

CHANGELOG.md

control-plane/connect-inject/endpoints_controller.go

control-plane/connect-inject/endpoints_controller_test.go

control-plane/subcommand/connect-init/command.go

CHANGELOG.md

control-plane/connect-inject/endpoints_controller_test.go

control-plane/subcommand/connect-init/command.go

kschoche

Looks great @t-eckert, great work on this one.
I left a couple comments, 👍🏽

…oint to the same pod.

Co-authored-by: Kyle Schochenmaier <kschoche@gmail.com>

Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>

Co-authored-by: Kyle Schochenmaier <kschoche@gmail.com>

ishustava

Looks great Thomas!

nflaig · 2021-12-16T15:40:18Z

@t-eckert is there a specific reason why this has to be set as a label? most/all helm charts support setting custom annotations through values but labels are sometimes not supported. Everything else in consul also uses annotations this seems to be the only case where a label is used instead.

nflaig · 2021-12-16T16:15:40Z

control-plane/subcommand/connect-init/command.go

@@ -179,6 +179,12 @@ func (c *Command) Run(args []string) int {
 				c.logger.Info("Check to ensure a Kubernetes service has been created for this application." +
 					" If your pod is not starting also check the connect-inject deployment logs.")
 			}
+			if len(serviceList) > 2 {
+				c.logger.Error("There are multiple Consul services registered for this pod when there must only be one." +


@t-eckert I am running multiple stateful sets in kubernets (kafka, zookeeper, elasticsearch, redis) and all of those have two services normal + headless but I dont get this error? shouldnt this fail the pod scheduling?

Hmm, yes it should be failing scheduling if both the headless and non-headless services select the same pods. Maybe open up an issue about this.

lkysow · 2021-12-17T23:05:48Z

is there a specific reason why this has to be set as a label?

Because we're reacting on changes to the Endpoint object which gets its label from the Service but it doesn't get annotations from the Service.

Note that if you have ACLs enabled we're planning on automatically ignoring all Services that don't have the same name as the serviceAccountName. In which case you won't need to label the other services with ignore.

nflaig · 2021-12-17T23:59:49Z

Note that if you have ACLs enabled we're planning on automatically ignoring all Services that don't have the same name as the serviceAccountName. In which case you won't need to label the other services with ignore.

It that a good idea? If you run stateful sets the headless service is used by most client-to-server connections. Or is there a solution for this? Maybe you could first test that with the most common stateful sets like kafka, zookeeper, etc.

What is recommendation, should stateful sets be used with the service mesh? I feel like the transparent proxy design does not consider stateful sets correctly, for example the issues mentioned in #932

lkysow · 2021-12-20T17:34:06Z

It that a good idea?

Currently Consul only supports one service pointing at one set of pods. If you have two services that point at the same set of pods then none of the pods will start. In addition, when ACLs are enabled the serviceAccountName of the pod must match the service name otherwise the pods won't start. So our plan to ignore services that don't match the serviceAccountName simply means that you won't get in a broken situation. Those services would never have worked before, they would have caused the pods to not come up.

RE statefulsets we definitely want to support them. We need to triage and get to your issue.

nflaig · 2021-12-20T18:28:33Z

statefulsets we definitely want to support them. We need to triage and get to your issue

ok thanks, that is definitely good to hear and of course this is not an easy problem to solve. The work you guys are doing is great and I really appreciate it

* Pass either build URL or build number to terraform resources for GCP/AWS/Azure. On GKE, we can't pass build URL because / is not a valid character in labels * Remove scripting logic that makes tests fail early (we don't need it in nightly acceptance tests) * Fix metrics merging test on GKE. We are running tests there with enterprise license enabled, but since we're setting the consul image to OSS one, we should also unset enterprise license values so that we don't try to run the enterprise license commands against OSS binary, where they don't exist

t-eckert changed the title ~~Ignore services in endpoint controller using consul.hashicorp.com/connect-ignore~~ Ignore services in endpoint controller using consul.hashicorp.com/service-ignore Nov 12, 2021

t-eckert force-pushed the endpoints-controller-connect-ignore branch from 1745150 to 7a21d1f Compare November 15, 2021 18:14

t-eckert marked this pull request as ready for review November 16, 2021 15:58

t-eckert commented Nov 16, 2021

View reviewed changes

control-plane/connect-inject/endpoints_controller.go Outdated Show resolved Hide resolved

t-eckert requested review from ishustava and kschoche November 16, 2021 15:59

thisisnotashwin reviewed Nov 16, 2021

View reviewed changes

control-plane/connect-inject/endpoints_controller_test.go Outdated Show resolved Hide resolved

control-plane/connect-inject/endpoints_controller_test.go Outdated Show resolved Hide resolved

control-plane/connect-inject/endpoints_controller.go Outdated Show resolved Hide resolved

thisisnotashwin reviewed Nov 16, 2021

View reviewed changes

control-plane/connect-inject/endpoints_controller_test.go Outdated Show resolved Hide resolved

t-eckert force-pushed the endpoints-controller-connect-ignore branch from 2ae5ec2 to 970ff02 Compare November 19, 2021 17:26