Validate all env. vars. before starting injecting env. vars #1141

avadhut123pisal · 2022-10-05T10:34:52Z

This PR adds an implementation to validate the environment variables before starting to mutate the actual container. If validation step fails then it skips the next steps related to common environment variables injection and OTEL SDK Configuration.

Closes #1094

…skipped

Kielek · 2022-10-05T14:43:32Z

pkg/instrumentation/dotnet_test.go

 		},
 	}

 	for _, test := range tests {
 		t.Run(test.name, func(t *testing.T) {
-			pod := injectDotNetSDK(logr.Discard(), test.DotNet, test.pod, 0)
+			pod, sdkInjectionSkipped := injectDotNetSDK(logr.Discard(), test.DotNet, test.pod, 0)


Can you change the contract to the positive value? sdkInjectionSkipped -> sdkInjected. It will requires changes in all places.

BTW current scenario looks like isDisabled=true, which is usually hard to understand and maintain.

Feel free to resolve.

…tor into prevent-incomplete-auto-instrumentation

Kielek · 2022-10-10T08:01:22Z

pkg/instrumentation/dotnet.go

+	// caller checks if there is at least one container.
+	container := &pod.Spec.Containers[index]
+
+	// validate container environment variables.


I think that most of the comments are redundant.

IMO // caller checks if there is at least one container. it is valid comment, but putting information // validate container environment variables. in front of validateContainerEnv does not make sense.

Please check whole PR in this context.

Feel free to resolve

pellared · 2022-10-10T07:27:21Z

pkg/instrumentation/dotnet.go

+	// validate container environment variables.
+	err := validateContainerEnv(container.Env, envDotNetStartupHook, envDotNetAdditionalDeps, envDotNetSharedStore)
+	if err != nil {
+		logger.Info("Skipping DotNet SDK injection", "reason:", err.Error(), "container Name", container.Name)


I think this is how logr should be used

Suggested change

logger.Info("Skipping DotNet SDK injection", "reason:", err.Error(), "container Name", container.Name)

logger.Info("Skipping DotNet SDK injection", "reason", err.Error(), "container", container.Name)

Feel free to resolve

pellared · 2022-10-10T07:30:10Z

pkg/instrumentation/dotnet.go

+	err := validateContainerEnv(container.Env, envDotNetStartupHook, envDotNetAdditionalDeps, envDotNetSharedStore)
+	if err != nil {
+		logger.Info("Skipping DotNet SDK injection", "reason:", err.Error(), "container Name", container.Name)
+		return pod, false


I would prefer if we return error instead of bool. The caller is already doing some loggging.

@pellared I didn't get your point (The caller is already doing some logging).

See: https://github.com/avadhut123pisal/opentelemetry-operator/blob/2a6569c3b168e9f80dda10916d96579bcf4993d5/pkg/instrumentation/sdk.go#L102-L103

I also think that as a rule of thumb the function should either log or return an error. Returning false looks like returning an error without a description.

What is more, if injectDotNetSDK would not log, then the logger would not be need as an argument and the signature would become:

func injectDotNetSDK(dotNetSpec v1alpha1.DotNet, pod corev1.Pod, index int) (corev1.Pod, error)

If we return the error from injectDotNetSDK and not log in injectDotNetSDK, then on the caller side, https://github.com/avadhut123pisal/opentelemetry-operator/blob/2a6569c3b168e9f80dda10916d96579bcf4993d5/pkg/instrumentation/sdk.go#L103
should we use that err value to just handle the condition and to log the message like this ?

pod, err = injectJavaagent(i.logger, otelinst.Spec.Java, pod, index) if err != nil { i.logger.Info("Skipping javaagent injection", "reason", err.Error(), "container", pod.Spec.Containers[index].Name) return pod }

Because if want to propagate the error further in the call stack then we need to modify the signature of inject function too.

That is correct

Feel free to resolve

pellared · 2022-10-10T07:37:25Z

pkg/instrumentation/dotnet.go

+		return pod, false
+	}
+
+	// inject .Net instrumentation spec env vars.


typo:

Suggested change

// inject .Net instrumentation spec env vars.

// inject .NET instrumentation spec env vars.

Feel free to resolve

pellared · 2022-10-10T08:01:16Z

pkg/instrumentation/dotnet.go

 }

-func trySetEnvVar(logger logr.Logger, container *corev1.Container, envVarName string, envVarValue string, concatValues bool) bool {
+// set env var to the container.
+func setDotNetEnvVar(container *corev1.Container, envVarName string, envVarValue string, concatValues bool) {


I suggest describing what concatValues is supposed to offer.
AFAIK it should be set to true if the env var supports multiple values supported by :. If it is set to false, the original container's env var value has priority.

Feel free to resolve

pellared · 2022-10-10T08:17:50Z

pkg/instrumentation/dotnet.go

-	if !trySetEnvVar(logger, &container, envDotNetOTelAutoHome, dotNetOTelAutoHomePath, doNotConcatEnvValues) {
-		return pod
-	}
+	setDotNetEnvVar(container, envDotNetOTelAutoHome, dotNetOTelAutoHomePath, doNotConcatEnvValues)


I think that we should additionally validate that the dotNetOTelAutoHomePath env var was not set in the original container. Otherwise, we cannot auto-instrument the .NET app. If someone set it then it would mean that somebody has already set the .NET AutoInstrumentation in the container.

@Kielek do you agree? I think it would be better addressed in a separate issue/PR.

Yeah. We should address this one in separate PR specific to .Net.

I created #1156. Free free to resolve this comment.

pavolloffay

It would be great to emit an k8s event if the injection fails.

avadhut123pisal · 2022-10-10T15:47:01Z

It would be great to emit an k8s event if the injection fails.

Yes. I will raise separate PR for that, as that would need changes in other places to get the access to the event Recorder.

pavolloffay · 2022-10-10T15:50:19Z

@pellared could you please review as well?

pavolloffay · 2022-10-10T15:50:44Z

or @Kielek could you please review?

pellared · 2022-10-10T16:14:33Z

pkg/instrumentation/sdk.go

+		pod, err = injectJavaagent(otelinst.Spec.Java, pod, index)
+		if err != nil {
+			i.logger.Info("Skipping javaagent injection", "reason", err.Error(), "container", pod.Spec.Containers[index].Name)
+			return pod


This return would skip other instrumentations from being processed. I see it as a bug.

PS. It would be good to add a unit test to make sure that such a bug would not be introduced in the future.

As per my understanding, inject function gets called for a single container at a time. So, there should be only one language instrumentation is required for that particular container.

@pellared Please let me know, if I'm missing something.

there should be only one language instrumentation is required for that particular container

I do not see any docs, code, nor reason that would disallow injecting more language instrumentations. You can have a container that has more than one process.

@pavolloffay Looking at the current implementation, it seems that using multiple instrumentations for the same container will not work. One reason is the duplicate volume mounts, because we use the same mount path. There might be some others things also that can break in context of init container.

I tried adding the annotations for two different language instrumentations, it failed with the error;
Error creating: Pod "spring-petclinic-5d6d58d9b8-pp268" is invalid: spec.containers[0].volumeMounts[2].mountPath: Invalid value: "/otel-auto-instrumentation": must be unique

@pellared Considering the current implementation (multiple instrumentations for a single pod) I don't think return statement would cause any issue.

The goal we had was to support multiple instrumentations for a single pod

@avadhut123pisal I suggest doing the following:

Change the inject function implementation in a way as if support for multiple instrumentations for a single pod is working (e.g. by using else instead of return pod in case of an error).

Create an issue.

Document it in README.md as a known issue.

@pavolloffay Does it seem reasonable?

The goal we had was to support multiple instrumentations for a single pod (e.g. one container java and other python etc.)

@avadhut123pisal feel free to book an issue to resolve this limitation if it is important or open a PR to document this in the readme.

sure !

@pellared Considering the current implementation (multiple instrumentations for a single pod) I don't think return statement would cause any issue.

The code is written in an unmaintainable way. The "error handling" suggests that it works only for one instrumentation. The "happy path scenario" suggests that it supports multiple instrumentations.

@pellared Considering the current implementation (multiple instrumentations for a single pod) I don't think return statement would cause any issue.

The code is written in an unmaintainable way. The "error handling" suggests that it works only for one instrumentation. The "happy path scenario" suggests that it supports multiple instrumentations.

Yeah. I got your point :)

pellared

#1141 (comment)

…avadhut123pisal/opentelemetry-operator into prevent-incomplete-auto-instrumentation

pellared

LGTM (but I was not testing it 😬 )

pavolloffay · 2022-10-12T12:43:18Z

I am merging this based on the approvals.

@avadhut123pisal / @pellared please book the issue to simplify the injection code as discussed before.

avadhut123pisal · 2022-10-12T13:05:55Z

I am merging this based on the approvals.

@avadhut123pisal / @pellared please book the issue to simplify the injection code as discussed before.

#1158

…emetry#1141) * skips env var injection and sdk configurations if agent injection is skipped * mutate container at the last of SDK injection step * validate first and then mutate the container with env variables * fixes go lint issues * incorporates review comments * fixes go lint issue * removes return statement in case of failed instrumentation

avadhut123pisal added 2 commits October 5, 2022 15:29

skips env var injection and sdk configurations if agent injection is …

34127db

…skipped

mutate container at the last of SDK injection step

134c44e

avadhut123pisal requested a review from a team as a code owner October 5, 2022 10:34

avadhut123pisal marked this pull request as draft October 5, 2022 12:32

Kielek reviewed Oct 5, 2022

View reviewed changes

avadhut123pisal added 3 commits October 8, 2022 12:23

Merge branch 'main' of github.com:avadhut123pisal/opentelemetry-opera…

928b2e0

…tor into prevent-incomplete-auto-instrumentation

validate first and then mutate the container with env variables

ff8d0e3

fixes go lint issues

2a6569c

avadhut123pisal changed the title ~~Skip env var injection and OTEL SDK configurations if agent injection is skipped~~ Validate all env. vars. before starting injecting env. vars Oct 8, 2022

avadhut123pisal marked this pull request as ready for review October 8, 2022 07:54

avadhut123pisal requested a review from Kielek October 8, 2022 07:55

Kielek reviewed Oct 10, 2022

View reviewed changes

pellared reviewed Oct 10, 2022

View reviewed changes

avadhut123pisal added 3 commits October 10, 2022 19:55

incorporates review comments

0b2fce7

fixes go lint issue

d703287

Merge branch 'main' into prevent-incomplete-auto-instrumentation

cea2e0d

pavolloffay approved these changes Oct 10, 2022

View reviewed changes

avadhut123pisal requested review from pellared and Kielek and removed request for pellared and Kielek October 10, 2022 15:48

pellared mentioned this pull request Oct 10, 2022

Do not enable .NET instrumentation if OTEL_DOTNET_AUTO_HOME is already set #1156

Closed

pellared reviewed Oct 10, 2022

View reviewed changes

Kielek approved these changes Oct 12, 2022

View reviewed changes

pellared suggested changes Oct 12, 2022

View reviewed changes

removes return statement in case of failed instrumentation

2710133

Merge branch 'prevent-incomplete-auto-instrumentation' of github.com:…

400fc2a

…avadhut123pisal/opentelemetry-operator into prevent-incomplete-auto-instrumentation

pellared approved these changes Oct 12, 2022

View reviewed changes

pavolloffay merged commit 992b681 into open-telemetry:main Oct 12, 2022

avadhut123pisal mentioned this pull request Oct 12, 2022

Support multiple instrumentations for a container #1158

Closed

	logger.Info("Skipping DotNet SDK injection", "reason:", err.Error(), "container Name", container.Name)
	logger.Info("Skipping DotNet SDK injection", "reason", err.Error(), "container", container.Name)

	// inject .Net instrumentation spec env vars.
	// inject .NET instrumentation spec env vars.

Validate all env. vars. before starting injecting env. vars #1141

Validate all env. vars. before starting injecting env. vars #1141

Conversation

avadhut123pisal commented Oct 5, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

avadhut123pisal Oct 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pellared Oct 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavolloffay left a comment

Choose a reason for hiding this comment

avadhut123pisal commented Oct 10, 2022

pavolloffay commented Oct 10, 2022

pavolloffay commented Oct 10, 2022

pellared Oct 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pellared left a comment

Choose a reason for hiding this comment

pellared left a comment

Choose a reason for hiding this comment

pavolloffay commented Oct 12, 2022

avadhut123pisal commented Oct 12, 2022

avadhut123pisal commented Oct 5, 2022 •

edited

avadhut123pisal Oct 10, 2022 •

edited

pellared Oct 10, 2022 •

edited

pellared Oct 10, 2022 •

edited