ISSUE-2674 Strip sensitive data from the url #6417

MALPI · 2022-08-03T14:12:08Z

Fixes #2674 by replacing basic auth information as part of the URL with username:password.

…ttributesExtractor

linux-foundation-easycla · 2022-08-03T14:12:12Z

The committers listed above are authorized under a signed CLA.

✅ login: MALPI / name: Malte (4ba4e71, bc1ff06, 89699d1)

mateuszrzeszutek · 2022-08-04T09:30:10Z

Hey @MALPI ,
Can you sign the CLA?

MALPI · 2022-08-04T14:07:49Z

hey @mateuszrzeszutek,

thanks for reaching out. We would need to sing a Company CLA. Could you provide us a PDF or something like that?

mateuszrzeszutek · 2022-08-04T14:18:52Z

Could you provide us a PDF or something like that?

I honestly have no idea where to get it from. Maybe you can try the "please submit a support request ticket" link in the CLA bot message above; or maybe you can ask around in the CNCF #opentelemetry slack.

MALPI · 2022-08-05T07:45:59Z

I am trying to sort this out with the legal department @mateuszrzeszutek. Is there anything that I can improve on the PR itself meanwhile?

...va/io/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractor.java

mateuszrzeszutek · 2022-08-05T08:14:00Z

...va/io/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractor.java

+      return url;
+    }
+    // replace username & password
+    return url.replaceFirst("(?<=\\/\\/)(.+)(?=@)", "username:password");


Regex pattern matching is rather expensive; even more so when you're not using a precompiled Pattern.

How about using indexOf() to find out the positions of // and @ instead?
Also, you could just simply remove the credentials (together with the @ character); this would be more in line with the spec:

http.url MUST NOTcontain credentials passed via URL in form of https://username:password@www.example.com/. In such case the attribute's value should be https://www.example.com/

…trumentation/api/instrumenter/http/HttpClientAttributesExtractor.java Co-authored-by: Mateusz Rzeszutek <mrzeszutek@splunk.com>

… potential performance issues.

… contains an at character.

mateuszrzeszutek · 2022-08-23T10:28:45Z

...va/io/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractor.java

+    if(url.contains("@")) {
+      return url.substring(0, url.indexOf("//")+2) + url.substring(url.indexOf("@")+1);


Nit: no need to lookup "@" twice:

Suggested change

if(url.contains("@")) {

return url.substring(0, url.indexOf("//")+2) + url.substring(url.indexOf("@")+1);

int atIndex = url.indexOf("@");

if (atIndex != -1) {

return url.substring(0, url.indexOf("//")+2) + url.substring(atIndex+1);

mateuszrzeszutek · 2022-08-23T10:30:15Z

@MALPI can you run ./gradlew spotlessApply? I think the formatting might need to be fixed

… run indexOf twice

mateuszrzeszutek

LGTM 👍
Thanks @MALPI !

laurit · 2022-08-23T15:06:46Z

...va/io/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractor.java

+    int atIndex = url.indexOf("@");
+    // replace username & password
+    if (atIndex != -1) {
+      return url.substring(0, url.indexOf("//") + 2) + url.substring(atIndex + 1);


What if the url is http://somehost?foo=b@r or http://somehost/p@ge? I believe you should check that @ occurs before any of /?# and strip only then. Secondly can we be certain that the url actually contains //?

Good point, will add the check for query parameters. I know that basic auth works as well with ftp, but this would require a protocol prefix too? I could add a check for // before @?

hey @MALPI, here's a similar url parser I wrote recently to pull out the "authority" part of a url, in case it helps with inspiration: https://github.com/microsoft/ApplicationInsights-Java/blob/026888fcaa5ba2e0d6a15c06bb68a8862d8852bd/agent/azure-monitor-exporter/src/main/java/com/azure/monitor/opentelemetry/exporter/implementation/utils/UrlParser.java#L10-L16

Lauri pointed out a scenario that needs to be implemented

MALPI · 2022-09-21T08:12:49Z

@laurit @mateuszrzeszutek @trask

Can you have another look?

...o/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractorTest.java

...va/io/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractor.java

…RLParser.

...va/io/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractor.java

…trumentation/api/instrumenter/http/HttpClientAttributesExtractor.java Co-authored-by: Trask Stalnaker <trask.stalnaker@gmail.com>

MALPI · 2022-10-04T08:37:27Z

hey @trask @mateuszrzeszutek anything else?

laurit · 2022-10-04T11:13:00Z

...va/io/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractor.java

+    }
+
+    int atIndex = url.lastIndexOf("@", index - 1);
+    int questionMarkIndex = url.indexOf("?");


is questionMarkIndex really needed?

laurit · 2022-10-04T11:21:05Z

...va/io/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractor.java

+    int questionMarkIndex = url.indexOf("?");
+
+    // '@' char is present and is not a query param
+    if (atIndex == -1 || (questionMarkIndex != -1 && atIndex > index)) {


I think you should also check whether @ is the last character and if it is return the url as is. Try with https://github.com@ java.net.URI parses it so that github.com@ is authority not userinfo.

mateuszrzeszutek · 2022-10-05T07:52:30Z

...va/io/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractor.java

+    int questionMarkIndex = url.indexOf("?");
+
+    // '@' char is present and is not a query param
+    if (atIndex == -1 || (questionMarkIndex != -1 && atIndex > index)) {


atIndex > index will always be false -- as Lauri's mentioned, you don't need that questionMarkIndex variable.

mateuszrzeszutek · 2022-10-05T07:53:37Z

hey @trask @mateuszrzeszutek anything else?

Sorry, I was on vacation, I didn't check GH last week at all.

…is last value of string instead.

MALPI · 2022-10-05T09:21:11Z

All good. Hope you enjoyed your vacation.

To be honest this PR got slightly out of hands and I am wondering if it was really the smartest decision to change the implementation from the Regex I had in the beginning to this.

We now have to exclude all the cases where an @ is not expected, and I'm by no means expert on URLs. Whereas we were looking for what we expect beforehand which makes it way more deterministic.

...va/io/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractor.java

mateuszrzeszutek · 2022-10-05T11:42:20Z

To be honest this PR got slightly out of hands and I am wondering if it was really the smartest decision to change the implementation from the Regex I had in the beginning to this.

Unfortunately regex patterns can be quite slow; and since this code runs on the hot path of every HTTP client call, it needs to be as fast as possible, and create as little garbage as possible. While manual iterating over the string can be inconvenient, it's faster.

…trumentation/api/instrumenter/http/HttpClientAttributesExtractor.java Co-authored-by: Mateusz Rzeszutek <mrzeszutek@splunk.com>

# Conflicts: # instrumentation-api-semconv/src/main/java/io/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractor.java

mateuszrzeszutek · 2022-10-07T11:18:36Z

Thanks @MALPI ! 🚀

Fixes open-telemetry#2674 by replacing basic auth information as part of the URL with `username:password`. Co-authored-by: Malte <MALPI@users.noreply.github.com> Co-authored-by: Mateusz Rzeszutek <mrzeszutek@splunk.com> Co-authored-by: Trask Stalnaker <trask.stalnaker@gmail.com>

ISSUE-2674 Strip sensitive data from the url when used in the clientA…

4ba4e71

…ttributesExtractor

MALPI requested a review from a team as a code owner August 3, 2022 14:12

mateuszrzeszutek reviewed Aug 5, 2022

View reviewed changes

MALPI and others added 2 commits August 5, 2022 15:01

Update instrumentation-api-semconv/src/main/java/io/opentelemetry/ins…

bc1ff06

…trumentation/api/instrumenter/http/HttpClientAttributesExtractor.java Co-authored-by: Mateusz Rzeszutek <mrzeszutek@splunk.com>

Update instrumentation-api-semconv/src/main/java/io/opentelemetry/ins…

89699d1

…trumentation/api/instrumenter/http/HttpClientAttributesExtractor.java Co-authored-by: Mateusz Rzeszutek <mrzeszutek@splunk.com>

MALPI closed this Aug 22, 2022

MALPI reopened this Aug 22, 2022

Issue-2674 Replace regular expression with index of in order to avoid…

4da196e

… potential performance issues.

MALPI closed this Aug 23, 2022

MALPI reopened this Aug 23, 2022

Issue-2674 Add check to only do replacement logic in case the address…

76e9bf8

… contains an at character.

mateuszrzeszutek reviewed Aug 23, 2022

View reviewed changes

Issue-2674 Fix issues reported by checkstyle, apply suggestion not to…

6adfacd

… run indexOf twice

mateuszrzeszutek previously approved these changes Aug 23, 2022

View reviewed changes

laurit reviewed Aug 23, 2022

View reviewed changes

MALPI closed this Sep 19, 2022

MALPI reopened this Sep 19, 2022

ISSUE-2674 Add check for the scheme and a few more test cases

5d90ea7

mateuszrzeszutek reviewed Sep 21, 2022

View reviewed changes

MALPI added 3 commits September 22, 2022 19:42

ISSUE-2674 Change to parameterized test

c9dcfb9

Merge branch 'main' into bug/issue-2674

462c6d2

ISSUE-2674 Add few more test cases

8904b56

ISSUE-2674 Check for the end of the host by making use of code from U…

bb30785

…RLParser.

trask reviewed Sep 26, 2022

View reviewed changes

MALPI and others added 5 commits September 27, 2022 15:49

Update instrumentation-api-semconv/src/main/java/io/opentelemetry/ins…

1101c16

…trumentation/api/instrumenter/http/HttpClientAttributesExtractor.java Co-authored-by: Trask Stalnaker <trask.stalnaker@gmail.com>

Update instrumentation-api-semconv/src/main/java/io/opentelemetry/ins…

92ee662

…trumentation/api/instrumenter/http/HttpClientAttributesExtractor.java Co-authored-by: Trask Stalnaker <trask.stalnaker@gmail.com>

Update instrumentation-api-semconv/src/main/java/io/opentelemetry/ins…

8434511

…trumentation/api/instrumenter/http/HttpClientAttributesExtractor.java Co-authored-by: Trask Stalnaker <trask.stalnaker@gmail.com>

Update instrumentation-api-semconv/src/main/java/io/opentelemetry/ins…

1f41637

…trumentation/api/instrumenter/http/HttpClientAttributesExtractor.java Co-authored-by: Trask Stalnaker <trask.stalnaker@gmail.com>

ISSUE-2674 Fix unit tests

ffdea7d

laurit reviewed Oct 4, 2022

View reviewed changes

mateuszrzeszutek reviewed Oct 5, 2022

View reviewed changes

ISSUE-2674 drop questionmarkindex variable and check if at character …

a25875e

…is last value of string instead.

mateuszrzeszutek approved these changes Oct 5, 2022

View reviewed changes

...va/io/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractor.java Outdated Show resolved Hide resolved

...va/io/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractor.java Outdated Show resolved Hide resolved

laurit approved these changes Oct 5, 2022

View reviewed changes

MALPI and others added 3 commits October 5, 2022 13:53

Update instrumentation-api-semconv/src/main/java/io/opentelemetry/ins…

6b97196

…trumentation/api/instrumenter/http/HttpClientAttributesExtractor.java Co-authored-by: Mateusz Rzeszutek <mrzeszutek@splunk.com>

ISSUE-2674 avoid looking up at character twice

85f4c4f

Merge remote-tracking branch 'origin/bug/issue-2674' into bug/issue-2674

054debc

# Conflicts: # instrumentation-api-semconv/src/main/java/io/opentelemetry/instrumentation/api/instrumenter/http/HttpClientAttributesExtractor.java

mateuszrzeszutek merged commit 7117f42 into open-telemetry:main Oct 7, 2022

krissrex mentioned this pull request Mar 17, 2023

OpenTelemetry tracing uses bad default span naming http4k/http4k#867

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ISSUE-2674 Strip sensitive data from the url #6417

ISSUE-2674 Strip sensitive data from the url #6417

MALPI commented Aug 3, 2022

linux-foundation-easycla bot commented Aug 3, 2022 •

edited

mateuszrzeszutek commented Aug 4, 2022

MALPI commented Aug 4, 2022

mateuszrzeszutek commented Aug 4, 2022 •

edited

MALPI commented Aug 5, 2022

mateuszrzeszutek Aug 5, 2022

mateuszrzeszutek Aug 23, 2022

mateuszrzeszutek commented Aug 23, 2022

mateuszrzeszutek left a comment

laurit Aug 23, 2022

MALPI Aug 28, 2022 •

edited

trask Sep 7, 2022

MALPI commented Sep 21, 2022

MALPI commented Oct 4, 2022

laurit Oct 4, 2022

laurit Oct 4, 2022

mateuszrzeszutek Oct 5, 2022

mateuszrzeszutek commented Oct 5, 2022

MALPI commented Oct 5, 2022

mateuszrzeszutek commented Oct 5, 2022

mateuszrzeszutek commented Oct 7, 2022

		if(url.contains("@")) {
		return url.substring(0, url.indexOf("//")+2) + url.substring(url.indexOf("@")+1);

ISSUE-2674 Strip sensitive data from the url #6417

ISSUE-2674 Strip sensitive data from the url #6417

Conversation

MALPI commented Aug 3, 2022

linux-foundation-easycla bot commented Aug 3, 2022 • edited

mateuszrzeszutek commented Aug 4, 2022

MALPI commented Aug 4, 2022

mateuszrzeszutek commented Aug 4, 2022 • edited

MALPI commented Aug 5, 2022

mateuszrzeszutek Aug 5, 2022

Choose a reason for hiding this comment

mateuszrzeszutek Aug 23, 2022

Choose a reason for hiding this comment

mateuszrzeszutek commented Aug 23, 2022

mateuszrzeszutek left a comment

Choose a reason for hiding this comment

laurit Aug 23, 2022

Choose a reason for hiding this comment

MALPI Aug 28, 2022 • edited

Choose a reason for hiding this comment

trask Sep 7, 2022

Choose a reason for hiding this comment

MALPI commented Sep 21, 2022

MALPI commented Oct 4, 2022

laurit Oct 4, 2022

Choose a reason for hiding this comment

laurit Oct 4, 2022

Choose a reason for hiding this comment

mateuszrzeszutek Oct 5, 2022

Choose a reason for hiding this comment

mateuszrzeszutek commented Oct 5, 2022

MALPI commented Oct 5, 2022

mateuszrzeszutek commented Oct 5, 2022

mateuszrzeszutek commented Oct 7, 2022

linux-foundation-easycla bot commented Aug 3, 2022 •

edited

mateuszrzeszutek commented Aug 4, 2022 •

edited

MALPI Aug 28, 2022 •

edited