Add net.peer.ip in requests & urllib3 instrumentations. #661

Oberon00 · 2021-09-06T13:38:21Z

Description

Capture net.peer.ip in requests & urllib3 instrumentations, using a shared new ~~package opentelemetry-instrumentation-http-base~~ instrumentor for httplib.client that could also be used to easily add net.peer.ip to other instrumentations based on the stdlib's httplib.client (but this PR starts with only the aforementioned two).

Type of change

Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Unit tests included. Due to how sharing of test code is currently set up, this requires a change to the main repo to add the test base class.

Does This PR Require a Core Repo Change?

Yes. - Link to PR: Add test base class with local HTTP server. opentelemetry-python#2101

Checklist:

Followed the style guidelines of this project
Changelogs have been updated
Unit tests have been added
~~Documentation has been updated~~

Oberon00 · 2021-09-06T13:56:09Z

What's up with these pylint errors? instrumentation/opentelemetry-instrumentation-botocore/tests/test_botocore_instrumentation.py:133:16: E1101: Instance of 'TestBotocoreInstrumentor' has no 'get_finished_spans' member (no-member)

lzchen · 2021-09-07T15:41:14Z

Any chance we can add this functionality to http utils? We already have that package to contain common http functionality.

Oberon00 · 2021-09-07T16:27:57Z

I think that should not be a technical problem. It will add a dependency on asgiref though and so far util-http only instruments server-side libs. If you prefer I can merge the new http-base package into util-http.

Oberon00 · 2021-09-14T10:34:50Z

@lzchen I'll go ahead and change the package.

Oberon00 · 2021-09-14T13:51:13Z

@lzchen Done. Instead of adding a new package, I now extend the util-http package, like you suggested.

.github/workflows/test.yml

CHANGELOG.md

lzchen · 2021-09-14T16:50:07Z

util/opentelemetry-util-http/src/opentelemetry/util/http/httplib.py

+logger = logging.getLogger(__name__)
+
+
+class HttpClientInstrumentor(BaseInstrumentor):


I'm not too sure how this is supposed to be used by the user. Is this Instrumentor instantiated as well along side RequestsInstrumentor? Or is RequestsInstrumentor supposed to extend this class?

Oh, actually now that you say it, this was the reason why it was in a separate package...

This is not supposed to be used by the user at all. Instead it should just be loaded by the instrumentation infrastructure and then it does its thing behind the scentes. The requests and urllib3 instrumentations are the actual users of this code, since they register their wish to have the IP set onto a span which this instrumentation fulfills once the standard library httplib opens a socket.

I think I have to add an entrypoint to the setup.cfg of util-http as well as the other instrumentation boilerplate. Do you think it would be better to keep it as separate package?

I don't mind having an entrypoint in http libraries for this, unless you feel strongly having another package. I personally don't like having many packages :).

Do all client libraries have a "send" method? If not, maybe we can just have trysetip and set_ip_on_next_http_connection as utility functions, and decorate the request functions directly? How come we need to create a new instrumentor for this feature?

How come we need to create a new instrumentor for this feature?

Because this instruments the standard library httplib.client. I could put it in the urllib3 instrumentation, but if we ever want to instrument another library based on httplib.client (e.g. there is a urllib instrumentation here that could also be enhanched with this), then where should we put it if not in a separate, independent instrumentor?

Do all client libraries have a "send" method?

No, just (those based on) httplib.client, which applies to requests and urllib3 at least.

Even the tests have:

HttpClientInstrumentor().instrument() URLLib3Instrumentor().instrument()

Yes. The tests need to call HttpClientInstrumentor().instrument() manually since no distribution (is it called that?) is loaded that auto-enables the instrumentation. URLLib3Instrumentor().instrument() cannot call HttpClientInstrumentor().instrument() IIUC, as that would lead to double instrumentation. Also, I wanted to have the feature optional. If users disable the HttpClientInstrumentor (not sure if this is possible today), they won't get net.peer.ip and potential associated overhead and bugs.

Just confirming here. As a user, if I manually instrument code, and load the RequestsInstrumentor, I will also need to manually load the HttpClientInstrumentor myself correct?

Would it be possible to call trysetip from the requests/urlib3 code and skip the additional instrumentor? Or is the problem that those libraries don't have access to the underlying socket to pull IP information from it?

Yes, the main reason was that its not easy to get the underlying socket. I did not check for every library separately, in some it may be possible to use some undocumented properties or instrument some constructors to get to the socket. Instrumenting httplib.client, although also awkward, seemed like the most clean way.

Got it, that makes sense. Thanks!

@codeboten To follow up here: Is manually using the instrumentors common enough that it would make sense to add a hint to use HttpClientInstrumentor to the READMEs of the requests and urllib3 instrumentations?

lzchen · 2021-09-14T16:51:54Z

@Oberon00

It will add a dependency on asgiref though

Sorry I might be missing something. Where is the dependency for asgiref in this library?

Oberon00 · 2021-09-14T17:54:53Z

@lzchen

Sorry I might be missing something. Where is the dependency for asgiref in this library?

The code in this PR does not depend on asgiref, but the util-http package does.

lzchen · 2021-09-14T18:13:04Z

@Oberon00

The code in this PR does not depend on asgiref, but the util-http package does.

Am I looking at the wrong place? I don't think the util-http package has any dependencies.

Oberon00 · 2021-09-15T07:22:52Z

@lzchen You are right, seems I somehow looked in the wrong place 😄

Oberon00 · 2021-09-15T10:10:20Z

util/opentelemetry-util-http/setup.cfg

@@ -40,3 +40,7 @@ packages=find_namespace:

 [options.packages.find]
 where = src
+
+[options.entry_points]
+opentelemetry_instrumentor =


Compared to the original PR version with a separate package for this:
On the minus side, this now means that everyone depending on util-http will get the httplib.client instrumentation, even if they don't use it. The overhead should be small though. On the plus side, there is no new package.

The benefit of not having a new package outweighs the small overhead I feel :)

Oberon00 · 2021-09-15T15:40:12Z

util/opentelemetry-util-http/setup.cfg

@@ -40,3 +40,7 @@ packages=find_namespace:

 [options.packages.find]
 where = src
+
+[options.entry_points]
+opentelemetry_instrumentor =


BTW, is there some integration test that loads the urllib3 or requests instrumentation via the entrypoint, that I could extend to assert on the net.peer.ip attribute being present, so I could verify the wiring is not messed up here?

Just to clarify, you are wanting to test if attribute is present if auto-instrumentation is used? If so, I don't believe we have integration tests for those. We currently only have unit tests for auto-instrumentation.

Oberon00 · 2021-09-23T09:58:37Z

Is there anything I can do to move this forward?

codeboten

The code looks good, just one clarifying question regarding the flow of how the instrumentation is used.

codeboten · 2021-09-23T21:36:20Z

util/opentelemetry-util-http/src/opentelemetry/util/http/httplib.py

+logger = logging.getLogger(__name__)
+
+
+class HttpClientInstrumentor(BaseInstrumentor):


Just confirming here. As a user, if I manually instrument code, and load the RequestsInstrumentor, I will also need to manually load the HttpClientInstrumentor myself correct?

Would it be possible to call trysetip from the requests/urlib3 code and skip the additional instrumentor? Or is the problem that those libraries don't have access to the underlying socket to pull IP information from it?

codeboten

👍

codeboten · 2021-09-24T21:33:28Z

util/opentelemetry-util-http/src/opentelemetry/util/http/httplib.py

+logger = logging.getLogger(__name__)
+
+
+class HttpClientInstrumentor(BaseInstrumentor):


Got it, that makes sense. Thanks!

Oberon00 mentioned this pull request Sep 6, 2021

Add test base class with local HTTP server. open-telemetry/opentelemetry-python#2101

Merged

3 tasks

Oberon00 marked this pull request as ready for review September 6, 2021 14:15

Oberon00 requested a review from a team as a code owner September 6, 2021 14:15

Oberon00 added 9 commits September 14, 2021 14:47

Add instrumentation-http-base.

e436636

Add net.peer.ip in requests & urllib3 instrumentations.

8f551c9

Update core repo SHA (to PR branch).

f7cdf1c

Add CHANGELOG entry.

ce61d45

Fix tox -e lint installation order.

1e37e5d

tox -e generate.

d60536f

Update to newer core PR revision.

9b4db03

Merge instrumentation-http-base into util-http.

b02f1d3

Bump core version.

b0437e3

Oberon00 force-pushed the http-client-peer-ip branch from fa04708 to b0437e3 Compare September 14, 2021 12:49

Oberon00 added 2 commits September 14, 2021 14:52

Remove final newline as generate wants it.

11b4a32

Pylint doesn't notice #type: comment.

daf3c94

lzchen reviewed Sep 14, 2021

View reviewed changes

.github/workflows/test.yml Outdated Show resolved Hide resolved

lzchen reviewed Sep 14, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

lzchen reviewed Sep 14, 2021

View reviewed changes

Merge branch 'main' into http-client-peer-ip

6f91702

Oberon00 added 2 commits September 15, 2021 10:45

Fix CHANGELOG.

d1d3342

Add entrypoint.

bd83dc6

Oberon00 commented Sep 15, 2021

View reviewed changes

Update to main branch of core repo (from PR branch)

5532a3a

codeboten reviewed Sep 23, 2021

View reviewed changes

codeboten approved these changes Sep 24, 2021

View reviewed changes

Merge branch 'main' into http-client-peer-ip

4fce7b9

lzchen approved these changes Sep 27, 2021

View reviewed changes

lzchen merged commit efaa257 into open-telemetry:main Sep 27, 2021

Oberon00 deleted the http-client-peer-ip branch October 5, 2021 11:00

adamantike mentioned this pull request Oct 15, 2021

util-http breaks auto-instrumentation since v0.25b0 #744

Closed

owais mentioned this pull request Oct 15, 2021

discuss opentelemetry-util-http #748

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add net.peer.ip in requests & urllib3 instrumentations. #661

Add net.peer.ip in requests & urllib3 instrumentations. #661

Oberon00 commented Sep 6, 2021 •

edited

Oberon00 commented Sep 6, 2021

lzchen commented Sep 7, 2021

Oberon00 commented Sep 7, 2021

Oberon00 commented Sep 14, 2021

Oberon00 commented Sep 14, 2021

lzchen Sep 14, 2021

Oberon00 Sep 14, 2021

Oberon00 Sep 14, 2021

lzchen Sep 14, 2021 •

edited

Oberon00 Sep 15, 2021 •

edited

Oberon00 Sep 20, 2021 •

edited

codeboten Sep 23, 2021

Oberon00 Sep 24, 2021

codeboten Sep 24, 2021

Oberon00 Oct 5, 2021

lzchen commented Sep 14, 2021

Oberon00 commented Sep 14, 2021

lzchen commented Sep 14, 2021 •

edited

Oberon00 commented Sep 15, 2021

Oberon00 Sep 15, 2021

lzchen Sep 20, 2021

Oberon00 Sep 15, 2021

lzchen Sep 20, 2021

Oberon00 commented Sep 23, 2021

codeboten left a comment

codeboten Sep 23, 2021

codeboten left a comment

codeboten Sep 24, 2021

		logger = logging.getLogger(__name__)


		class HttpClientInstrumentor(BaseInstrumentor):

Add net.peer.ip in requests & urllib3 instrumentations. #661

Add net.peer.ip in requests & urllib3 instrumentations. #661

Conversation

Oberon00 commented Sep 6, 2021 • edited

Description

Type of change

How Has This Been Tested?

Does This PR Require a Core Repo Change?

Checklist:

Oberon00 commented Sep 6, 2021

lzchen commented Sep 7, 2021

Oberon00 commented Sep 7, 2021

Oberon00 commented Sep 14, 2021

Oberon00 commented Sep 14, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lzchen Sep 14, 2021 • edited

Choose a reason for hiding this comment

Oberon00 Sep 15, 2021 • edited

Choose a reason for hiding this comment

Oberon00 Sep 20, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lzchen commented Sep 14, 2021

Oberon00 commented Sep 14, 2021

lzchen commented Sep 14, 2021 • edited

Oberon00 commented Sep 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Oberon00 commented Sep 23, 2021

codeboten left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codeboten left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Oberon00 commented Sep 6, 2021 •

edited

lzchen Sep 14, 2021 •

edited

Oberon00 Sep 15, 2021 •

edited

Oberon00 Sep 20, 2021 •

edited

lzchen commented Sep 14, 2021 •

edited