Skip to content
This repository was archived by the owner on Jun 13, 2023. It is now read-only.

feat(urrlib3): collecting payload#356

Merged
henperez merged 13 commits intomasterfrom
feat/collect-urllib3-payload-EP-0000
Jun 3, 2021
Merged

feat(urrlib3): collecting payload#356
henperez merged 13 commits intomasterfrom
feat/collect-urllib3-payload-EP-0000

Conversation

@henperez
Copy link
Copy Markdown
Contributor

@henperez henperez commented Jun 1, 2021

No description provided.

@henperez henperez requested a review from a team as a code owner June 1, 2021 17:03
@henperez henperez requested a review from ophiryael June 1, 2021 17:03
@ophiryael ophiryael requested review from adavidai and removed request for ophiryael June 2, 2021 07:58
Comment thread epsagon/events/urllib3.py Outdated

if not is_payload_collection_blacklisted(full_url):
if (
response.tell() > 0 and
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the headers are still useful even if the response is empty, WDYT?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch.

Comment thread tests/test_utils.py Outdated
:return: None
"""

original = epsagon.http_filters.BLACKLIST_URLS # Storing old state
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extract to pytest.fixture like:

original = epsagon.http_filters.BLACKLIST_URLS
yield
epsagon.http_filters.BLACKLIST_URLS = original

(instead of making it a part of this specific test)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment thread tests/modules/test_urllib3.py Outdated
}

http = urllib3.PoolManager()
return http.request('POST', TEST_URL, headers=headers, body='', preload_content=True)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. POST? (was expecting a GET)
  2. What happens if preload_content=False?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Fixed
  2. Added another test

Comment thread tests/modules/test_urllib3.py Outdated
http = urllib3.PoolManager()
conn = http.connection_from_url(TEST_URL)
urllib_response = conn.urlopen(
method='POST',
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GET?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe better to test both cause the body may be handled differently

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Comment thread tests/modules/test_urllib3.py Outdated

response = wrapped_function()
data = response.read()
# In this case data will not be available in the buffer
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a valid behavior (does it behave the same way without the Epsagon wrapper)?
I suggest a different test - can we run the same request twice, once with Epsagon wrapper and once without it, and compare the response objects? (same for the conn.urlopen test)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@pull-request-size pull-request-size Bot added size/L and removed size/M labels Jun 2, 2021
Comment thread tests/modules/test_urllib3.py Outdated
urllib3_event = trace_transport.last_trace.events[-1]
# Payload will be collected and stored inside the event data
assert(len(urllib3_event.resource['metadata']['response_body']) > 0)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assert (response.data == wrapped_data)
Should we also check the peek attribute of the responses to make sure they are the same?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can't do that since the data is different

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sure we can find a site for testing that returns the same data each time.
Such a test would be very powerful, and would make sure the original response is not harmed by the instrumentation

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. Done

Comment thread tests/modules/test_urllib3.py Outdated
assert(len(urllib3_event.resource['metadata']['response_body']) > 0)


def test_data_capture_with_pool_manager_pc_false(trace_transport):
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be a parameterized test with a preload_content arg:

@pytest.mark.parametrize("preload_content", [true, false])
def test_data_capture_with_pool_manager(preload_content):

Instead of two different tests, no?
(and compare the wrapped and non wrapped .read() as well as .data

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll fix this

@henperez henperez merged commit b5a56c3 into master Jun 3, 2021
@ranrib
Copy link
Copy Markdown
Contributor

ranrib commented Jun 3, 2021

🎉 This PR is included in version 1.68.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

@ranrib ranrib added the released label Jun 3, 2021
Alon-Katz pushed a commit that referenced this pull request Jul 22, 2021
Alon-Katz pushed a commit that referenced this pull request Jul 22, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants