-
Notifications
You must be signed in to change notification settings - Fork 96
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AIOBotocore instrumentation #1135
Merged
Merged
Changes from 10 commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
1d029e7
Instrument aiobotocore
lrafeei 8d24725
Replace __version__ in flask instrumentation to avoid deprecation
lrafeei 9ccf17b
Merge branch 'main' into aiobotocore-instrumentation
lrafeei 95a3338
Merge branch 'main' into aiobotocore-instrumentation
lrafeei cbbb923
Merge branch 'main' into aiobotocore-instrumentation
mergify[bot] 028f9bd
Merge branch 'main' into aiobotocore-instrumentation
mergify[bot] 9a90104
Disable browser monitoring
lrafeei da6b566
Fix typo
lrafeei bb75e16
Disable browser monitoring with aiobotocore
lrafeei 95cf918
Fix linter errors
lrafeei 4c19af1
Revert to disabling settings in conftest
lrafeei d9b86d8
Merge branch 'main' into aiobotocore-instrumentation
mergify[bot] f956c29
Remove browser monitoring disabling flag
lrafeei 8a08032
Merge branch 'main' into aiobotocore-instrumentation
lrafeei b160a9c
Merge branch 'main' into aiobotocore-instrumentation
lrafeei File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
# Copyright 2010 New Relic, Inc. | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
from newrelic.api.external_trace import ExternalTrace | ||
from newrelic.common.object_wrapper import wrap_function_wrapper | ||
|
||
|
||
def _bind_make_request_params(operation_model, request_dict, *args, **kwargs): | ||
return operation_model, request_dict | ||
|
||
|
||
def bind__send_request(request_dict, operation_model, *args, **kwargs): | ||
return operation_model, request_dict | ||
|
||
|
||
async def wrap_endpoint_make_request(wrapped, instance, args, kwargs): | ||
operation_model, request_dict = _bind_make_request_params(*args, **kwargs) | ||
url = request_dict.get("url") | ||
method = request_dict.get("method") | ||
|
||
with ExternalTrace(library="aiobotocore", url=url, method=method, source=wrapped) as trace: | ||
try: | ||
# Because AIOBotocore's proxy functionality uses aiohttp | ||
# and urllib3 under the hood, New Relic has portions that | ||
# are classified as Web Transactions. This means that | ||
# browser monitoring will now be true. However, this will | ||
# inject unwanted JS Agent Header Fragments into SQS responses. | ||
trace.settings.browser_monitoring.enabled = False | ||
trace._add_agent_attribute("aws.operation", operation_model.name) | ||
except: | ||
pass | ||
|
||
result = await wrapped(*args, **kwargs) | ||
try: | ||
request_id = result[1]["ResponseMetadata"]["RequestId"] | ||
trace._add_agent_attribute("aws.requestId", request_id) | ||
except: | ||
pass | ||
return result | ||
|
||
|
||
def instrument_aiobotocore_endpoint(module): | ||
wrap_function_wrapper(module, "AioEndpoint.make_request", wrap_endpoint_make_request) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,151 @@ | ||
# Copyright 2010 New Relic, Inc. | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
import functools | ||
import logging | ||
import socket | ||
import threading | ||
|
||
import moto.server | ||
import werkzeug.serving | ||
from testing_support.fixture.event_loop import ( # noqa: F401, pylint: disable=W0611 | ||
event_loop as loop, | ||
) | ||
from testing_support.fixtures import ( # noqa: F401, pylint: disable=W0611 | ||
collector_agent_registration_fixture, | ||
collector_available_fixture, | ||
) | ||
|
||
PORT = 4443 | ||
AWS_ACCESS_KEY_ID = "AAAAAAAAAAAACCESSKEY" | ||
AWS_SECRET_ACCESS_KEY = "AAAAAASECRETKEY" # nosec | ||
HOST = "127.0.0.1" | ||
|
||
|
||
_default_settings = { | ||
"transaction_tracer.explain_threshold": 0.0, | ||
"transaction_tracer.transaction_threshold": 0.0, | ||
"transaction_tracer.stack_trace_threshold": 0.0, | ||
"debug.log_data_collector_payloads": True, | ||
"debug.record_transaction_failure": True, | ||
} | ||
collector_agent_registration = collector_agent_registration_fixture( | ||
app_name="Python Agent Test (external_aiobotocore)", | ||
default_settings=_default_settings, | ||
linked_applications=["Python Agent Test (external_aiobotocore)"], | ||
) | ||
|
||
|
||
def get_free_tcp_port(release_socket: bool = False): | ||
sckt = socket.socket(socket.AF_INET, socket.SOCK_STREAM) | ||
sckt.bind((HOST, 0)) | ||
_, port = sckt.getsockname() # address, port | ||
if release_socket: | ||
sckt.close() | ||
return port | ||
|
||
return sckt, port | ||
|
||
|
||
class MotoService: | ||
"""Will Create MotoService. | ||
Service is ref-counted so there will only be one per process. Real Service will | ||
be returned by `__aenter__`.""" | ||
|
||
_services = {} # {name: instance} | ||
|
||
def __init__(self, service_name: str, port: int = None, ssl: bool = False): | ||
self._service_name = service_name | ||
|
||
if port: | ||
self._socket = None | ||
self._port = port | ||
else: | ||
self._socket, self._port = get_free_tcp_port() | ||
|
||
self._thread = None | ||
self._logger = logging.getLogger("MotoService") | ||
self._refcount = None | ||
self._ip_address = HOST | ||
self._server = None | ||
self._ssl_ctx = werkzeug.serving.generate_adhoc_ssl_context() if ssl else None | ||
self._schema = "http" if not self._ssl_ctx else "https" | ||
|
||
@property | ||
def endpoint_url(self): | ||
return f"{self._schema}://{self._ip_address}:{self._port}" | ||
|
||
def __call__(self, func): | ||
async def wrapper(*args, **kwargs): | ||
await self._start() | ||
try: | ||
result = await func(*args, **kwargs) | ||
finally: | ||
await self._stop() | ||
return result | ||
|
||
functools.update_wrapper(wrapper, func) | ||
wrapper.__wrapped__ = func | ||
return wrapper | ||
|
||
async def __aenter__(self): | ||
svc = self._services.get(self._service_name) | ||
if svc is None: | ||
self._services[self._service_name] = self | ||
self._refcount = 1 | ||
await self._start() | ||
return self | ||
else: | ||
svc._refcount += 1 | ||
return svc | ||
|
||
async def __aexit__(self, exc_type, exc_val, exc_tb): | ||
self._refcount -= 1 | ||
|
||
if self._socket: | ||
self._socket.close() | ||
self._socket = None | ||
|
||
if self._refcount == 0: | ||
del self._services[self._service_name] | ||
await self._stop() | ||
|
||
def _server_entry(self): | ||
self._main_app = moto.server.DomainDispatcherApplication( | ||
moto.server.create_backend_app # , service=self._service_name | ||
) | ||
self._main_app.debug = True | ||
|
||
if self._socket: | ||
self._socket.close() # release right before we use it | ||
self._socket = None | ||
|
||
self._server = werkzeug.serving.make_server( | ||
self._ip_address, | ||
self._port, | ||
self._main_app, | ||
True, | ||
ssl_context=self._ssl_ctx, | ||
) | ||
self._server.serve_forever() | ||
|
||
async def _start(self): | ||
self._thread = threading.Thread(target=self._server_entry, daemon=True) | ||
self._thread.start() | ||
|
||
async def _stop(self): | ||
if self._server: | ||
self._server.shutdown() | ||
|
||
self._thread.join() |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this something you can do, the settings object is not a copy but just a reference to the application settings. You're permanently disabling browser monitoring everywhere in this application.
Browser monitoring shouldn't be at all relevant to an ExternalTrace anyway. The issue you're likely having is the mock server having issues with it being an aiohttp server. Why don't you try just disabling browser monitoring in
conftest
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also for the record, the correct way to disable browser monitoring for an individual transaction is
newrelic.agent.disable_browser_autorum()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I kind of went back and forth with this. I originally had it disabled in conftest but my rationale here was that, if a customer wanted to use the proxy functionality of aiobotocore SQS, the customer would have to disable the setting themselves just because of how the proxy server in aiobotocore works. If we are OK with this, I can put something in the docs about this as well.