diff --git a/CHANGELOG.md b/CHANGELOG.md index 4996f9b9f3..80035f1db0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ([#1402](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1402)) - Add support for py3.11 ([#1415](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1415)) +- `opentelemetry-instrumentation-django` Add support for regular expression matching and sanitization of HTTP headers. + ([#1411](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1411)) - `opentelemetry-instrumentation-falcon` Add support for regular expression matching and sanitization of HTTP headers. ([#1412](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1412)) - `opentelemetry-instrumentation-flask` Add support for regular expression matching and sanitization of HTTP headers. diff --git a/instrumentation/opentelemetry-instrumentation-django/src/opentelemetry/instrumentation/django/__init__.py b/instrumentation/opentelemetry-instrumentation-django/src/opentelemetry/instrumentation/django/__init__.py index 7fdbc92eb8..ea2ba8598b 100644 --- a/instrumentation/opentelemetry-instrumentation-django/src/opentelemetry/instrumentation/django/__init__.py +++ b/instrumentation/opentelemetry-instrumentation-django/src/opentelemetry/instrumentation/django/__init__.py @@ -94,8 +94,9 @@ Exclude lists ************* -To exclude certain URLs from being tracked, set the environment variable ``OTEL_PYTHON_DJANGO_EXCLUDED_URLS`` -(or ``OTEL_PYTHON_EXCLUDED_URLS`` as fallback) with comma delimited regexes representing which URLs to exclude. +To exclude certain URLs from tracking, set the environment variable ``OTEL_PYTHON_DJANGO_EXCLUDED_URLS`` +(or ``OTEL_PYTHON_EXCLUDED_URLS`` to cover all instrumentations) to a string of comma delimited regexes that match the +URLs. For example, @@ -107,8 +108,8 @@ Request attributes ******************** -To extract certain attributes from Django's request object and use them as span attributes, set the environment variable ``OTEL_PYTHON_DJANGO_TRACED_REQUEST_ATTRS`` to a comma -delimited list of request attribute names. +To extract attributes from Django's request object and use them as span attributes, set the environment variable +``OTEL_PYTHON_DJANGO_TRACED_REQUEST_ATTRS`` to a comma delimited list of request attribute names. For example, @@ -116,14 +117,15 @@ export OTEL_PYTHON_DJANGO_TRACED_REQUEST_ATTRS='path_info,content_type' -will extract path_info and content_type attributes from every traced request and add them as span attritbues. +will extract the ``path_info`` and ``content_type`` attributes from every traced request and add them as span attributes. Django Request object reference: https://docs.djangoproject.com/en/3.1/ref/request-response/#attributes Request and Response hooks *************************** -The instrumentation supports specifying request and response hooks. These are functions that get called back by the instrumentation right after a Span is created for a request -and right before the span is finished while processing a response. The hooks can be configured as follows: +This instrumentation supports request and response hooks. These are functions that get called +right after a span is created for a request and right before the span is finished for the response. +The hooks can be configured as follows: .. code:: python @@ -140,50 +142,94 @@ def response_hook(span, request, response): Capture HTTP request and response headers ***************************************** -You can configure the agent to capture predefined HTTP headers as span attributes, according to the `semantic convention `_. +You can configure the agent to capture specified HTTP headers as span attributes, according to the +`semantic convention `_. Request headers *************** -To capture predefined HTTP request headers as span attributes, set the environment variable ``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST`` -to a comma-separated list of HTTP header names. +To capture HTTP request headers as span attributes, set the environment variable +``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST`` to a comma delimited list of HTTP header names. For example, :: - export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST="content_type,custom_request_header" + export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST="content-type,custom_request_header" -will extract content_type and custom_request_header from request headers and add them as span attributes. +will extract ``content-type`` and ``custom_request_header`` from the request headers and add them as span attributes. -It is recommended that you should give the correct names of the headers to be captured in the environment variable. -Request header names in django are case insensitive. So, giving header name as ``CUStom_Header`` in environment variable will be able capture header with name ``custom-header``. +Request header names in Django are case-insensitive. So, giving the header name as ``CUStom-Header`` in the environment +variable will capture the header named ``custom-header``. -The name of the added span attribute will follow the format ``http.request.header.`` where ```` being the normalized HTTP header name (lowercase, with - characters replaced by _ ). -The value of the attribute will be single item list containing all the header values. +Regular expressions may also be used to match multiple headers that correspond to the given pattern. For example: +:: + + export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST="Accept.*,X-.*" + +Would match all request headers that start with ``Accept`` and ``X-``. + +To capture all request headers, set ``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST`` to ``".*"``. +:: + + export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST=".*" -Example of the added span attribute, +The name of the added span attribute will follow the format ``http.request.header.`` where ```` +is the normalized HTTP header name (lowercase, with ``-`` replaced by ``_``). The value of the attribute will be a +single item list containing all the header values. + +For example: ``http.request.header.custom_request_header = [","]`` Response headers **************** -To capture predefined HTTP response headers as span attributes, set the environment variable ``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE`` -to a comma-separated list of HTTP header names. +To capture HTTP response headers as span attributes, set the environment variable +``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE`` to a comma delimited list of HTTP header names. For example, :: - export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE="content_type,custom_response_header" + export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE="content-type,custom_response_header" + +will extract ``content-type`` and ``custom_response_header`` from the response headers and add them as span attributes. -will extract content_type and custom_response_header from response headers and add them as span attributes. +Response header names in Django are case-insensitive. So, giving the header name as ``CUStom-Header`` in the environment +variable will capture the header named ``custom-header``. -It is recommended that you should give the correct names of the headers to be captured in the environment variable. -Response header names captured in django are case insensitive. So, giving header name as ``CUStomHeader`` in environment variable will be able capture header with name ``customheader``. +Regular expressions may also be used to match multiple headers that correspond to the given pattern. For example: +:: -The name of the added span attribute will follow the format ``http.response.header.`` where ```` being the normalized HTTP header name (lowercase, with - characters replaced by _ ). -The value of the attribute will be single item list containing all the header values. + export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE="Content.*,X-.*" -Example of the added span attribute, +Would match all response headers that start with ``Content`` and ``X-``. + +To capture all response headers, set ``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE`` to ``".*"``. +:: + + export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE=".*" + +The name of the added span attribute will follow the format ``http.response.header.`` where ```` +is the normalized HTTP header name (lowercase, with ``-`` replaced by ``_``). The value of the attribute will be a +single item list containing all the header values. + +For example: ``http.response.header.custom_response_header = [","]`` +Sanitizing headers +****************** +In order to prevent storing sensitive data such as personally identifiable information (PII), session keys, passwords, +etc, set the environment variable ``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS`` +to a comma delimited list of HTTP header names to be sanitized. Regexes may be used, and all header names will be +matched in a case-insensitive manner. + +For example, +:: + + export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS=".*session.*,set-cookie" + +will replace the value of headers such as ``session-id`` and ``set-cookie`` with ``[REDACTED]`` in the span. + +Note: + The environment variable names used to capture HTTP headers are still experimental, and thus are subject to change. + API --- diff --git a/instrumentation/opentelemetry-instrumentation-django/tests/test_middleware.py b/instrumentation/opentelemetry-instrumentation-django/tests/test_middleware.py index 8b584bfc3b..3845c55dcb 100644 --- a/instrumentation/opentelemetry-instrumentation-django/tests/test_middleware.py +++ b/instrumentation/opentelemetry-instrumentation-django/tests/test_middleware.py @@ -48,6 +48,7 @@ format_trace_id, ) from opentelemetry.util.http import ( + OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS, OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST, OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE, _active_requests_count_attrs, @@ -530,6 +531,14 @@ def test_django_with_wsgi_instrumented(self): ) +@patch.dict( + "os.environ", + { + OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS: ".*my-secret.*", + OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST: "Custom-Test-Header-1,Custom-Test-Header-2,Custom-Test-Header-3,Regex-Test-Header-.*,Regex-Invalid-Test-Header-.*,.*my-secret.*", + OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE: "Custom-Test-Header-1,Custom-Test-Header-2,Custom-Test-Header-3,my-custom-regex-header-.*,invalid-regex-header-.*,.*my-secret.*", + }, +) class TestMiddlewareWsgiWithCustomHeaders(WsgiTestBase): @classmethod def setUpClass(cls): @@ -542,18 +551,9 @@ def setUp(self): tracer_provider, exporter = self.create_tracer_provider() self.exporter = exporter _django_instrumentor.instrument(tracer_provider=tracer_provider) - self.env_patch = patch.dict( - "os.environ", - { - OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST: "Custom-Test-Header-1,Custom-Test-Header-2,Custom-Test-Header-3", - OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE: "Custom-Test-Header-1,Custom-Test-Header-2,Custom-Test-Header-3", - }, - ) - self.env_patch.start() def tearDown(self): super().tearDown() - self.env_patch.stop() teardown_test_environment() _django_instrumentor.uninstrument() @@ -570,10 +570,18 @@ def test_http_custom_request_headers_in_span_attributes(self): "http.request.header.custom_test_header_2": ( "test-header-value-2", ), + "http.request.header.regex_test_header_1": ("Regex Test Value 1",), + "http.request.header.regex_test_header_2": ( + "RegexTestValue2,RegexTestValue3", + ), + "http.request.header.my_secret_header": ("[REDACTED]",), } Client( HTTP_CUSTOM_TEST_HEADER_1="test-header-value-1", HTTP_CUSTOM_TEST_HEADER_2="test-header-value-2", + HTTP_REGEX_TEST_HEADER_1="Regex Test Value 1", + HTTP_REGEX_TEST_HEADER_2="RegexTestValue2,RegexTestValue3", + HTTP_MY_SECRET_HEADER="My Secret Value", ).get("/traced/") spans = self.exporter.get_finished_spans() self.assertEqual(len(spans), 1) @@ -607,6 +615,13 @@ def test_http_custom_response_headers_in_span_attributes(self): "http.response.header.custom_test_header_2": ( "test-header-value-2", ), + "http.response.header.my_custom_regex_header_1": ( + "my-custom-regex-value-1,my-custom-regex-value-2", + ), + "http.response.header.my_custom_regex_header_2": ( + "my-custom-regex-value-3,my-custom-regex-value-4", + ), + "http.response.header.my_secret_header": ("[REDACTED]",), } Client().get("/traced_custom_header/") spans = self.exporter.get_finished_spans() diff --git a/instrumentation/opentelemetry-instrumentation-django/tests/test_middleware_asgi.py b/instrumentation/opentelemetry-instrumentation-django/tests/test_middleware_asgi.py index 941fda49bb..784f8e24ec 100644 --- a/instrumentation/opentelemetry-instrumentation-django/tests/test_middleware_asgi.py +++ b/instrumentation/opentelemetry-instrumentation-django/tests/test_middleware_asgi.py @@ -43,6 +43,7 @@ format_trace_id, ) from opentelemetry.util.http import ( + OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS, OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST, OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE, get_excluded_urls, @@ -424,6 +425,14 @@ async def test_tracer_provider_traced(self): ) +@patch.dict( + "os.environ", + { + OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS: ".*my-secret.*", + OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST: "Custom-Test-Header-1,Custom-Test-Header-2,Custom-Test-Header-3,Regex-Test-Header-.*,Regex-Invalid-Test-Header-.*,.*my-secret.*", + OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE: "Custom-Test-Header-1,Custom-Test-Header-2,Custom-Test-Header-3,my-custom-regex-header-.*,invalid-regex-header-.*,.*my-secret.*", + }, +) class TestMiddlewareAsgiWithCustomHeaders(SimpleTestCase, TestBase): @classmethod def setUpClass(cls): @@ -437,18 +446,9 @@ def setUp(self): tracer_provider, exporter = self.create_tracer_provider() self.exporter = exporter _django_instrumentor.instrument(tracer_provider=tracer_provider) - self.env_patch = patch.dict( - "os.environ", - { - OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST: "Custom-Test-Header-1,Custom-Test-Header-2,Custom-Test-Header-3", - OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE: "Custom-Test-Header-1,Custom-Test-Header-2,Custom-Test-Header-3", - }, - ) - self.env_patch.start() def tearDown(self): super().tearDown() - self.env_patch.stop() teardown_test_environment() _django_instrumentor.uninstrument() @@ -465,12 +465,20 @@ async def test_http_custom_request_headers_in_span_attributes(self): "http.request.header.custom_test_header_2": ( "test-header-value-2", ), + "http.request.header.regex_test_header_1": ("Regex Test Value 1",), + "http.request.header.regex_test_header_2": ( + "RegexTestValue2,RegexTestValue3", + ), + "http.request.header.my_secret_header": ("[REDACTED]",), } await self.async_client.get( "/traced/", **{ "custom-test-header-1": "test-header-value-1", "custom-test-header-2": "test-header-value-2", + "Regex-Test-Header-1": "Regex Test Value 1", + "regex-test-header-2": "RegexTestValue2,RegexTestValue3", + "My-Secret-Header": "My Secret Value", }, ) spans = self.exporter.get_finished_spans() @@ -510,6 +518,13 @@ async def test_http_custom_response_headers_in_span_attributes(self): "http.response.header.custom_test_header_2": ( "test-header-value-2", ), + "http.response.header.my_custom_regex_header_1": ( + "my-custom-regex-value-1,my-custom-regex-value-2", + ), + "http.response.header.my_custom_regex_header_2": ( + "my-custom-regex-value-3,my-custom-regex-value-4", + ), + "http.response.header.my_secret_header": ("[REDACTED]",), } await self.async_client.get("/traced_custom_header/") spans = self.exporter.get_finished_spans() diff --git a/instrumentation/opentelemetry-instrumentation-django/tests/views.py b/instrumentation/opentelemetry-instrumentation-django/tests/views.py index f97933cfd8..452a7c0fdd 100644 --- a/instrumentation/opentelemetry-instrumentation-django/tests/views.py +++ b/instrumentation/opentelemetry-instrumentation-django/tests/views.py @@ -35,6 +35,13 @@ def response_with_custom_header(request): response = HttpResponse() response["custom-test-header-1"] = "test-header-value-1" response["custom-test-header-2"] = "test-header-value-2" + response[ + "my-custom-regex-header-1" + ] = "my-custom-regex-value-1,my-custom-regex-value-2" + response[ + "my-custom-regex-header-2" + ] = "my-custom-regex-value-3,my-custom-regex-value-4" + response["my-secret-header"] = "my-secret-value" return response