Add posibility to define excludes from logging

README.md

@@ -1,22 +1,26 @@
 # sag_py_web_common
+
 [![Maintainability][codeclimate-image]][codeclimate-url]
 [![Coverage Status][coveralls-image]][coveralls-url]
 [![Known Vulnerabilities][snyk-image]][snyk-url]
 
 This contains samhammer specific and internally used helper functions for web projects.
 
 Requirements for code to be added here:
+
 - It's sag/project specific and not of general/public use, so that it does not make sense to create a individual lib
 - It's nothing private either, that should not be publically on the internet
 - It has no, or very little dependencies
-   (because the deps are in all projects using the lib, even if the feaute isn't required)
+  (because the deps are in all projects using the lib, even if the feaute isn't required)
 
 Note: See this as last option and try to create individual libs as much as possible.
 
 ### Installation
+
 pip install sag-py-web-common
 
 ## How to use
+
 ### Default routing
 
 All requests to the main route / are redirected to /swagger if nothing specified.
@@ -37,8 +41,12 @@ app.include_router(build_default_route(ingress_base_path=config.ingress_base_pat
 Extends the asgi-logger and adds a log entry for received requests.
 Furthermore the requests are filtered, so that health checks don't spam the logs.
 
-For requests to be filtered, they need to have the header "healthcheck" with one of these values:
-"livenessprobe", "readinessprobe", "startupprobe", "prtg"
+Requests can be filtered via one of two ways:
+
+- via the optional parameter "excluded_paths": Simply append all paths that should be ignored, separated by comma.
+- via the optional parameter "exclude_header": This one will require you to define a name for the header (f.ex. 'myHeaderExclude'), and then also send this defined header as an HTTP Header with your requests.
+
+This filter will apply to substrings, as well since the filter is using a contains-search.
 
 ```python
 from sag_py_web_common.filtered_access_logger import FilteredAccessLoggerMiddleware
@@ -49,6 +57,8 @@ app.add_middleware(
     FilteredAccessLoggerMiddleware,
     format="Completed: %(R)s - %(st)s - %(L)s",
     logger=logging.getLogger("access"),
+    excluded_paths=["pathPart/partOne", "pathPart/partTwo"], # optional
+    exclude_header="myHeaderExclude" # optional
 )
 ```
 
@@ -66,6 +76,7 @@ from sag_py_web_common.json_exception_handler import handle_unknown_exception
 
 app.add_exception_handler(Exception, handle_unknown_exception)
 ```
+
 For logging any HHTP-Exception use the **log_exception** function.
 
 ```python
@@ -85,35 +96,36 @@ Just install vscode with dev containers extension. All required extensions and c
 
 ### With pycharm
 
-* Install latest pycharm
-* Install pycharm plugin BlackConnect
-* Install pycharm plugin Mypy
-* Configure the python interpreter/venv
-* pip install requirements-dev.txt
-* pip install black[d]
-* Ctl+Alt+S => Check Tools => BlackConnect => Trigger when saving changed files
-* Ctl+Alt+S => Check Tools => BlackConnect => Trigger on code reformat
-* Ctl+Alt+S => Click Tools => BlackConnect => "Load from pyproject.yaml" (ensure line length is 120)
-* Ctl+Alt+S => Click Tools => BlackConnect => Configure path to the blackd.exe at the "local instance" config (e.g. C:\Python310\Scripts\blackd.exe)
-* Ctl+Alt+S => Click Tools => Actions on save => Reformat code
-* Restart pycharm
+- Install latest pycharm
+- Install pycharm plugin BlackConnect
+- Install pycharm plugin Mypy
+- Configure the python interpreter/venv
+- pip install requirements-dev.txt
+- pip install black[d]
+- Ctl+Alt+S => Check Tools => BlackConnect => Trigger when saving changed files
+- Ctl+Alt+S => Check Tools => BlackConnect => Trigger on code reformat
+- Ctl+Alt+S => Click Tools => BlackConnect => "Load from pyproject.yaml" (ensure line length is 120)
+- Ctl+Alt+S => Click Tools => BlackConnect => Configure path to the blackd.exe at the "local instance" config (e.g. C:\Python310\Scripts\blackd.exe)
+- Ctl+Alt+S => Click Tools => Actions on save => Reformat code
+- Restart pycharm
 
 ## How to publish
-* Update the version in setup.py and commit your change
-* Create a tag with the same version number
-* Let github do the rest
+
+- Update the version in setup.py and commit your change
+- Create a tag with the same version number
+- Let github do the rest
 
 ## How to test
 
 To avoid publishing to pypi unnecessarily you can do as follows
 
-* Tag your branch however you like
-* Use the chosen tag in the requirements.txt-file of the project you want to test this library in, eg. `sag_py_web_common==<your tag>`
-* Rebuild/redeploy your project
+- Tag your branch however you like
+- Use the chosen tag in the requirements.txt-file of the project you want to test this library in, eg. `sag_py_web_common==<your tag>`
+- Rebuild/redeploy your project
 
-[codeclimate-image]:https://api.codeclimate.com/v1/badges/533686a1f4d644151adb/maintainability
-[codeclimate-url]:https://codeclimate.com/github/SamhammerAG/sag_py_web_common/maintainability
-[coveralls-image]:https://coveralls.io/repos/github/SamhammerAG/sag_py_web_common/badge.svg?branch=master
-[coveralls-url]:https://coveralls.io/github/SamhammerAG/sag_py_web_common?branch=master
-[snyk-image]:https://snyk.io/test/github/SamhammerAG/sag_py_web_common/badge.svg
-[snyk-url]:https://snyk.io/test/github/SamhammerAG/sag_py_web_common
+[codeclimate-image]: https://api.codeclimate.com/v1/badges/533686a1f4d644151adb/maintainability
+[codeclimate-url]: https://codeclimate.com/github/SamhammerAG/sag_py_web_common/maintainability
+[coveralls-image]: https://coveralls.io/repos/github/SamhammerAG/sag_py_web_common/badge.svg?branch=master
+[coveralls-url]: https://coveralls.io/github/SamhammerAG/sag_py_web_common?branch=master
+[snyk-image]: https://snyk.io/test/github/SamhammerAG/sag_py_web_common/badge.svg
+[snyk-url]: https://snyk.io/test/github/SamhammerAG/sag_py_web_common

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "sag-py-web-common"
-version = "1.0.1"
+version = "1.0.2"
 description = "Small helper functions for web projects"
 authors = ["Samhammer AG"]
 license = "MIT"

sag_py_web_common/filtered_access_logger.py

@@ -1,13 +1,30 @@
+import logging
+from typing import List, Union
+
 from asgi_logger.middleware import AccessInfo, AccessLogAtoms, AccessLoggerMiddleware
-from asgiref.typing import ASGIReceiveCallable, ASGISendCallable, HTTPScope
+from asgiref.typing import ASGI3Application, ASGIReceiveCallable, ASGISendCallable, HTTPScope
 
 
 class FilteredAccessLoggerMiddleware(AccessLoggerMiddleware):
     """The lib asgi-logger wrapped to exclude prtg and health checks from being logged
     Furthermore it adds logging of the incoming requests
     """
 
-    async def __call__(self, scope: HTTPScope, receive: ASGIReceiveCallable, send: ASGISendCallable) -> None:
+    def __init__(
+        self,
+        app: ASGI3Application,
+        format: Union[str, None],
+        logger: Union[logging.Logger, None],
+        excluded_paths: Union[List[str], None],
+        exclude_header: Union[str, None],
+    ) -> None:
+        super().__init__(app, format, logger)
+        self.excluded_paths = excluded_paths or []
+        self.exclude_header = exclude_header.strip().lower() if exclude_header else ""
+
+    async def __call__(
+        self, scope: HTTPScope, receive: ASGIReceiveCallable, send: ASGISendCallable
+    ) -> None:  # pragma: no cover
         if self._should_log(scope):
             self.logger.info("Received: %s %s", scope["method"], scope["path"])
 
@@ -22,13 +39,29 @@ def log(self, scope: HTTPScope, info: AccessInfo) -> None:
                 self.logger.warning(self.format, AccessLogAtoms(scope, info), extra=extra_args)
 
     def _should_log(self, scope: HTTPScope) -> bool:
-        return scope["type"] == "http" and not self._has_health_check_header(scope)
-
-    def _has_health_check_header(self, scope: HTTPScope) -> bool:
-        header_dict: dict[bytes, bytes] = dict(scope["headers"])
-        return b"healthcheck" in header_dict and header_dict[b"healthcheck"] in {
-            b"livenessprobe",
-            b"readinessprobe",
-            b"startupprobe",
-            b"prtg",
-        }
+        return (
+            scope["type"] == "http"
+            and not FilteredAccessLoggerMiddleware._is_excluded_via_path(scope, self.excluded_paths)
+            and not FilteredAccessLoggerMiddleware._is_excluded_via_header(scope, self.exclude_header)
+        )
+
+    @staticmethod
+    def _is_excluded_via_path(scope: HTTPScope, excluded_paths: List[str]) -> bool:
+        if not excluded_paths:
+            return False
+
+        path: str = str(scope["path"])
+        return any(excluded in path for excluded in excluded_paths)
+
+    @staticmethod
+    def _is_excluded_via_header(scope: HTTPScope, exclude_header: str) -> bool:
+        if not exclude_header:
+            return False
+
+        headers = scope.get("headers", [])
+        target_header_bytes = exclude_header.encode("latin-1").lower()
+
+        for header_key, _ in headers:
+            if header_key.lower() == target_header_bytes:
+                return True
+        return False

setup.py

@@ -11,7 +11,7 @@
 
 setuptools.setup(
     name="sag-py-web-common",
-    version="1.0.1",
+    version="1.0.2",
     description="Small helper functions for web projects",
     long_description=LONG_DESCRIPTION,
     long_description_content_type="text/markdown",