refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only log message #996
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…gainst JSON, now only on log Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
ViBiOh
changed the title
… done against JSON, now only on log Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
ge0Aja
reviewed
Sep 26, 2025
| logs_to_forward = filter_logs( | ||
| logs_to_forward, INCLUDE_AT_MATCH, EXCLUDE_AT_MATCH | ||
| ) | ||
| if matcher.match(evaluated_log): |
There was a problem hiding this comment.
We should properly catch the exception raised by the matcher here and maybe log a warn?
There was a problem hiding this comment.
The previous behavior was to catch the exception and raise a new one. I think it's better to crash in this case. Otherwise Lambda will "succeed" but with bad filtering which is hidding a problem.
ge0Aja
approved these changes
Sep 26, 2025
ViBiOh
added a commit
that referenced
this pull request
Oct 2, 2025
…gainst JSON, now only log message (#996) * refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
ViBiOh
added a commit
that referenced
this pull request
Oct 6, 2025
…gainst JSON, now only log message (#996) * refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
ViBiOh
added a commit
that referenced
this pull request
Oct 8, 2025
…gainst JSON, now only log message (#996) * refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
ViBiOh
added a commit
that referenced
this pull request
Oct 23, 2025
* refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only log message (#996) * refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * feat(aws)!: Drop deprecated TCP support (#1000) Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * feat(aws)!: Remove deprecated PrivateLink env variable (#1002) Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * docs(aws): Removing JSON explanation in the include at match documentation (#1007) Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * Add environment variable to support backend storage tag enrichment (#1006) * feat(aws): AWSX-1566 Adding storage tag HTTP header Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * feat(aws): AWSX-1566 Adding new variable in the cloudformation template Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * feat: AWSX-1566 Restoring fetch in the integration test to see metrics Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! feat: AWSX-1566 Restoring fetch in the integration test to see metrics Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * docs(aws): AWSX-1566 Refine documentation around the new variable Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fix(aws): Fixing discrepancies of the fetch configuration between cloudformation and code Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * feat: Add a changelog for the v5 (#1009) * feat: Add a changelog for the v5 Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! feat: Add a changelog for the v5 Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Raphaël Allier <118757729+RaphaelAllier@users.noreply.github.com> * fixup! Update aws/logs_monitoring/CHANGELOG.md Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> Co-authored-by: Raphaël Allier <118757729+RaphaelAllier@users.noreply.github.com> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com> * Update aws/logs_monitoring/CHANGELOG.md Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com> * docs: Updating changelog from comments Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! docs: Updating changelog from comments Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * Add reference to the changelog in README (for public doc) (#1011) * feat: Add link to the changelog in the read toe reflect the documentation The readme is feeding this page https://docs.datadoghq.com/logs/guide/forwarder/?tab=cloudformation Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! feat: Add link to the changelog in the read toe reflect the documentation Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * refactor: Improve error handling for exception (#1012) * refactor: Improve error handling for exception Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! refactor: Improve error handling for exception Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * fixup! refactor: Improve error handling for exception Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> * Update aws/logs_monitoring/lambda_function.py Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com> * fixup! Update aws/logs_monitoring/lambda_function.py Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> --------- Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com> Co-authored-by: Raphaël Allier <118757729+RaphaelAllier@users.noreply.github.com> Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fix a behavior of running the
INCLUDE_AT_MATCHandEXCLUDE_AT_MATCHagainst the JSON dump of the log being sent to Datadog instead of the log as received.This is a BREAKING CHANGE if regexes where defined for JSON escaped string.
Previously, filtering a the
statusfield of a JSON log was possible like thisNow, it will be like this
Motivation
The Regex was hard to reason about. Searching for the regex in a log file was working but not in the forwarder, this leaded to some support from customers.
Testing Guidelines
Additional Notes
Types of changes
Check all that apply