Skip to content

AWS Forwarder v5 #1010

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 18 commits into from
Oct 23, 2025
Merged

AWS Forwarder v5 #1010

merged 18 commits into from
Oct 23, 2025

Conversation

@ViBiOh
Copy link
Contributor

@ViBiOh ViBiOh commented Oct 17, 2025

What does this PR do?

Motivation

Testing Guidelines

Additional Notes

Types of changes

  • Bug fix
  • New feature
  • Breaking change
  • Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

  • This PR's description is comprehensive
  • This PR contains breaking changes that are documented in the description
  • This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
  • This PR impacts documentation, and it has been updated (or a ticket has been logged)
  • This PR's changes are covered by the automated tests
  • This PR collects user input/sensitive content into Datadog
  • This PR passes the integration tests (ask a Datadog member to run the tests)
  • This PR passes the unit tests
  • This PR passes the installation tests (ask a Datadog member to run the tests)

ViBiOh and others added 7 commits October 8, 2025 17:37
@ViBiOh
refactor(aws)!: AWSX-1703 Change matching of regex. Previously done a…
4df1098 
…gainst JSON, now only log message (#996)

* refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

* fixup! refactor(aws)!: AWSX-1703 Change matching of regex. Previously done against JSON, now only on log

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

---------

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
@ViBiOh
feat(aws)!: Drop deprecated TCP support (#1000)
a9301a9 
Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
@ViBiOh
feat(aws)!: Remove deprecated PrivateLink env variable (#1002)
b3342fc 
Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
@ViBiOh
docs(aws): Removing JSON explanation in the include at match document…
f7fb1e1 
…ation (#1007)

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
@ViBiOh
Add environment variable to support backend storage tag enrichment (#…
7a39141 
…1006)

* feat(aws): AWSX-1566 Adding storage tag HTTP header

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

* feat(aws): AWSX-1566 Adding new variable in the cloudformation template

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

* feat: AWSX-1566 Restoring fetch in the integration test to see metrics

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

* fixup! feat: AWSX-1566 Restoring fetch in the integration test to see metrics

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

* docs(aws): AWSX-1566 Refine documentation around the new variable

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

---------

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
@ViBiOh
fix(aws): Fixing discrepancies of the fetch configuration between clo…
14e7d11 
…udformation and code

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
@ViBiOh @RaphaelAllier
feat: Add a changelog for the v5 (#1009)
b50198d 
* feat: Add a changelog for the v5

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

* fixup! feat: Add a changelog for the v5

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

* Update aws/logs_monitoring/CHANGELOG.md

Co-authored-by: Raphaël Allier <118757729+RaphaelAllier@users.noreply.github.com>

* fixup! Update aws/logs_monitoring/CHANGELOG.md

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

---------

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
Co-authored-by: Raphaël Allier <118757729+RaphaelAllier@users.noreply.github.com>
@ViBiOh ViBiOh requested a review from a team as a code owner October 17, 2025 09:47
@github-actions github-actions bot added the aws label Oct 17, 2025
@ge0Aja ge0Aja self-assigned this Oct 17, 2025
ge0Aja
ge0Aja reviewed Oct 17, 2025
View reviewed changes
aws/logs_monitoring/CHANGELOG.md Outdated

### Overview

Version 5.0.0 of the Datadog Lambda Forwarder introduces several breaking changes that remove deprecated features and improve log filtering behavior. This release introduces a new way to enrich your logs with tags that will reduce AWS Lambda related cost (S3, KMS and Lambda).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: prefer to remove you, yours, and just use abstracts i.e. new way to enrich logs

ge0Aja
ge0Aja reviewed Oct 17, 2025
View reviewed changes
aws/logs_monitoring/CHANGELOG.md Outdated

**Migration Required:**

- **Review and update your filtering regex patterns**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: same

ge0Aja
ge0Aja reviewed Oct 17, 2025
View reviewed changes
aws/logs_monitoring/CHANGELOG.md

- Removed the `DD_USE_TCP` / `DdUseTcp` environment variable and parameter
- Deleted the TCP client implementation
- All logs now **must** be sent via HTTP/HTTPS
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TCP (transport) doesn't relate to HTTP/HTTPs (application) maybe we should also clarify that logs are sent to a different intake as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a different intake than the direct TCP intake yes, but it was an internal configuration depending on the DD_SITE and it was not configurable.

ge0Aja
ge0Aja reviewed Oct 17, 2025
View reviewed changes
aws/logs_monitoring/CHANGELOG.md Outdated
**Migration Required:**

- Remove any configuration setting `DD_USE_TCP=true` or `DdUseTcp=true`
- The forwarder will now exclusively use HTTP transport
Copy link
Contributor

@ge0Aja ge0Aja Oct 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: HTTP intake (endpoint) instead of transport

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to HTTP Protocol

ge0Aja
ge0Aja reviewed Oct 17, 2025
View reviewed changes
ge0Aja
ge0Aja reviewed Oct 17, 2025
View reviewed changes
ge0Aja
ge0Aja reviewed Oct 17, 2025
View reviewed changes
ge0Aja
ge0Aja reviewed Oct 17, 2025
View reviewed changes
aws/logs_monitoring/CHANGELOG.md Outdated
aws lambda get-function-configuration --function-name "<YOUR_FORWARDER>" --query 'Environment.Variables.DD_USE_TCP'
```

2. **Verify you're not using the deprecated PrivateLink variable:**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
2. **Verify you're not using the deprecated PrivateLink variable:**
2. **Verify that deprecated PrivateLink variable is not used:**

ge0Aja
ge0Aja reviewed Oct 17, 2025
View reviewed changes
ge0Aja
ge0Aja reviewed Oct 17, 2025
View reviewed changes
ge0Aja
ge0Aja reviewed Oct 17, 2025
View reviewed changes
aws/logs_monitoring/CHANGELOG.md Outdated

**Migration Required:**

- **Review and update your filtering regex patterns**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- **Review and update your filtering regex patterns**
- **Review and update filtering regex patterns**

ge0Aja
ge0Aja approved these changes Oct 17, 2025
View reviewed changes
Copy link
Contributor

@ge0Aja ge0Aja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, some comments on the change-log

ViBiOh and others added 8 commits October 17, 2025 14:16
@ViBiOh @ge0Aja
Update aws/logs_monitoring/CHANGELOG.md
9e5b3f4 
Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com>
@ViBiOh @ge0Aja
Update aws/logs_monitoring/CHANGELOG.md
84a369c 
Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com>
@ViBiOh @ge0Aja
Update aws/logs_monitoring/CHANGELOG.md
4899a4c 
Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com>
@ViBiOh @ge0Aja
Update aws/logs_monitoring/CHANGELOG.md
f2d2298 
Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com>
@ViBiOh @ge0Aja
Update aws/logs_monitoring/CHANGELOG.md
6121a28 
Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com>
@ViBiOh
docs: Updating changelog from comments
df4d09c 
Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
@ViBiOh
fixup! docs: Updating changelog from comments
8edda81 
Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
@ViBiOh
Add reference to the changelog in README (for public doc) (#1011)
2356c4e 
* feat: Add link to the changelog in the read toe reflect the documentation

The readme is feeding this page https://docs.datadoghq.com/logs/guide/forwarder/?tab=cloudformation

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

* fixup! feat: Add link to the changelog in the read toe reflect the documentation

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

---------

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
@ViBiOh
refactor: Improve error handling for exception (#1012)
2712e51 
* refactor: Improve error handling for exception

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

* fixup! refactor: Improve error handling for exception

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

* fixup! refactor: Improve error handling for exception

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>

---------

Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
ge0Aja
ge0Aja reviewed Oct 22, 2025
View reviewed changes
aws/logs_monitoring/enhanced_lambda_metrics.py Outdated
"Encountered an error while trying to parse and submit enhanced metrics for log %s",
except Exception as e:
logger.error(
"Encountered an error while trying to parse and submit enhanced metrics for log %s: %s",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we could use the same formatted string approach that's used every where else

ge0Aja
ge0Aja reviewed Oct 22, 2025
View reviewed changes
@ViBiOh @ge0Aja
Update aws/logs_monitoring/lambda_function.py
58ce3b7 
Co-authored-by: Georgi <georgi.ajaeiya@datadoghq.com>
ge0Aja
ge0Aja approved these changes Oct 22, 2025
View reviewed changes
Copy link
Contributor

@ge0Aja ge0Aja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ViBiOh
fixup! Update aws/logs_monitoring/lambda_function.py
1220669 
Signed-off-by: Vincent Boutour <vincent.boutour@datadoghq.com>
@ViBiOh ViBiOh merged commit c36a79d into master Oct 23, 2025
12 checks passed
@ViBiOh ViBiOh deleted the aws-forwarder-v5-preparation branch October 23, 2025 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ge0Aja ge0Aja ge0Aja approved these changes

Assignees

@ge0Aja ge0Aja

Labels

aws

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ViBiOh @ge0Aja