feat(metrics): add Datadog observability provider

[roger-zhangg]

Issue number:
#2903
#2015

Summary

We didn't included the Datadog metrics provider in our initial metrics provider release due to a dependency conflict. Now that the conflict is resolved, This RFC is for adding back the previously included Datadog metrics provider

Changes

Please provide a summary of what's being changed
If our customer would like to use Datadog for metrics. This pre-defined Datadog metrics provider is the solution. With pre-defined Datadog metrics provider, a current Cloudwatch EMF metrics user can switch to use Datadog metrics by making few lines of code changes.

⚠️ Customer still need to setup either a Datadog Lambda Layer or a Datadog log forwarder to send the metrics.

User experience

from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools.metrics.provider import DatadogMetrics
from aws_lambda_powertools.metrics.provider import DatadogMetricsProvider

# Use datadog-defined metrics provider
provider = DatadogMetricsProvider()
metrics = DatadogMetrics(provider=provider)

@metrics.log_metrics
def lambda_handler(event: dict, context: LambdaContext):
    metrics.add_metric(name="SuccessfulBooking", value=1, tags={"product":"ticket","order":"online"})
    
# JSON output to log
# {
#     "m": "SuccessfulBooking", 
#     "v": 1,
#     "e": 1678509106, 
#     "t": "['product:ticket', 'order:online']"
# }

    metrics.add_metric(name="SuccessfulBooking", value=1, product="ticket", order="online")
# JSON output to log
# {
#     "m": "SuccessfulBooking", 
#     "v": 1,
#     "e": 1678509106, 
#     "t": "['product:ticket', 'order:online']"
# }

How this provider works

stateDiagram-v2
    direction LR
    LambdaCode: Lambda code with Powertools for AWS Lambda
    DatadogSDK: Datadog SDK
    DatadogExtension: Datadog Lambda Extension
    Datadog: Datadog Dashboard

    LambdaCode --> DatadogSDK
    DatadogSDK --> DatadogExtension
    DatadogExtension --> Datadog

Loading

Please share what the user experience looks like before and after this change

Checklist

If your change doesn't seem to apply, please leave them unchecked.

• Meet tenets criteria
• I have performed a self-review of this change
• Changes have been tested
• Changes are documented
• PR title follows conventional commit semantics

Is this a breaking change?

RFC issue number:

Checklist:

• Migration process documented
• Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

[heitorlessa]

E2E tests are failing on Metrics after the provider merge - let's hold on before we continue here.

[codecov-commenter]

Codecov Report

Patch coverage: 99.26% and project coverage change: +0.13% 🎉

Comparison is base (0485c8a) 96.42% compared to head (400486c) 96.56%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2906      +/-   ##
===========================================
+ Coverage    96.42%   96.56%   +0.13%     
===========================================
  Files          171      175       +4     
  Lines         7690     7825     +135     
  Branches       570     1476     +906     
===========================================
+ Hits          7415     7556     +141     
+ Misses         219      217       -2     
+ Partials        56       52       -4     

Files Changed	Coverage Δ
aws_lambda_powertools/metrics/metrics.py	96.36% <ø> (ø)
...da_powertools/metrics/provider/datadog/warnings.py	83.33% <83.33%> (ø)
aws_lambda_powertools/metrics/provider/base.py	95.65% <100.00%> (+0.41%)	⬆️
...da_powertools/metrics/provider/datadog/__init__.py	100.00% <100.00%> (ø)
...bda_powertools/metrics/provider/datadog/datadog.py	100.00% <100.00%> (ø)
...bda_powertools/metrics/provider/datadog/metrics.py	100.00% <100.00%> (ø)
aws_lambda_powertools/shared/constants.py	100.00% <100.00%> (ø)

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[leandrodamascena]

E2E tests are failing on Metrics after the provider merge - let's hold on before we continue here.

We fixed the problem in the PR #2910. We can go ahead with PR.

[leandrodamascena]

Still missing in this PR:

• Add default tags to provide a similar experience as default_dimenions in the EMF provider.
• Clean the code
• Improve docstrings
• Create documentation
• Check code coverage and add # pragma: no cover to parts of code that we cannot cover with functional/unit tests.

[leandrodamascena]

Hi @roger-zhangg and @heitorlessa! I think this PR is ready to review and I have some information to make easy the review and tests:

1 - TAGS
Datadog uses tags instead of dimensions and you can define one tag per metric. Therefore, a tag defined in the add_metric method takes precedence over a tag defined in the set_default_tags method or default_tags in @metrics.log_metrics.

2 - ENV
Similar to CloudWatchEMFProvider, the customer can set POWERTOOLS_METRICS_NAMESPACE to define the default metrics namespace. If the customer doesn't inform a namespace, then default will be assumed.

3 - Flushing metrics
We support both methods to flush a metrics:
3.1 - Using datadog SDK: https://docs.datadoghq.com/serverless/installation/python/?tab=datadogcli#monitor-custom-business-logic
3.2 - Using Datadog Log Forward: https://docs.datadoghq.com/logs/guide/forwarder/?tab=cloudformation

The customer just need to set the env variable DD_FLUSH_TO_LOG or setting flush_to_log in the Provider/Metric class.

We have these two supports thanks to @roger-zhangg fantastic job investigating supported methods.

4 - Live test

4.1 - Clone this branch
4.2 - Create a SAM application using any supported Python runtime
4.3 - Add in requirements.txt the directory where you cloned the branch. E.g: /home/leandro/DEVEL-PYTHON/lambda-powertools
4.4 - Add the code below in your app.py

from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools.metrics.provider.datadog import DatadogMetrics, DatadogProvider

provider = DatadogProvider()
metrics = DatadogMetrics(provider=provider)
metrics.set_default_tags(powertools="datadog")

@metrics.log_metrics(capture_cold_start_metric=True)
def lambda_handler(event: dict, context: LambdaContext):
    
    #DATADOG
    metrics.add_metric(name="PRDataDog", value=1)

4.4 - If you want to deploy to your AWS account and send metrics to Datadog then follow steps 4.5 and 4.6, otherwise skip to 4.7

4.5 - In template.yaml you need to add environment variables DD_API_KEY and DD_SITE with respective values - you need to create an account in Datadog for this.
4.6 - Add the layers arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Extension:45 and arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Python310:75. Deploy to Lambda and test.

4.7 - If you want to test locally to see if the metrics are created, just add the environment variable DD_FLUSH_TO_LOG with value "True" and execute your Lambda locally.

It is still missing create the documentation, but I think the code is done.

Thanks so much again @roger-zhang! We can only get this far because you've done most of the work. 🚀

[roger-zhangg]

Thanks Leandro for the refactor! Looks good to me

[heitorlessa]

UX suggestions to improve the customer experience in setting default tags, and overriding tags within a add_metric operation.

[heitorlessa]

Final tasks on docs for me in this last pass -- next time, we should really do this in a separate PR given the level of detail we need.

---

• Consider /provider for each observability provider
• Collapse navigation by default due to size
• Enhance Datadog mermaid
• Provide minimal terminologies to improve customers experience
• Correct typo in terminologies
• Remove duplicate messaging
• Use blockquote instead of note in prime reading areas e.g., Install note on Datadog Forwarder
• Shorten install wording to give info upfront
• Fix SAM highlights
• Keep add_metrics simple without provider, focusing on timestamp / no timestamp
• Fix env vars placement
• Ensure default tags call out the DatadogMetrics instance
• Call out that Datadog Forwarder is a legacy option
• Move Datadog Forwarder as an advanced, largely to stay out of focus
• Ensure metric tag is called out instead of tag
• Ensure we call out Datadog SDK config too (e.g., API Key)
• Add callout why we use DD_FLUSH_TO_LOG in tests
• Call out how to pass API Key for Datadog (e.g., DD_API_KEY or DD_. Secer)
• Missing tags validation
• Missing test for timestamp feature
• Missing test for default_tags feature with two DatadogMetrics instance
• Missing test for tags warning

Separate PR

• Consider Event Handler index just like Metrics after nav collapse
• Add mermaid for Metrics core too
• Add Observability Provider section in Metrics.md

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
10 Code Smells

No Coverage information
0.0% Duplication

[heitorlessa]

approved - LGTM! I'd only ask to create a separate issue to investigate testing when datadog SDK isn't available, so we're fully covered.