Skip to content

Conversation

@ge0Aja
Copy link
Contributor

@ge0Aja ge0Aja commented Mar 14, 2024

What does this PR do?

Add a prefix string to tags cache filenames using a hash of the function ARN
This will allow to store cache files on a single S3 bucket for multiple Lambda forwarders.
We can access the function ARN from the context object provided by AWS on function invocation
https://docs.aws.amazon.com/lambda/latest/dg/python-context.html
Also, update readme file by adding a troubleshooting section when encountering issues creating S3 triggers

Motivation

Testing Guidelines

Additional Notes

Types of changes

  • Bug fix
  • New feature
  • Breaking change
  • Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

  • This PR's description is comprehensive
  • This PR contains breaking changes that are documented in the description
  • This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
  • This PR impacts documentation, and it has been updated (or a ticket has been logged)
  • This PR's changes are covered by the automated tests
  • This PR collects user input/sensitive content into Datadog
  • This PR passes the integration tests (ask a Datadog member to run the tests)
  • This PR passes the unit tests
  • This PR passes the installation tests (ask a Datadog member to run the tests)

@github-actions github-actions bot added the aws label Mar 14, 2024
@ge0Aja ge0Aja force-pushed the georgi/cache_file_prefix branch 5 times, most recently from 1d7a85b to f075d18 Compare March 14, 2024 17:51
Use lambda function name as a prefix when creating tags cache files accessed by the forwarder.
This will allow to use the same S3 bucket to store cache files from different forwarderst push
@ge0Aja ge0Aja force-pushed the georgi/cache_file_prefix branch from f075d18 to 1275418 Compare March 15, 2024 09:24
@ge0Aja ge0Aja marked this pull request as ready for review March 15, 2024 09:28
@ge0Aja ge0Aja requested a review from a team as a code owner March 15, 2024 09:28


class BaseTagsCache(object):
CACHE_PREFIX = None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use lowercased, instance variables, initialized by the constructor for these 3 ?

The way this is defined here makes it possible to create an instance of BaseTagsCache (or subclass) which does not properly initialize the cache prefix

parsed = parse(event, context)
enriched = enrich(parsed)
# set the prefix for cache layer
if not cache_layer.prefix:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we instead create the CacheLayer with the proper prefix directly here ? and pass the prefix to Caches in CacheLayer.__init__

"""Retrieves extra tags from lambda, either read from the function arn, or by fetching lambda tags from the function itself.
Args:
log (dict<str, str | dict | int>): a log parsed from the event in the split method
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the best arg name log or log_event ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be a lambda log event here given the context. I think it should be good to use log_event

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, please just rename it in the comment then.

ge0Aja and others added 3 commits March 15, 2024 11:40
@ge0Aja ge0Aja requested a review from tyrcho March 15, 2024 17:22
@ge0Aja ge0Aja force-pushed the georgi/cache_file_prefix branch from a0166a7 to 988b073 Compare March 18, 2024 09:13
@ge0Aja ge0Aja merged commit b54ce7f into master Mar 18, 2024
@ge0Aja ge0Aja deleted the georgi/cache_file_prefix branch March 18, 2024 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants