Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(event_sources): extract CloudWatch Logs in Kinesis streams #1710

Merged

Conversation

heitorlessa
Copy link
Contributor

@heitorlessa heitorlessa commented Nov 14, 2022

Issue number: #1488

Summary

Changes

Please provide a summary of what's being changed

This PR introduces two functions to decode, decompress, and extract CloudWatchLogsDecodedData from Kinesis Data Streams.

  • extract_cloudwatch_logs_from_event. Returns a list of CloudWatchLogsDecodedData from KinesisStreamEvent. Useful for simple processing.
  • extract_cloudwatch_logs_from_record. Returns CloudWatchLogsDecodedData from KinesisStreamRecord. Useful for robust batch processing and seamless integration with Batch Processing

User experience

Please share what the user experience looks like before and after this change

Batch processing

from aws_lambda_powertools.utilities.batch import (BatchProcessor, EventType,
                                                   batch_processor)
from aws_lambda_powertools.utilities.data_classes.kinesis_stream_event import (
    KinesisStreamRecord, extract_cloudwatch_logs_from_record)

processor = BatchProcessor(event_type=EventType.KinesisDataStreams)


def record_handler(record: KinesisStreamRecord):
    log = extract_cloudwatch_logs_from_record(record)
    return log.message_type == "DATA_MESSAGE"


@batch_processor(record_handler=record_handler, processor=processor)
def lambda_handler(event, context):
    return processor.response()

Simple processing

from typing import List

from aws_lambda_powertools.utilities.data_classes import event_source
from aws_lambda_powertools.utilities.data_classes.cloud_watch_logs_event import \
    CloudWatchLogsDecodedData
from aws_lambda_powertools.utilities.data_classes.kinesis_stream_event import (
    KinesisStreamEvent, extract_cloudwatch_logs_from_event)


@event_source(data_class=KinesisStreamEvent)
def simple_handler(event: KinesisStreamEvent, context):
    logs: List[CloudWatchLogsDecodedData] = extract_cloudwatch_logs_from_event(event)
    for log in logs:
        if log.message_type == "DATA_MESSAGE":
            return "success"
    return "success"

Checklist

If your change doesn't seem to apply, please leave them unchecked.

Is this a breaking change?

RFC issue number:

Checklist:

  • Migration process documented
  • Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.


View rendered docs/utilities/data_classes.md

@heitorlessa heitorlessa requested a review from a team as a code owner November 14, 2022 12:37
@heitorlessa heitorlessa requested review from rubenfonseca and removed request for a team November 14, 2022 12:37
@boring-cyborg boring-cyborg bot added documentation Improvements or additions to documentation tests labels Nov 14, 2022
@pull-request-size pull-request-size bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 14, 2022
@heitorlessa heitorlessa changed the title feat(event_sources): extract CW logs in Kinesis feat(event_sources): extract CloudWatch Logs in Kinesis streams Nov 14, 2022
@github-actions github-actions bot added the feature New feature or functionality label Nov 14, 2022
@heitorlessa
Copy link
Contributor Author

customer approved design. Merging after extra due diligence since @rubenfonseca and @leandrodamascena are unavailable.

@heitorlessa heitorlessa merged commit 28239ad into aws-powertools:develop Nov 16, 2022
@heitorlessa heitorlessa deleted the feat/kinesis-extract-cwl-logs branch November 16, 2022 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation feature New feature or functionality size/L Denotes a PR that changes 100-499 lines, ignoring generated files. tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature request: Data Class for Cloudwatch Logs passed through Kinesis Stream
1 participant