Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(parser): extract CloudWatch Logs in Kinesis streams #1726

Conversation

heitorlessa
Copy link
Contributor

@heitorlessa heitorlessa commented Nov 16, 2022

Issue number: #1725

Summary

Changes

Please provide a summary of what's being changed

This PR introduces two functions to decode, decompress, and extract CloudWatchLogsDecode from Kinesis Data Streams.

  • extract_cloudwatch_logs_from_event. Returns a list of CloudWatchLogsDecode from KinesisDataStreamModel. Useful for simple processing.
  • extract_cloudwatch_logs_from_record. Returns CloudWatchLogsDecode from KinesisDataStreamRecord. Useful for robust batch processing and seamless integration with Batch Processing

User experience

Please share what the user experience looks like before and after this change

Simple processing

from aws_lambda_powertools.utilities.parser.models.cloudwatch import CloudWatchLogsDecode
from aws_lambda_powertools.utilities.parser.models import KinesisDataStreamModel
from aws_lambda_powertools.utilities.parser.models.kinesis import extract_cloudwatch_logs_from_record
from aws_lambda_powertools.utilities.typing import LambdaContext


@event_parser(model=KinesisDataStreamModel)
def lambda_handler(event: KinesisDataStreamModel, context: LambdaContext):
    logs: CloudWatchLogsDecode = extract_cloudwatch_logs_from_event(event)
    for log in logs:
        if log.messageType == "DATA_MESSAGE":
            return "success"
    return "success"

Processing batch of logs

from aws_lambda_powertools.utilities.batch import (BatchProcessor, EventType,
                                                   batch_processor)
from aws_lambda_powertools.utilities.parser.models import (
    KinesisDataStreamModel, KinesisDataStreamRecord)
from aws_lambda_powertools.utilities.parser.models.kinesis import \
    extract_cloudwatch_logs_from_record

processor = BatchProcessor(event_type=EventType.KinesisDataStreams, model=KinesisDataStreamModel)


def record_handler(record: KinesisDataStreamRecord):
    log = extract_cloudwatch_logs_from_record(record)
    return log.messageType == "DATA_MESSAGE"


@batch_processor(record_handler=record_handler, processor=processor)
def lambda_handler(event, context):
    return processor.response()

Checklist

If your change doesn't seem to apply, please leave them unchecked.

Is this a breaking change?

RFC issue number:

Checklist:

  • Migration process documented
  • Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@heitorlessa heitorlessa requested a review from a team as a code owner November 16, 2022 10:18
@heitorlessa heitorlessa requested review from leandrodamascena and removed request for a team November 16, 2022 10:18
@boring-cyborg boring-cyborg bot added the tests label Nov 16, 2022
@pull-request-size pull-request-size bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 16, 2022
@heitorlessa
Copy link
Contributor Author

cc @ran-isenberg - waiting for CI checks to pass before merging.

@github-actions github-actions bot added the feature New feature or functionality label Nov 16, 2022
@codecov-commenter
Copy link

codecov-commenter commented Nov 16, 2022

Codecov Report

Base: 99.27% // Head: 99.27% // Increases project coverage by +0.00% 🎉

Coverage data is based on head (c55cd29) compared to base (28239ad).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #1726   +/-   ##
========================================
  Coverage    99.27%   99.27%           
========================================
  Files          129      129           
  Lines         6068     6079   +11     
  Branches       404      406    +2     
========================================
+ Hits          6024     6035   +11     
  Misses          20       20           
  Partials        24       24           
Impacted Files Coverage Δ
...mbda_powertools/utilities/parser/models/kinesis.py 100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@heitorlessa
Copy link
Contributor Author

@leandrodamascena for when you're back - added working examples in the PR to reuse when refactoring Parser's docs.

Merging

@heitorlessa heitorlessa merged commit 2b4740a into aws-powertools:develop Nov 16, 2022
@heitorlessa heitorlessa deleted the feat/parser-kinesis-extract-cwl-logs branch November 16, 2022 10:25
@heitorlessa heitorlessa linked an issue Nov 16, 2022 that may be closed by this pull request
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or functionality size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature request: Extract CloudWatch Logs from Kinesis in Parser
2 participants