Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add describe_log_streams and filter_log_events to the CloudWatch module #1785

Merged
merged 26 commits into from
Nov 28, 2022
Merged

Add describe_log_streams and filter_log_events to the CloudWatch module #1785

merged 26 commits into from
Nov 28, 2022

Conversation

KhueNgocDang
Copy link
Contributor

Feature

  • Feature

Detail

  • Add describe_log_streams to CloudWatch module for getting log streams' details from specified log group.
  • Add filter_log_events to CloudWatch module for returning filtered log events from specified log group's log streams.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@KhueNgocDang KhueNgocDang marked this pull request as draft November 20, 2022 09:21
@malachi-constant
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: 839e790
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@malachi-constant
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: 8f86972
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@malachi-constant
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: 792f6e8
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@KhueNgocDang KhueNgocDang marked this pull request as ready for review November 20, 2022 11:10
@malachi-constant
Copy link
Contributor

Hi @KhueNgocDang looks like we are getting a test failure on the change.

tests/test_cloudwatch.py::test_describe_log_streams_and_filter_log_events

@malachi-constant malachi-constant added the enhancement New feature or request label Nov 20, 2022
@malachi-constant
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: c83a623
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@malachi-constant
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: 4a15322
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@KhueNgocDang KhueNgocDang marked this pull request as draft November 21, 2022 12:29
@KhueNgocDang
Copy link
Contributor Author

@malachi-constant
The test for commit id: 4a15322 failed due to lack of policy logs:filterLogEvents in the CodeBuild's service role. Would you please add the policy to the service role?

@LeonLuttenberger
Copy link
Contributor

@malachi-constant The test for commit id: 4a15322 failed due to lack of policy logs:filterLogEvents in the CodeBuild's service role. Would you please add the policy to the service role?

Hey, I've added the permission to the policy and re-ran the CodeBuild.

@malachi-constant
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: 4a15322
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@malachi-constant
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: 5e7b568
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@malachi-constant
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: 2c33c90
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@KhueNgocDang
Copy link
Contributor Author

@malachi-constant @LeonLuttenberger
Thank you for helping me with this.

I encountered a bug in CloudWatch's 'test_query' while debugging the pull request in 5e7b568. This did not happen to the other commits.
image

@KhueNgocDang KhueNgocDang marked this pull request as ready for review November 22, 2022 14:04
@KhueNgocDang KhueNgocDang removed their assignment Nov 22, 2022
@LeonLuttenberger
Copy link
Contributor

@malachi-constant @LeonLuttenberger Thank you for helping me with this.

I encountered a bug in CloudWatch's 'test_query' while debugging the pull request in 5e7b568. This did not happen to the other commits. image

Hey,

Yes, that's a transient error in our testing suite. It's not consistent, and re-running the job usually gets rid of it. We haven't gotten around to addressing the root cause of the error yet sadly.

But, the good news is that it's not an issue with your code :)

Cheers,
Leon

@malachi-constant malachi-constant added feature and removed enhancement New feature or request labels Nov 22, 2022
LeonLuttenberger
LeonLuttenberger requested changes Nov 22, 2022
View reviewed changes
Copy link
Contributor

@LeonLuttenberger LeonLuttenberger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overrall, I just found a few issues which I described in the comments.

awswrangler/cloudwatch.py Outdated Show resolved Hide resolved
awswrangler/cloudwatch.py Outdated
log_stream_names = describe_log_streams(
log_group_name=log_group_name, log_stream_name_prefix=log_stream_name_prefix, boto3_session=boto3_session
)["logStreamName"].tolist()
assert log_stream_names is not None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we use assert anywhere in this codebase apart from the unit tests. It would be better to expicitely throw an error here.

However, can thus value even be None in this caqse? If describe_log_streams doesn't return any results, then wouldn't log_stream_names just be an empty list?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used assert to prevent mypy from raising Argument 1 to "len" has incompatible type "Optional[List[str]]"; expected "Sized". There are some cases like mine in the codebase (lines 407 and 410 in postgresql.py) that used # type: ignore to bypass the errors. but I don't like that approach, so I used assert instead.
I'll replace the assert with a one-liner if statement at line 434 that assigns an empty list to log_stream_names If describe_log_streams doesn't return any results.

tests/test_cloudwatch.py Show resolved Hide resolved
@malachi-constant
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: f583804
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jaidisido jaidisido merged commit 2b8190e into aws:main Nov 28, 2022
@kukushking kukushking added this to the 2.18.0 milestone Dec 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LeonLuttenberger LeonLuttenberger LeonLuttenberger approved these changes

@jaidisido jaidisido jaidisido approved these changes

Assignees
No one assigned
Labels
feature
Projects
None yet
Milestone
2.18.0
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@KhueNgocDang @malachi-constant @LeonLuttenberger @jaidisido @kukushking