-
Notifications
You must be signed in to change notification settings - Fork 1
Add lambda function log table #177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
70f69dd
to
433272e
Compare
da55abd
to
60c36b5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds a new aws_lambda_log
table to collect and structure AWS Lambda function logs from multiple sources, including S3, CloudWatch Log Groups, and generic artifact sources. Key changes include:
- Definition of
LambdaLogTable
with enrichment logic and new source metadata. - Implementation of
LambdaLogMapper
to parse various Lambda log formats (S3, CloudWatch, JSON, plain text). - Comprehensive documentation and example queries for the new
aws_lambda_log
table.
Reviewed Changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
File | Description |
---|---|
tables/lambda_log/lambda_log_table.go | Introduces LambdaLogTable , source metadata, and enrich logic |
tables/lambda_log/lambda_log_mapper.go | Implements LambdaLogMapper with parsing for multiple log formats |
tables/lambda_log/lambda_log.go | Defines LambdaLog schema and column descriptions |
go.mod | Bumps tailpipe-plugin-sdk and klauspost/compress versions |
docs/tables/aws_lambda_log/queries.md | Adds 16+ example queries for the new aws_lambda_log table |
docs/tables/aws_lambda_log/index.md | Documents table usage, message formats, and configuration |
aws/plugin.go | Registers the new aws_lambda_log table in the plugin |
Comments suppressed due to low confidence (3)
tables/lambda_log/lambda_log_table.go:44
- This comment incorrectly labels the CloudWatch Log Group source as an S3 source. Update it to
// CloudWatch Log Group source
for clarity.
// S3 artifact source
tables/lambda_log/lambda_log_mapper.go:325
- There is a typo in the function name
parseLambdaPainTextLog
—it should beparseLambdaPlainTextLog
to match its purpose.
func parseLambdaPainTextLog(line string, log *LambdaLog) (*LambdaLog, error) {
tables/lambda_log/lambda_log_mapper.go:55
- [nitpick] This mapper contains complex handling for multiple log formats; consider adding unit tests to cover each parsing path to ensure reliability and catch future regressions.
func (m *LambdaLogMapper) Map(_ context.Context, a any, _ ...mappers.MapOption[*LambdaLog]) (*LambdaLog, error) {
// tp_index | ||
row.TpIndex = schema.DefaultIndex | ||
|
||
var arnRegex = regexp.MustCompile(`arn:aws:[^,\s'"\\]+`) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Compiling the ARN regex on every row enrich call can be expensive; consider moving regexp.MustCompile
to a package-level variable so it's compiled only once at startup.
Copilot uses AI. Check for mistakes.
if (strings.HasPrefix(raw, "REPORT") || strings.HasPrefix(raw, "END") || strings.HasPrefix(raw, "START") || strings.HasPrefix(raw, "INIT_START") || strings.HasPrefix(raw, "EXTENSION") || strings.HasPrefix(raw, "TELEMETRY") || (isTextFormat && !isTimestamp(strings.Fields(raw)[0]) && !strings.HasPrefix(raw, "["))) && parsableJsonData == nil { | ||
lambdaLog, err := parseLambdaPainTextLog(raw, row) | ||
if err != nil { | ||
return nil, fmt.Errorf("error parsing lambda pain text log: %w", err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The error message contains a typo: lambda pain text log
should be lambda plain text log
.
Copilot uses AI. Check for mistakes.
…ich the row if row does not have the timestamp
1628b41
to
3096098
Compare
Example query results
Results