Skip to content

Add lambda function log table #177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Conversation

ParthaI
Copy link
Contributor

@ParthaI ParthaI commented May 20, 2025

Example query results

Results
Add example SQL query results here (please include the input queries as well)

@ParthaI ParthaI self-assigned this May 20, 2025
@ParthaI ParthaI marked this pull request as draft May 20, 2025 09:43
@ParthaI ParthaI force-pushed the add-lambda-function-log-table branch from 70f69dd to 433272e Compare May 28, 2025 08:46
@ParthaI ParthaI force-pushed the add-lambda-function-log-table branch from da55abd to 60c36b5 Compare June 23, 2025 05:38
@ParthaI ParthaI marked this pull request as ready for review June 26, 2025 16:53
@rajlearner17 rajlearner17 added the tailpipe Tailpipe plugin issues label Jul 2, 2025
@rajlearner17 rajlearner17 requested a review from Copilot July 7, 2025 11:06
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a new aws_lambda_log table to collect and structure AWS Lambda function logs from multiple sources, including S3, CloudWatch Log Groups, and generic artifact sources. Key changes include:

  • Definition of LambdaLogTable with enrichment logic and new source metadata.
  • Implementation of LambdaLogMapper to parse various Lambda log formats (S3, CloudWatch, JSON, plain text).
  • Comprehensive documentation and example queries for the new aws_lambda_log table.

Reviewed Changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tables/lambda_log/lambda_log_table.go Introduces LambdaLogTable, source metadata, and enrich logic
tables/lambda_log/lambda_log_mapper.go Implements LambdaLogMapper with parsing for multiple log formats
tables/lambda_log/lambda_log.go Defines LambdaLog schema and column descriptions
go.mod Bumps tailpipe-plugin-sdk and klauspost/compress versions
docs/tables/aws_lambda_log/queries.md Adds 16+ example queries for the new aws_lambda_log table
docs/tables/aws_lambda_log/index.md Documents table usage, message formats, and configuration
aws/plugin.go Registers the new aws_lambda_log table in the plugin
Comments suppressed due to low confidence (3)

tables/lambda_log/lambda_log_table.go:44

  • This comment incorrectly labels the CloudWatch Log Group source as an S3 source. Update it to // CloudWatch Log Group source for clarity.
			// S3 artifact source

tables/lambda_log/lambda_log_mapper.go:325

  • There is a typo in the function name parseLambdaPainTextLog—it should be parseLambdaPlainTextLog to match its purpose.
func parseLambdaPainTextLog(line string, log *LambdaLog) (*LambdaLog, error) {

tables/lambda_log/lambda_log_mapper.go:55

  • [nitpick] This mapper contains complex handling for multiple log formats; consider adding unit tests to cover each parsing path to ensure reliability and catch future regressions.
func (m *LambdaLogMapper) Map(_ context.Context, a any, _ ...mappers.MapOption[*LambdaLog]) (*LambdaLog, error) {

// tp_index
row.TpIndex = schema.DefaultIndex

var arnRegex = regexp.MustCompile(`arn:aws:[^,\s'"\\]+`)
Copy link
Preview

Copilot AI Jul 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compiling the ARN regex on every row enrich call can be expensive; consider moving regexp.MustCompile to a package-level variable so it's compiled only once at startup.

Copilot uses AI. Check for mistakes.

if (strings.HasPrefix(raw, "REPORT") || strings.HasPrefix(raw, "END") || strings.HasPrefix(raw, "START") || strings.HasPrefix(raw, "INIT_START") || strings.HasPrefix(raw, "EXTENSION") || strings.HasPrefix(raw, "TELEMETRY") || (isTextFormat && !isTimestamp(strings.Fields(raw)[0]) && !strings.HasPrefix(raw, "["))) && parsableJsonData == nil {
lambdaLog, err := parseLambdaPainTextLog(raw, row)
if err != nil {
return nil, fmt.Errorf("error parsing lambda pain text log: %w", err)
Copy link
Preview

Copilot AI Jul 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error message contains a typo: lambda pain text log should be lambda plain text log.

Copilot uses AI. Check for mistakes.

@ParthaI ParthaI force-pushed the add-lambda-function-log-table branch from 1628b41 to 3096098 Compare July 7, 2025 14:43
@rajlearner17 rajlearner17 requested review from misraved and cbruno10 July 9, 2025 06:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tailpipe Tailpipe plugin issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants