Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Event handler loop fails on large database events using Athena audit backend #36442

Open
greedy52 opened this issue Jan 9, 2024 · 0 comments
Assignees
Labels
audit-log Issues related to Teleports Audit Log bug c-lo Internal Customer Reference support-load This issue generates support load

Comments

@greedy52
Copy link
Contributor

greedy52 commented Jan 9, 2024

Expected behavior:
Oversize database events should be trimmed to 1 MiB

Current behavior:

ERRO   Error ingesting Audit Log err:rpc error: code = Unknown desc = db.session.postgres.statements.parse event e3c6ff72-3085-4d6f-8248-20aee63b38f4 is 1.1 MiB and cannot be returned because it exceeds the maximum response size of 1.0 MiB event-handler/events_job.go:96
ERRO   Watcher event loop failed error:[
ERROR REPORT:
Original Error: *status.Error rpc error: code = Unknown desc = db.session.postgres.statements.parse event e3c6ff72-3085-4d6f-8248-20aee63b38f4 is 1.1 MiB and cannot be returned because it exceeds the maximum response size of 1.0 MiB
Stack Trace:
    /go/pkg/mod/github.com/gravitational/teleport/api@v0.0.0-20231229230854-390d33c42bbe/client/client.go:2208 github.com/gravitational/teleport/api/client.(*Client).SearchUnstructuredEvents
    /workspace/event-handler/teleport_events_watcher.go:225 main.(*TeleportEventsWatcher).getEvents
    /workspace/event-handler/teleport_events_watcher.go:153 main.(*TeleportEventsWatcher).fetch
    /workspace/event-handler/teleport_events_watcher.go:262 main.(*TeleportEventsWatcher).Events.func1
    /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit
User Message: rpc error: code = Unknown desc = db.session.postgres.statements.parse event e3c6ff72-3085-4d6f-8248-20aee63b38f4 is 1.1 MiB and cannot be returned because it exceeds the maximum response size of 1.0 MiB] event-handler/events_job.go:81
ERRO   Terminating with fatal error [1]... error:[
ERROR REPORT:
Original Error: *status.Error rpc error: code = Unknown desc = db.session.postgres.statements.parse event e3c6ff72-3085-4d6f-8248-20aee63b38f4 is 1.1 MiB and cannot be returned because it exceeds the maximum response size of 1.0 MiB
Stack Trace:
    /go/pkg/mod/github.com/gravitational/teleport/api@v0.0.0-20231229230854-390d33c42bbe/client/client.go:2208 github.com/gravitational/teleport/api/client.(*Client).SearchUnstructuredEvents
    /workspace/event-handler/teleport_events_watcher.go:225 main.(*TeleportEventsWatcher).getEvents
    /workspace/event-handler/teleport_events_watcher.go:153 main.(*TeleportEventsWatcher).fetch
    /workspace/event-handler/teleport_events_watcher.go:262 main.(*TeleportEventsWatcher).Events.func1
    /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit
User Message: rpc error: code = Unknown desc = db.session.postgres.statements.parse event e3c6ff72-3085-4d6f-8248-20aee63b38f4 is 1.1 MiB and cannot be returned because it exceeds the maximum response size of 1.0 MiB] lib/bail.go:28

Bug details:

  • Teleport version: v14

#35402 already implements trimming of large events in Athena querier. However, not all database events implements TrimToMaxSize.

@greedy52 greedy52 added bug audit-log Issues related to Teleports Audit Log labels Jan 9, 2024
@greedy52 greedy52 added the support-load This issue generates support load label Jan 9, 2024
@programmerq programmerq added P2 c-lo Internal Customer Reference labels Jan 9, 2024
@zmb3 zmb3 removed the P2 label Jan 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
audit-log Issues related to Teleports Audit Log bug c-lo Internal Customer Reference support-load This issue generates support load
Projects
None yet
Development

No branches or pull requests

4 participants