Skip to content

fix(logs-cleanup): listing active workspaces into mem + download time streaming lims#4692

Merged
icecrasher321 merged 13 commits into
stagingfrom
improvement/cleanup-logs-mem-reads
May 21, 2026
Merged

fix(logs-cleanup): listing active workspaces into mem + download time streaming lims#4692
icecrasher321 merged 13 commits into
stagingfrom
improvement/cleanup-logs-mem-reads

Conversation

@icecrasher321
Copy link
Copy Markdown
Collaborator

@icecrasher321 icecrasher321 commented May 21, 2026

Summary

  • Server side internal routes missing for typeform and google slides exports (anti-pattern)
  • Skill to check for memory materialization caps in PRs
  • Execution logs cleanup should load workspaces in paginated fashion
  • Execution logs overall tracespan size check

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Other: Performance + Memory Safety

Testing

Tested manually

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel
Copy link
Copy Markdown

vercel Bot commented May 21, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped May 21, 2026 11:15pm

Request Review

@cursor
Copy link
Copy Markdown

cursor Bot commented May 21, 2026

PR Summary

Medium Risk
Medium risk due to broad changes across several production API/tool routes and altered error/status behavior (notably new 413 paths), though the changes are mostly additive safety checks and bounded reads.

Overview
Hardens multiple API and tool routes against unbounded memory/materialization by switching from raw arrayBuffer()/formData() reads to stream-limits helpers with explicit byte caps, and by returning 413 for oversized payloads/responses.

/api/files/parse now enforces a max 10 files, validates file reference shape (rejects inline/data-like inputs), streams external/cloud downloads with maxResponseBytes, and adds a combined parsed-output cap with partial-result preservation for multi-file requests.

File upload and CSV import routes now read multipart bodies/files with size limits (supporting chunked uploads without content-length) and add tests asserting early rejection before materializing file contents.

Adds new internal tool routes for Google Slides export and Typeform file download that download with byte caps and store results as files (execution or copilot storage), and updates DocuSign, image, TTS, and video tool proxies to use bounded JSON/text/buffer reads, timeouts/abort propagation, and capped reference/media downloads; related block outputs gain a file output where applicable.

Also adds a reviewer skill doc (memory-load-check) and tightens log cleanup reference checks by scoping the retained-reference query by workspace.

Reviewed by Cursor Bugbot for commit 56116aa. Configure here.

Comment thread apps/sim/lib/billing/cleanup-dispatcher.ts
Comment thread apps/sim/lib/billing/cleanup-dispatcher.ts
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 21, 2026

Greptile Summary

This PR addresses memory and performance safety in the execution log cleanup pipeline, adds server-side proxy routes for Typeform and Google Slides file transfers, and introduces a new stream-size enforcement library used across all download routes.

  • Paginated workspace loading (cleanup-dispatcher.ts): replaces a full-table workspace load with a cursor-based page loop (500 rows per page), eliminating the OOM risk on large deployments.
  • Execution log compaction (logger.ts): adds a three-tier size-bounding strategy to prevent oversized JSONB writes, plus new extractFilesFromExecution for cleanup tracking — but file extraction happens after compaction, so files in large span outputs can be silently dropped from the files column.
  • Streaming size limits (stream-limits.ts): new module with bounded stream readers and PayloadSizeLimitError, adopted by all tool download routes.

Confidence Score: 3/5

The log-compaction path can silently drop file references, causing permanent storage leaks for large executions; the cross-tenant reference scan gap in cleanup-logs.ts is also still present.

The execution logger compacts trace span outputs exceeding 8 KB before extractFilesFromExecution runs, so file references in large span outputs are never written to the files column and are never deleted during log cleanup. This storage resource leak is introduced by the ordering of two new operations. The cross-tenant reference-scan narrowing was previously flagged and remains unaddressed.

apps/sim/lib/logs/execution/logger.ts — file extraction must happen before compaction; apps/sim/background/cleanup-logs.ts — cross-tenant reference scan is still narrowed to only the deleting workspaces

Important Files Changed

Filename Overview
apps/sim/background/cleanup-logs.ts Adds workspace ID filtering to the large-value reference scan. Cross-tenant reference-scan correctness concern (previously flagged) still present.
apps/sim/lib/billing/cleanup-dispatcher.ts Refactors to streaming forEachCleanupChunk with cursor-based pagination, fixing full-table-load memory issue. Chunk numbering is globally unique for non-enterprise plans.
apps/sim/lib/logs/execution/logger.ts Adds three-tier execution data compaction and file-reference extraction, but extraction runs after compaction so files in large span outputs are silently dropped from the files column.
apps/sim/lib/core/utils/stream-limits.ts New module providing size-bounded stream/response reading utilities with PayloadSizeLimitError.
apps/sim/app/api/tools/google_slides/export-presentation/route.ts New server-side route with encodeURIComponent, DNS-pinned SSRF mitigation, and size-bounded reads.
apps/sim/app/api/tools/typeform/files/route.ts New server-side route with proper URL encoding and DNS-pinned fetch.
apps/sim/lib/execution/payloads/materialization.server.ts Adds byte-limit constants, enforcement functions, and assertUserFileContentAccess for execution file scope checking.
apps/sim/tools/google_slides/export_presentation.ts Migrates to server-side proxy route with optional _context for execution file upload.
apps/sim/tools/typeform/files.ts Migrates to server-side proxy route, removing client-side binary handling.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[dispatchCleanupJobs] --> B{isTriggerAvailable?}
    B -- yes --> C[forEachCleanupChunk]
    B -- no --> D{shouldExecuteInline?}
    D -- yes --> C
    D -- no --> E[JobQueue enqueue]
    C --> F[listActiveWorkspaceCleanupScopeRowsPage]
    F --> G[resolvePlanTypesByWorkspaceId]
    G --> H[emit chunks per plan]
    H --> I{dispatch path}
    I -- Trigger.dev --> J[tasks.batchTrigger]
    I -- inline --> K[runCleanupLogs]
    I -- job queue --> E
    K --> L[cleanupWorkflowExecutionLogs]
    L --> M[chunkedBatchDelete]
    M --> N[filterLargeValueKeys scoped to workspaceIds]
    N --> O[deleteExecutionFiles from row.files]
    N --> P[deleteLargeValueStorageKeys]
Loading

Reviews (8): Last reviewed commit: "more fixes" | Re-trigger Greptile

Comment thread apps/sim/lib/billing/cleanup-dispatcher.ts Outdated
@icecrasher321
Copy link
Copy Markdown
Collaborator Author

@greptile

@icecrasher321
Copy link
Copy Markdown
Collaborator Author

bugbot run

Comment thread apps/sim/app/api/files/parse/route.ts Outdated
Comment thread apps/sim/background/cleanup-logs.ts
@icecrasher321
Copy link
Copy Markdown
Collaborator Author

@greptile

@icecrasher321
Copy link
Copy Markdown
Collaborator Author

bugbot run

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 0b28132. Configure here.

@icecrasher321
Copy link
Copy Markdown
Collaborator Author

bugbot run

@icecrasher321
Copy link
Copy Markdown
Collaborator Author

@greptile

Comment thread apps/sim/app/api/files/parse/route.ts Outdated
Comment thread apps/sim/app/api/files/parse/route.ts
@icecrasher321
Copy link
Copy Markdown
Collaborator Author

@greptile

@icecrasher321
Copy link
Copy Markdown
Collaborator Author

bugbot run

Comment thread apps/sim/background/cleanup-logs.ts
Comment thread apps/sim/app/api/files/upload/route.ts
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 55da2f6. Configure here.

@icecrasher321
Copy link
Copy Markdown
Collaborator Author

bugbot run

@icecrasher321
Copy link
Copy Markdown
Collaborator Author

@greptile

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 8d342ec. Configure here.

Comment thread apps/sim/app/api/tools/google_slides/export-presentation/route.ts
@icecrasher321
Copy link
Copy Markdown
Collaborator Author

@greptile

@icecrasher321
Copy link
Copy Markdown
Collaborator Author

bugbot run

@icecrasher321
Copy link
Copy Markdown
Collaborator Author

@greptile

Comment thread apps/sim/blocks/blocks/file.ts
@icecrasher321
Copy link
Copy Markdown
Collaborator Author

bugbot run

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 56116aa. Configure here.

Comment thread apps/sim/app/api/tools/docusign/route.ts
@icecrasher321 icecrasher321 merged commit 47bd7fa into staging May 21, 2026
9 checks passed
@waleedlatif1 waleedlatif1 deleted the improvement/cleanup-logs-mem-reads branch May 22, 2026 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant