fix(ENG-12604): chunk batch classification to bound ONNX memory usage

[hiskudin]

Summary

• classifyBatch() now processes sentences in chunks of 32 instead of all at once
• Prevents OOM in Lambda environments when scanning large list responses (e.g. 100+ notes)

Problem

When a list response (e.g. ats_list_notes with 100 items) is passed to defendToolResult(), extractStrings() collects all text and classifyBySentence() splits it into hundreds of sentences. classifyBatch() then allocated a single ONNX tensor for all sentences — the attention matrices scale as O(batch × seqLen²) in native memory (outside V8 heap), reaching several GB for large batches and crashing Lambda.

Fix

Split classifyBatch() into chunks of 32 sentences max. Each chunk runs a separate session.run() call with bounded memory (~50MB). Results are concatenated.

Changes

• src/classifiers/onnx-classifier.ts — classifyBatch() now loops in chunks via classifyBatchChunk(); added MAX_BATCH_CHUNK = 32 constant
• specs/onnx-classifier.spec.ts — added test with 40 texts to verify cross-chunk correctness

Test plan

• Existing classifyBatch test passes (3 items, single chunk)
• New test with 40 items passes (forces 2 chunks)
• All tests pass
• Memory stays bounded for large payloads

🤖 Generated with Claude Code

---

Summary by cubic

Chunked batch classification to bound ONNX native memory and prevent Lambda OOM on large payloads. Addresses ENG-12604 by processing texts in chunks of 32 per inference without changing outputs.

• Bug Fixes
  • Process batches in chunks (max 32) via a new internal classifyBatchChunk(); concatenates results and caps memory per call (~50MB).
  • Added a spec with 40 texts to verify cross-chunk ordering and scores.

^{Written for commit 86e4a60. Summary will update on new commits.}

[cubic-dev-ai]

No issues found across 2 files

_{Auto-approved: Safely introduces batch chunking to the ONNX classifier to prevent OOM issues. Includes unit tests for the new logic.}

[Copilot]

Pull request overview

This PR updates the ONNX-based prompt-injection classifier to avoid Lambda OOMs by bounding native (non-V8) memory usage during batch inference, especially when classifying hundreds of sentences extracted from large tool/list responses.

Changes:

• Split OnnxClassifier.classifyBatch() into fixed-size chunks (max 32 texts per session.run()).
• Added classifyBatchChunk() to run a single bounded ONNX inference call and concatenate results.
• Added a test that classifies 40 texts to exercise cross-chunk behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/classifiers/onnx-classifier.ts	Implements chunked batch inference to cap ONNX attention-matrix memory usage.
specs/onnx-classifier.spec.ts	Adds a larger batch test to validate correctness across multiple chunks.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The test comment hard-codes MAX_BATCH_CHUNK = 32, but that constant is private to OnnxClassifier and could change later; the comment would then become inaccurate even if the test still passes. Consider making the comment value-agnostic (e.g., “40 texts exceeds the default chunk size”) or deriving the threshold from the implementation if you want to guarantee multi-chunk behavior.

Suggested change

	// arrange — 40 texts forces multiple chunks (MAX_BATCH_CHUNK = 32)
	// arrange — use enough texts to exceed the default chunk size