Bound SAM3 concept-path postprocessing memory with chunked, cap-first pipeline

[bigbitbus]

Problem

The pinned sam3==0.1.4 package's PostProcessImage has three serving-hostile behaviors on the concept path (segment_with_text_prompts → /sam3/concept_segment):

1. Unbounded detections — we pass max_dets_per_img=-1; every above-threshold instance survives.
2. Full-batch mask interpolation — all k kept masks are upsampled to original resolution as ONE k×1×H×W float32 tensor (~6.5 GiB for 100 masks at 4032²). On GPU OOM, silent try/except fallbacks (interpolate, robust_rle_encode) materialize that batch in host RAM.
3. The cap runs last — the max_dets_per_img topk happens in process_results after all masks are interpolated and encoded, so it bounds only the response, never peak memory. Measured: cap+RLE enabled together still spiked +14.1 GiB host RSS at 100 instances on a 4032² image.

Direct contributor to the host-RAM profile behind the 2026-06-22 async-serverless OOM crash loop; upstream facebookresearch/sam3 has no fix or open issue for any of this.

Fix

Blast radius: concept path only — the visual/interactive path and RF Instant are untouched, and defaults are bitwise-identical to main (verified below).

ChunkedPostProcessImage (new inference_models/models/sam3/chunked_postprocessing.py) subclasses the pinned package's postprocessor — subclassing keeps the sam3 pin untouched and ships the fix on our release cadence. Two methods differ from the parent:

• Cap-first: keep becomes a list of index tensors built whenever a threshold or cap is active — including with thresholding disabled (output_prob_thresh ≤ 0). Capped survivors are score-descending, matching the parent's late-topk ordering exactly; the parent's late topk remains as a no-op safety net.
• Chunked: _process_masks interpolates + encodes in fixed slices (INFERENCE_MODELS_SAM3_MASK_PROCESSING_CHUNK_SIZE, validated, default 8), so neither device ever holds the full k×H×W float32 batch, and the GPU→CPU fallback blast radius is one chunk (~0.5 GiB at 4032²).

Reviewer note: _process_boxes_and_labels intentionally mirrors ~30 lines of the parent with one added block — review it side-by-side with sam3/eval/postprocessors.py, then the sam3_torch.py wiring; the parity tests pin equivalence. When the sam3 pin is bumped, re-diff that method against upstream.

Wiring: segment_with_text_prompts gains max_detections: int = -1 and mask_format: "dense" | "rle" (RLE size stays original image dims, so app consumers are unaffected). max_detections is production-reachable via the new SAM3_MAX_DETECTIONS env (default -1 — no behavior change until an operator opts in). mask_format="rle" is a model-API opt-in for now: the concept adapter's cross-prompt NMS consumes dense masks, and reworking it onto run lists (pycocotools.mask.iou) is a self-contained follow-up. The RLE measurement rows below exercise the model API, not shipped adapter behavior; the shipped default path gets the chunking wins.

Measurements (L4, real sam3/sam3_final weights)

Synthetic scale matrix (k tiled objects, cap 25; "before" = the stock postprocessor with the same knobs forced — main's API hardcoded -1/dense):

scenario	config	RSS peak Δ	wall	CUDA peak
k=100 @4032²	dense uncapped	+3235 → +3271 MiB	2.54 → 3.02 s	17906 → 6370 MiB
k=100 @4032²	cap 25	+1999 → +835	2.18 → 1.04	17906 → 6358
k=100 @4032²	RLE	+14144 → 0.0	5.30 → 0.75	17906 → 6743
k=100 @4032²	cap + RLE	+14140 → 0.0	5.51 → 0.50	17906 → 6731

Mask counts identical in every cell; CUDA peak collapses to the model-forward floor in all configs — chunking alone recovers ~11.5 GiB on the default path.

Real photos (people-walking / bleachers / coin-counting at 4032-wide):

case	masks	payload main→RLE	RSS peak main→RLE	CUDA peak main→branch
walking "person"	47	410 MiB → 0.02 MiB	+857 → +42 MiB	8650 → 5930 MiB
bleachers "person"	34	297 → 0.02	+627 → +66	7740 → 5928
coins "coin"	9	186 → ~0	+420 → +172	6852 → 6688

Real-data parity vs origin/main (5 photos incl. multi-prompt): every mask IoU = 1.0000, score Δ = 0.00e+00, identical counts; branch RLE decodes pixel-identical to baseline dense masks. Re-verified after each review-fix commit.

Tests

78 passed, 0 failed on a GPU box: 50 inference_models — including new test_chunked_postprocessing.py (8 parity tests vs the stock parent: chunk sizes 1/3/>k, RLE counts, cap-as-topk ordering, cap with thresholding disabled, zero-detection shapes, bitwise default parity) and the real-weights integration suite — plus 28 inference-side. Env validation verified: non-integer or < 1 chunk-size values raise InvalidEnvVariable.

Review

Cross-model review findings addressed in 27982b85 (cap without threshold, ordering parity, parity tests) and 4d05b9f4 (validated+documented env var, real help_url, SAM3_MAX_DETECTIONS adapter wiring). Remaining follow-up: adapter-level RLE consumption.

Notes

• Like Fix OWLv2 embeddings cache defeated by re-signed image URLs #2659/Check OWLv2 embeddings cache before materializing reference images #2660, reaching serverless requires an inference-models release + pin bump.
• An upstream issue to facebookresearch/sam3 (cap-before-interpolate + chunked encode) is worth filing; none exists today.

🤖 Generated with Claude Code

[github-actions]

👋 Thanks for the pull request! Here is how automated Claude review works here, so you spend credits (and reviewer time) wisely.

🚦 This PR is marked Ready for review, so automated Claude review will run — and every pass spends real credits.

Warning

💸 The Claude reviewer bills in credits, not vibes

Automated review spins up a real agent that reads real code and spends real credits on every pass. It is glad to help — but it is not a rubber duck, a linter you poke in a loop, or a substitute for reading the contributing guide. Treat it like an expensive senior reviewer whose time you booked, and show up prepared.

Draft when unsure, Ready when you mean it:

• 🌱 Not sure the PR is in good shape yet? Keep it (or set it back) as a draft — drafts pause review, so you can push and iterate without burning credits on a moving target.
• 💪 Feel strong about the contents? Mark it Ready for review and the reviewer will take a look.

However you get there, arrive prepared:

• 🧱 Bring a SOLID, thorough PR. Point your local agent at our skills/ to tune it to our guidelines first — or, if you are one of those fabled carbon-based contributors, read them yourself. A half-baked diff costs exactly the same to review as a finished one.
• ✅ Resolve every comment before you re-request review. Re-requesting with threads still open means paying twice for the same conversation.
• 🔁 Do not use CI review as an inner loop for a local agent. The reviewer is not a step-by-step debugger — do the unfolding locally and arrive with the answer, not the search.
• 🙋 If something looks off, ask a human. One question to a maintainer is cheaper and faster than three rounds of agent re-review chasing a misread.

Reviews are not free. A draft costs nothing to review; a Ready PR is a promise that it is worth reviewing.

• Prefer to skip automated review entirely? Add the skip-claude-review label.

[github-actions]

🤖 Claude review started at commit 27982b85377fb666897a648d65eeae171fd37ed3.

New commits are not auto-reviewed. Add the claude-review label (remove & re-add it to trigger again) when you want another review.

[github-actions]

FLAG — new env var is unvalidated, non-standard-named, and undocumented.

SAM3_MASK_PROCESSING_CHUNK_SIZE is read at module import with a bare int(os.getenv(...)). Three issues:

1. Import-time crash on bad input: a non-integer value (e.g. SAM3_MASK_PROCESSING_CHUNK_SIZE=big) raises ValueError while importing chunked_postprocessing, which is imported by sam3_torch.py — so the SAM3 model fails to load with an unhelpful traceback rather than a validated InvalidEnvVariable. A value like 0 or a negative number silently passes here and is only clamped later by max(1, ...).
2. Naming: the inference_models convention for new env flags is INFERENCE_MODELS_* (see configuration.py); SAM3_MASK_PROCESSING_CHUNK_SIZE does not follow it.
3. Docs: not added to inference_models/docs/how-to/environment-variables.md and no changelog entry.

Please validate via configuration.py (raising InvalidEnvVariable), rename to the INFERENCE_MODELS_* scheme, and document it.

[bigbitbus]

Fixed in 4d05b9f. Moved to configuration.py as INFERENCE_MODELS_SAM3_MASK_PROCESSING_CHUNK_SIZE via get_integer_from_env — non-integer values now raise InvalidEnvVariable (verified: bogus and 0 both produce the proper error with the runtime-environment docs anchor, 0/negatives are rejected with an explicit >= 1 message rather than silently clamped). Documented in docs/how-to/environment-variables.md (Advanced Configuration → SAM3) and added a changelog entry under Unreleased.

[github-actions]

NIT — help_url="https://todo" is a placeholder. Per the inference_models error convention, errors should carry a real docs anchor (https://inference-models.roboflow.com/errors/<page>/#<lowercased-classname>). Please point this at a real anchor or drop the arg.

[bigbitbus]

Fixed in 4d05b9f — now points at https://inference-models.roboflow.com/errors/input-validation/#modelinputerror (same anchor scheme the yolov8 models use for ModelInputError).

[github-actions]

IMPORTANT — these two new params appear unwired to the production concept path.

The only caller of segment_with_text_prompts on the concept path is inference/models/sam3/segment_anything3_inference_models.py::segment_image (line ~118), which invokes it as:

self._model.segment_with_text_prompts(
    images=[np_image],
    prompts=prompt_dicts,
    output_prob_thresh=float(min_threshold),
)

It never forwards max_detections or mask_format, and segment_image itself doesn't accept them from the Sam3SegmentationRequest. So on /sam3/concept_segment these always fall back to max_detections=-1 (uncapped) and mask_format="dense".

Net effect: of the three fixes in the PR, only the chunked interpolation on the dense path is reachable in production. The cap-first truncation and the RLE fast path — which the headline measurements attribute the +14 GiB→0 RSS win to (RLE and cap + RLE rows) — are never exercised by the running endpoint.

Please confirm whether wiring these through the request/adapter is intended for this PR or a follow-up, and if follow-up, note that the RLE/cap measurements don't reflect the shipped behavior.

[bigbitbus]

Good catch — addressed in 4d05b9f, split by lever:

max_detections is now wired. New SAM3_MAX_DETECTIONS env in inference/core/env.py (default -1 = uncapped, so shipped behavior is unchanged until an operator opts in), forwarded by segment_image in segment_anything3_inference_models.py. The cap-first path is reachable in production via deploy config.

mask_format="rle" stays a model-API opt-in for this PR, deliberately. The adapter's cross-prompt NMS (_apply_nms_cross_prompt) and its polygon/RLE response conversion both consume dense arrays today; wiring RLE end-to-end means reworking that NMS to operate on run lists (pycocotools.mask.iou) — a self-contained follow-up already noted in the description.

On the measurements: the RLE / cap + RLE rows exercise the model API directly, as labeled. What ships by default is the chunked dense path — which is the row that removes the CUDA-peak scaling (17.9 → 6.4 GiB) — and the +14 GiB CPU-fallback transient those RLE rows eliminate is only reachable through the same model API that now avoids it. Description updated to state this explicitly.

[github-actions]

@bigbitbus — thanks for the detailed writeup and the memory profiling.

⏳ This PR is ON HOLD pending your answers. The review will not advance to sign-off and the PR should not proceed to merge until the IMPORTANT items below are resolved. Unanswered items may keep this out of a release.

Blocking finding

1. Missing version bump + changelog companions (BLOCK). This is a functional change to inference_models (new module, new segment_with_text_prompts params, new env var), but the PR bumps neither inference_models/pyproject.toml (version still 0.31.0) nor adds a docs/changelog.md entry. Because the change reaches the server (the /sam3/concept_segment path), inference/core/version.py (__version__) should be bumped too. Per the inference_models review standard, a functional change without these is a merge blocker.

IMPORTANT questions

1. Are max_detections / mask_format meant to be wired to /sam3/concept_segment in this PR? (See inline on sam3_torch.py:535.) The concept adapter segment_anything3_inference_models.py::segment_image never forwards them, so production always runs max_detections=-1, mask_format="dense". Only the chunked-interpolation win is actually reachable; the cap-first and RLE wins (the RLE / cap + RLE rows in your table, +14 GiB→0) are not exercised by the endpoint. Is the wiring intended here or a follow-up?

2. Confirm the concept path takes the _process_masks(consistent=False) branch, and that the new list-typed keep is compatible with the pinned parent. ChunkedPostProcessImage only chunks in the not consistent branch; the consistent=True branch defers to the parent and hands it a keep that is now a list of index tensors instead of the parent's boolean mask. Two concerns I cannot verify statically (the sam3==0.1.4 package is not in the checkout):

   • If the concept path ever hits consistent=True, the full-batch interpolation (the thing this PR bounds) runs unchanged — no memory fix — and the parent may index masks with a list-typed keep it expects to be boolean.
   • _process_boxes_and_labels now returns keep as a list of index tensors on the default thresholded path (output_prob_thresh=0.5). Does the parent's process_results consume keep for anything other than the _process_masks call you override (e.g. re-indexing scores/presence tokens as a boolean mask)? If so, output would diverge.

   Please confirm which branch the concept path takes and that keep's type change is safe against the parent's use of it.

3. CI coverage of the parity claims. test_chunked_postprocessing.py is gated by pytest.importorskip("sam3"), and sam3 is the CUDA-only package stubbed out in the unit-test env — so none of the parity/cap/RLE/zero-detection assertions run in CI. The correctness argument therefore rests entirely on manual GPU runs. How do you propose to guard against regressions here (e.g. a scheduled GPU job, or a lightweight stub-backed test exercising the keep/chunk logic without the real package)?

Other (non-blocking, see inline)

• SAM3_MASK_PROCESSING_CHUNK_SIZE: unvalidated (import-time int() crash on bad input), non-INFERENCE_MODELS_* naming, undocumented.
• help_url="https://todo" placeholder in the new ModelInputError.

Re-review

New commits are not auto-reviewed. After you answer the questions and/or push changes, add the claude-review label (remove & re-add to re-trigger) to request a fresh review.

Commands used: gh pr diff, gh api .../{issues,pulls}/2670/{comments,reviews,commits}, and static reads of sam3_torch.py, segment_anything3_inference_models.py, segment_anything3.py, and http_api.py.

Reviewed at HEAD: 27982b8