Content Extractors Limits and Overrides

Content Extractors — Limits and Overrides

The TMI server extracts text from operator-trusted documents (DOCX, PPTX, XLSX, PDF, HTML, plain text) so Timmy and the rest of the content pipeline can index it. The OOXML extractors (DOCX, PPTX, XLSX) ship with a layered set of limits: operator-tunable defaults, hardcoded server ceilings, and per-user concurrency overrides. This page is the operator's reference for those controls and the failure-classification codes they produce.

The content_extractors.* limits on this page are operational settings — DB-backed, seeded from defaults on first run, and editable at runtime via the admin API. See Configuration-Model for how operational config is stored and edited, Managing-Operational-Settings for the admin-API workflow, and Configuration-Reference for the full content_extractors.* key list.

What runs and when

The content pipeline is invoked when a threat model imports or refreshes a document via a configured content source (Google Drive, OneDrive, Confluence, etc.). Each fetched byte stream goes through:

1. Source fetch — bounded by the SafeHTTPClient's body cap (operator-configured, separate from this page).
2. Extractor dispatch — based on declared content type, the pipeline routes to the matching extractor: plain text, HTML, PDF, DOCX, PPTX, XLSX.
3. Per-user concurrency limiter — the user owning the document holds a semaphore slot for the duration of the extraction.
4. Wall-clock budget — a single hard timeout caps how long any extraction may run.
5. Output — Markdown (≤ the configured size cap) emitted to the caller.

The OOXML extractors enforce additional structural caps on top of the byte caps — slide count, cell count, ZIP nesting depth, XML element depth, ZIP compression ratio, and so on. Most are hardcoded ceilings, not operator-tunable; see the table below.

Defaults

The values below are the project-wide defaults from internal/config/content_extractors.go::DefaultContentExtractorsConfig. These are what your server uses if no content_extractors: block exists in YAML and no TMI_CONTENT_EXTRACTORS_* environment variable is set.

Knob	Default	Hardcoded ceiling
compressed_size_bytes	20 MiB	50 MiB
decompressed_size_bytes	50 MiB	100 MiB
part_size_bytes	20 MiB	50 MiB
pptx_slides	100	250
xlsx_cells	1000	10000
markdown_size_bytes	128 KiB	256 KiB
wall_clock_budget	30 s	60 s
per_user_concurrency_default	2	16

If the YAML or environment value exceeds the ceiling, the server refuses to start with a clear validation error. Defaults below the ceiling are deliberately conservative — bump them when you have evidence (real users hitting them, or trusted machine accounts with known-large documents).

Hardcoded ceilings (server-only)

These are not operator-tunable. They are constants in internal/config/content_extractors.go and api/content_extractor_ooxml_common.go. They exist to bound worst-case resource usage even if an operator misconfigures the tunable knobs.

Constant	Purpose
maxCompressedSizeBytes (50 MiB)	Hard ceiling on the on-disk OOXML container size.
maxDecompressedSizeBytes (100 MiB)	Hard ceiling on the sum of all decompressed parts inside the OOXML zip. Limits zip-bomb damage.
maxPartSizeBytes (50 MiB)	Hard ceiling on any single part inside the zip.
maxPPTXSlides (250)	Hard ceiling on slide count for PPTX.
maxXLSXCells (10 000)	Hard ceiling on populated cell count across all sheets for XLSX.
maxMarkdownSizeBytes (256 KiB)	Hard ceiling on the extractor's emitted Markdown.
maxWallClockBudget (60 s)	Hard ceiling on the per-extraction wall-clock timeout.
MaxPerUserConcurrency (16)	Hard ceiling on per-user concurrent extractions, including override values.

Additional structural limits live in the OOXML common module:

• ZIP nesting depth cap (rejects nested-archive bombs).
• ZIP path traversal guard (rejects entries with .. segments or absolute paths).
• ZIP compression-ratio cap (rejects entries with absurd decompression ratios).
• XML element-depth cap (rejects billion-laughs / quadratic-blowup XML).

Configuration

YAML (config-development.yml, config-production.yml, etc.)

content_extractors:
  compressed_size_bytes: 20971520     # 20 MiB
  decompressed_size_bytes: 52428800   # 50 MiB
  part_size_bytes: 20971520           # 20 MiB
  pptx_slides: 100
  xlsx_cells: 1000
  markdown_size_bytes: 131072         # 128 KiB
  wall_clock_budget: 30s              # any time.ParseDuration string
  per_user_concurrency_default: 2

Any field omitted falls back to its default.

Environment variables

Each YAML key has a matching env var. Env vars override YAML.

YAML key	Environment variable
compressed_size_bytes	TMI_CONTENT_EXTRACTORS_COMPRESSED_SIZE_BYTES
decompressed_size_bytes	TMI_CONTENT_EXTRACTORS_DECOMPRESSED_SIZE_BYTES
part_size_bytes	TMI_CONTENT_EXTRACTORS_PART_SIZE_BYTES
pptx_slides	TMI_CONTENT_EXTRACTORS_PPTX_SLIDES
xlsx_cells	TMI_CONTENT_EXTRACTORS_XLSX_CELLS
markdown_size_bytes	TMI_CONTENT_EXTRACTORS_MARKDOWN_SIZE_BYTES
wall_clock_budget	TMI_CONTENT_EXTRACTORS_WALL_CLOCK_BUDGET
per_user_concurrency_default	TMI_CONTENT_EXTRACTORS_PER_USER_CONCURRENCY_DEFAULT

Boot-time validation (ContentExtractorsConfig.Validate) rejects any value that's ≤ 0 or above its hardcoded ceiling.

Per-user concurrency override

Most users get the operator-configured default (per_user_concurrency_default, default 2). A trusted machine account that legitimately drives many concurrent extractions — for example, an automation that bulk-imports a document corpus — can be granted a higher allowance via a per-user override stored on the users table.

When to use

Only for trusted machine principals (service accounts, addons, internal automations) where you've measured the load and know the server can absorb it. Don't raise individual human users' overrides for convenience; bump the global default if many users are hitting it.

Grant an override

UPDATE users
   SET extraction_concurrency_override = 8
 WHERE internal_uuid = '<uuid>';

The lookup is a SELECT extraction_concurrency_override FROM users WHERE internal_uuid = ? cached for the lifetime of the user's first extraction in the process. Override changes don't resize an existing semaphore — restart the server (or wait for the user's worker to drop out of cache) to apply the new value. This is a known limitation of the limiter in cmd/server/main.go::buildContentPipeline.

The override is hard-clamped to MaxPerUserConcurrency (16). A value above 16 in the database is silently treated as 16. A NULL override means "use the operator default."

Revert an override

UPDATE users
   SET extraction_concurrency_override = NULL
 WHERE internal_uuid = '<uuid>';

Find users with overrides

SELECT internal_uuid, email, extraction_concurrency_override
  FROM users
 WHERE extraction_concurrency_override IS NOT NULL;

Failure-classification reason codes

When extraction fails, the document row's access_status is set to extraction_failed and access_reason_code records the specific failure. tmi-ux localizes user-facing messages by this stable code; operators read it to triage.

These constants live in api/access_diagnostics.go:

Reason code	What happened
extraction_limit:compressed_size	The on-disk OOXML container exceeded compressed_size_bytes. Bump that knob if the value is legitimate.
extraction_limit:decompressed_size	Sum of decompressed parts exceeded decompressed_size_bytes. Either a legitimately large document or a zip bomb — inspect first.
extraction_limit:part_size	A single part inside the zip exceeded part_size_bytes.
extraction_limit:part_count	The OOXML container has more parts than the structural cap allows.
extraction_limit:markdown_size	The extractor's emitted Markdown exceeded markdown_size_bytes. Bump that knob if the document genuinely has that much text.
extraction_limit:timeout	Extraction ran past wall_clock_budget. Bump the budget or investigate why the document is so slow to parse (often a malformed XML structure).
extraction_limit:xml_depth	XML element depth exceeded the structural cap. Almost always a malformed document or an attempted XML bomb.
extraction_limit:zip_nested	OOXML zip contains another zip past the nesting cap. Almost always a zip bomb.
extraction_limit:zip_path	A zip entry has a .. segment or an absolute path. Reject — never bump.
extraction_limit:compression_ratio	A zip entry's decompression ratio is implausibly high. Reject — never bump.
extraction_malformed	The extractor parsed the file but the structure was invalid (e.g., an XLSX missing its workbook part).
extraction_unsupported	The pipeline routed the document to an extractor that doesn't support its declared subtype.
extraction_internal	An unexpected error inside the extractor. Investigate via server logs.

Most extraction_limit:* codes are operator-tunable: bump the knob if the limit is legitimate. The path-traversal, nested-zip, and compression-ratio codes are never tunable — they exist to reject hostile input.

Troubleshooting

"All my users' extractions are failing"

Likely causes, in order of likelihood:

1. The document source upstream is unreachable — check access_status first. If it's not extraction_failed, the failure is upstream of the extractor (token revoked, source down, etc.). See Delegated Content Providers.
2. The server is under load and the per-user limiter is queueing extractions past the wall-clock budget. Check server CPU and the oncall log for extraction queue depth warnings.
3. An operator just changed a YAML knob below a value users were already exceeding. Roll back the knob.

Investigating a single failed extraction

SELECT
  id, name, content_type, access_status, access_reason_code, access_status_updated_at
FROM documents
WHERE access_status = 'extraction_failed'
  AND threat_model_id = '<tm-uuid>'
ORDER BY access_status_updated_at DESC
LIMIT 25;

The access_reason_code column tells you which limit (if any) was hit. Cross-reference with the table above.

For deeper debugging, consult /logs (see the operator Cloud Logging page) — the extractor logs the reason-code tag plus any underlying error message at WARN or ERROR level.

Confirming a config change took effect

Restart the server and check the startup banner. Look for the OOXML extractors enabled log line:

OOXML extractors enabled: docx, pptx, xlsx (per-user concurrency default: 2, wall-clock budget: 30s)

Both numbers should match what you put in YAML or env vars. If they don't, the server fell back to defaults — re-check the env-var name (capitalization is significant).

References

• internal/config/content_extractors.go — defaults, ceilings, validator.
• api/access_diagnostics.go — reason-code constants.
• api/content_pipeline.go — pipeline wiring.
• cmd/server/main.go::buildContentPipeline — runtime assembly and the per-user override lookup.
• Originating issue: #287 (OOXML extractors feature).