Reference

Deep detail that does not map 1:1 to a single Config, Log, or Stats control. Per-control detail: UI tooltips. Tab-specific technical depth: Config Panel, Log Panel, Stats Panel.

On this page

• Data Integrity - raw vs derived, OpenSearch write triage, agent HTTP layout, query cookbook; capacity / Failures accounting cross-links
• Live Proxy History-field correlation - downstream evidence, asynchronous History enrichment, run boundaries, and troubleshooting
• Parameter integrity messages - DEBUG log catalog and wiki anchors (Log + Stats cross-surface)
• Export efficiency - cross-cutting outcomes; implementation in Contributor Guide
• Troubleshooting - symptom routing to Config, Log, Stats, OpenSearch wiki

Panel technical detail: Config Panel | Log Panel | Stats Panel

Example placeholders: URLs use opensearch.url. Index names use INDEX_BASE_NAME (default tool-burp).

Related documentation

Step	Where
First run	Home - Quick Start
Config, destinations, indexes, file re-import	Config Panel
Log lines, retries, ingestion errors	Log Panel
Counts, Overview / Traffic Spill, Database Capacity, stats_snapshot, memory	Stats Panel
Local OpenSearch + Dashboards	OpenSearch wiki - Home
Build and internals	Contributor Guide
Cross-panel integrity and queries	This page

OpenSearch API: Bulk API | Bulk streaming | Java client bulk guide

Data Integrity

This section helps answer the first question raised by an integrity log: was raw data lost? For Parameter Integrity entries, the normal answer is no: raw request/response evidence is preserved, and the exporter adjusted derived helper fields such as request.parameters to avoid indexing junk or misleading data. OpenSearch write/retry/drop entries are different; those can mean indexed documents are missing or delayed.

Overview

• Raw HTTP evidence is the primary fidelity guarantee. Files retain the full prepared document. body.b64 is the exact captured wire representation; body.decoded_b64 is the deepest successfully decoded Content-Encoding representation when at least one layer was removed. Database export normally retains both available representations and WebSocket payload bytes, but may prefix-truncate large search fields under bulk-budget pressure; affected objects set truncated=true, meta.search_truncated=true, and length fields retain the original captured wire length.
• Derived fields such as request.parameters, request.body.text, response.body.text, request.path.query, burp.repeater.*, and exporter stats are convenience/search/analysis surfaces. They can be intentionally absent, capped, null, or best-effort without implying raw evidence was lost. Binary decoded data remains available in decoded_b64 even when text is null.
• Most high-cardinality or odd BODY-parameter signals in testing and research traffic come from fuzzing, analytics, stats, tracking, generated fields, compressed bodies, or intentionally noisy endpoints. Treat them as derived-field hygiene unless the host/endpoint and helper field matter to the analysis objective.
• Exporter stats_snapshot data, LogPanel summaries, session stop Stats JSON, and parameter_integrity_detail documents are validation surfaces. LogPanel gives the bottom line and exact wiki anchor; OpenSearch drill-down is optional and useful only when the affected derived data may matter.

Raw Evidence vs Derived Fields

This distinction separates derived-field adjustments from raw evidence loss:

• Wire evidence: Files preserve full body.b64 and the captured length. Database documents normally do too; when search-only pressure fitting activates, body.b64 can contain a retained prefix while body.length stays the original wire length and body.truncated=true marks the difference.
• Decoded evidence: body.decoded_b64 preserves the deepest successful progressive decode, even if the remaining representation is binary or a later encoding layer fails. body.content_encoding.applied, .remaining, and .complete make partial decoding explicit. Up to eight layers are attempted with an 8 MiB expanded-output bound per layer.
• Derived text: body.text is searchable and useful for LLM/agent workflows, but it is conditional and derives from the deepest decoded bytes. A missing body.text usually means the representation is binary or the media type is not appropriate for text extraction; it does not mean the bytes were thrown away.
• Derived parameters: request.parameters is a search helper built from Montoya parameter APIs plus exporter safeguards. It may be capped, supplemented, or intentionally incomplete while raw body bytes remain complete.
• Best-effort metadata: Repeater tab/group fields and timing on non-Proxy live paths depend on Montoya/Swing surfaces. Token-bound live Proxy HTTP attempts exact Proxy History enrichment during the bounded window described below; a complete downstream exchange can still export with null History-only fields when no unique match is available. Null on a documented best-effort field can mean "not safely attributable," not "export failed."
• Export Stats: exporter stats_snapshot and parameter_integrity_detail documents explain what happened during export; they are not application traffic evidence.

Logging

Parameter integrity and related Log panel messages: Parameter integrity messages.

Burp-first triage questions:

1. Is the host/endpoint in scope?
2. Does the test depend on request.parameters, URL/query helpers, or another derived helper field?
3. If not, the entry is usually informational and no OpenSearch query is needed.
4. If it does, the Stats row and exact wiki anchor from the log line provide the next level of detail.
5. OpenSearch drill-down can add evidence when the expected derived data may affect a finding or validation set.

Severity policy: DEBUG for routine derived-field hygiene; WARN/ERROR when action may be needed. Full message catalog: Parameter integrity messages.

OpenSearch write may be missing or delayed

These entries affect whether documents reached OpenSearch or whether exact-search helper fields are less complete.

permanent_drops

Definition: OpenSearch rejected a document in a way that retrying will not fix under the current mapping/data shape.

Impact: OpenSearch may be missing documents. Non-zero permanent drops are actionable. file export may still have written the document if file sinks are enabled, but OpenSearch is missing the rejected document.

Example: retry coordinator logs include [PERMANENT], and Stats rows show non-zero Permanent Drops.

When to investigate: investigate any non-zero permanent drop count.

How to validate: grep exporter logs for [PERMANENT], check Stats Permanent Drops / Permanent Drop Reasons, and inspect sparse event.data.stats.permanent_drop_reason_counts in exporter stats_snapshot docs.

Expected benign cases: none for release-quality runs. Test runs that intentionally push malformed docs clean up the resulting data and document the reason.

retry_queue_growth

Definition: OpenSearch writes are queued for retry because a push failed or the destination is temporarily unavailable.

Impact: OpenSearch may be behind. Queued docs are not fully indexed until retry pressure drains. pending OpenSearch docs may lag behind file output and Stats exported counters.

Example: Stats rows Queue Depth and Oldest Queued Age grow while Consecutive Failures remains non-zero or last success is stale.

When to investigate: investigate if age grows continuously, one index stalls while others drain, or Stop does not flush the queue.

How to validate: use Stats Queue Depth, Oldest Queued Age, Consecutive Failures, and per-index retry/drop rows. Check OpenSearch connectivity and credentials.

Expected benign cases: short cluster outages, TLS handshake hiccups, temporary 429/502/503/504 / timeout-style capacity pressure (soft outage + shared cooldown), or Stop waiting for in-flight pushes. Failures can rise and later match Recovered Failures without document loss when the retry drain succeeds.

immense_term_rejected

Definition: Lucene rejects a keyword value longer than 32,766 UTF-8 bytes if the mapping does not guard it.

Impact: Raw _source can preserve the value, but exact keyword search/aggregation/sort may miss over-long values. current mappings set ignore_above: 8191 on keyword fields so oversized values stay in _source and are skipped only from the keyword inverted index.

Example: OpenSearch returns Document contains at least one immense term.

When this matters: review if the error appears in logs or if exact-match queries miss very long values.

How to validate if it matters: search exporter logs for immense term; check whether the failed field is a very long keyword helper. After an extension upgrade, delete stale indexes and Start again if schema errors persist.

Common benign causes: very long headers, URLs, cookies, payload-derived strings, or evidence blobs that are best read from _source instead of keyword fields.

nested_document_limit_exceeded

Definition: OpenSearch's default nested-object limit is 10,000 nested children per document; huge parameter/header/cookie arrays can exceed it.

Impact: OpenSearch may be missing the affected document. This is a write-blocking integrity problem if it appears. the affected OpenSearch document is rejected unless the exporter caps/filtering prevented the oversized nested arrays first.

Example: OpenSearch returns nested documents has exceeded.

When to investigate: investigate any occurrence. Current caps and synthetic BODY filtering are designed to prevent this without raising the mapping limit.

How to validate: search logs for nested documents has exceeded, check permanent drops, and review Parameter Integrity caps/drops.

Expected benign cases: none in normal operation.

Count and reconciliation logs

Stop reconciliation lines ([Stats] Session stop, final stats push): Stats Panel — Stop reconciliation. Compare with OpenSearch validation below.

Viewing limitations

discover_large_body_highlighting_failures

Definition: Dashboards/Kibana Discover can fail to render large evidence bodies even though OpenSearch stored the document.

Impact: This is a viewer limitation, not an exporter write failure. writes, _doc fetches, non-highlight _search, reindex, snapshot/restore, and embedding pipelines that read _source are unaffected.

Example: search_phase_execution_exception -> parse_exception -> stream_constraints_exception with a 50,000,000-character string ceiling.

When this matters: review if analysts rely on Discover preview/highlighting for large bodies or findings with many nested evidence pairs.

How to validate if it matters: fetch by _id or run _search without highlighting. In Dashboards/Kibana, set doc_table:highlight to false; for backend parsing limits, raise opensearch.json.max_string_length (or Elasticsearch equivalent) and restart the backend.

Common benign causes: large request/response bodies, large findings evidence arrays, and wildcard highlighting on body fields.

Canonical HTTP document layout for agents

The Traffic index, Sitemap HTTP documents, Findings requests_responses, and Findings Collaborator HTTP sub-documents now share the same agent-oriented HTTP shape:

• request.url is an object: raw, text, scheme, host, port, path, query, and fragment. Use request.url.raw (or requests_responses.request.url.raw inside Findings) for exact URL filters. Use request.url.text for full-text URL search, including long or fuzzed query strings.
• request.headers and response.headers are ordered nested lists. Each row has name, raw, value, and ordinal. name is normalized for grouping; raw preserves the original header name casing Burp exposed. Duplicate headers stay as separate rows instead of dynamic fields.
• request.cookies is an ordered nested list parsed from Cookie headers: name, value, raw, and ordinal.
• response.cookies is an ordered nested list parsed from Set-Cookie headers when available, including domain, path, expires, max_age, secure, http_only, same_site, partitioned, raw, and ordinal. The raw Set-Cookie line remains the authoritative evidence when attributes are malformed or unusual. expires and max_age are typed helpers for date/range queries: recognized Expires forms may be normalized to ISO for indexing, while max_age remains the observed seconds string in _source and indexes as a long when parseable.
• request.parameters remains the parameter list for URL/BODY/JSON/XML-style parameters. COOKIE rows are intentionally excluded and live under request.cookies.
• request.content_type is a derived/declared content-type object, not a header entry. It includes raw header value, parsed media type, charset, Burp-declared value/enum, and exporter inference.
• response.mime_type is a MIME/content-type object with raw Content-Type, parsed media type, charset, Burp MIME, stated MIME, and body-inferred MIME.
• response.header_attributes.date is a typed helper for the standard response Date header. The raw header row remains in response.headers; common HTTP date strings are indexed as dates, while malformed dates are preserved in _source and skipped from the date index instead of rejecting the document.
• Removed dynamic HTTP header fields include request.header.<name> and response.header.<name>. Use nested header rows instead.

WebSocket traffic uses websocket.message_type for the type supplied by Burp and websocket.inferred_message_type for the exporter's payload-byte classification. Historic Proxy WebSocket rows expose websocket.history_has_edited_payload; live Proxy frames expose websocket.change_stages when both received and about-to-be-sent payloads were observed. websocket.payload.b64 remains the byte-preserving representation even when text is unavailable.

Field Selection and Partial Export

• The Fields panel controls document _source, not the search database mapping. Unchecked optional leaves are stripped before database/file export; required fields remain.
• Partial-field exports remain valid sparse documents. Optional fields can be absent or null, especially large bodies, headers, cookies, Repeater metadata, and derived text fields.
• Disabling a field can reduce storage/bandwidth but can also remove inputs an agent expected. Check the active field selection before treating absent optional fields as evidence that Burp did not capture something.
• Re-imported file output uses the same field filter that OpenSearch export used during the original run.

Mapping and Field Guarantees

• Indexes are created on Start with fixed, bundled mappings (not user-editable). Mapping JSON under src/main/resources/search/mappings/ is applied at index creation only.
• Document shape is built in Java. The mapping describes what OpenSearch can index; it does not decide which fields producers emit.
• The Fields panel controls document _source only. Unchecked optional leaves are stripped before export; the index schema is unchanged.
• Required metadata fields (meta.schema_version, meta.extension_version, meta.indexed_at) are always present on exported documents.
• Files preserve complete body evidence. Database pressure fitting may shorten body.b64, body.decoded_b64, and body.text while preserving original wire length and setting body.truncated=true plus meta.search_truncated=true.
• Upgrade / schema mismatch: If indexes predate an extension upgrade, delete affected indexes and Start again so current bundled mappings apply (including body.b64, body.decoded_b64, body.content_encoding.*, and optional body.text).

Export size limits

These limits prevent one derived field or document from destabilizing the export. They do not all have the same integrity impact:

Mechanism	Effect	Threshold	Integrity impact
URL / BODY parameter lists	Keeps the first 1,000 structured parameters. Raw URL and body evidence remain available unless the separate search-pressure fitter later activates.	1,000 entries per source/type	Derived parameter enumeration can be incomplete.
Content-Encoding decompression	Stops safely at the first unsupported, malformed, or over-limit layer; wire body.b64 and the deepest successful body.decoded_b64 remain available.	8 MiB per step; 8 layers	body.content_encoding.remaining explains unfinished work; body.text may be absent while binary decoded bytes remain.
Mapping ignore_above	Keeps the value in _source but omits the over-long keyword term from exact search, sorting, and aggregation.	8,191 characters on guarded keyword fields	Exact .raw queries may miss the value; _source retains it.
Live database bulk budget	Derives a search-only fitted copy, prefix-truncating bodies/payloads and then other large fields/lists as needed.	Adaptive 512 KiB-5 MiB request ceiling	Indexed payload can be incomplete; truncated=true, original length, and meta.search_truncated=true make this queryable. Files remain full fidelity.
Absolute maximum fit	Permanently rejects a search document only when the smallest supported fitted shape still exceeds 5 MiB.	5 MiB	Database document is absent; rich ERROR and max_fit_exceeded reason identify it. Files copy remains unchanged.
Mapping/parse/validation rejection	Stops retrying deterministic poison work.	Destination rejection	Database document is absent; inspect Permanent Drop Reasons and ERROR detail.
Traffic queue/spill/retry admission	Rejects new work when bounded memory/disk rails are exhausted.	Live runtime budgets	Document may never be exported; Stats and overflow summaries record the loss.
File/disk guard	Stops or refuses file writes before configured/volume limits are exceeded.	Configured file cap plus free-space rules	File path stops; database export may continue when still enabled.

Useful database queries:

meta.search_truncated:true
response.body.truncated:true
response.body.truncated:true AND response.body.length:>1048576

For Findings evidence, apply the same checks under requests_responses.request.body or requests_responses.response.body. Length is the original captured size, not the retained prefix size.

Malformed typed helpers

Some traffic-controlled fields are useful as typed OpenSearch helpers, but Burp traffic can still contain fuzzed or malformed values. These helpers use ignore_malformed: true where OpenSearch supports it, including request.url.port, response.status.code, response.header_attributes.date, response.cookies.expires, and response.cookies.max_age. Date helpers accept common HTTP/cookie date strings such as Fri, 27 Feb 2026 04:49:21 GMT, Mon, 6 Apr 2026 14:59:38 GMT, Mon, 05-Oct-26 17:18:25 GMT, Sun, 13-Sep-2026 05:22:25 GMT, Fri, 05-Feb-2027 20:32:14 GMT+0000, and 05 Feb 2028 20:32:01 Z. Browser-style cookie dates such as Fri Sep 18 2026 01:39:10 GMT+0000 (Coordinated Universal Time) are normalized into the typed helper while the original Set-Cookie line stays in response.cookies.raw.

When OpenSearch cannot parse one of those helper values, it indexes the rest of the document, keeps the original value in _source, and records the skipped helper in _ignored. This preserves the exporter's evidence-transport role: raw rows such as response.headers and response.cookies.raw remain available, while typed helper queries simply exclude malformed values.

The _ignored metadata field helps show whether malformed helper values affected a query. OpenSearch 3.6 supports exists and exact term queries on _ignored, but not terms aggregations on that metadata field.

{
  "query": {
    "term": {
      "_ignored": "response.cookies.expires"
    }
  }
}

For a broad check that any helper was ignored, use:

{
  "query": {
    "exists": {
      "field": "_ignored"
    }
  }
}

Document Identity, Dedupe, and Re-import

Document identity (meta and Bulk action _id)

• meta in _source: Every exported document includes meta.schema_version, meta.extension_version, and meta.indexed_at. Search-only fitting additionally sets meta.search_truncated=true when large fields are shortened for a database request.
• Bulk action _id: Each newly prepared export operation receives a client-generated UUID. The identifier is action metadata, not _source, and remains stable across retries, spill, search fitting, and the matching Files NDJSON pair.
• Bulk NDJSON files: Each document is two lines: {"index":{"_id":"..."}} followed by the JSON body. _index is omitted so the user selects the destination through /{index}/_bulk.

Deduplication policy

Mechanism	Scope	Purpose
Prepared-operation _id	One logical prepared operation across Database, Files, retry, spill, and fitted search copies	Repeated delivery of the same operation updates the same destination document instead of creating another copy. Re-sending the same exported file therefore reuses its IDs.
New export run	Cross-run	New preparation creates new UUIDs. Append-mode Files output retains earlier pairs and adds the new run's pairs; destination _count can therefore grow across runs.
Repeater Tabs captureKey / fingerprint	Single export run	Suppresses duplicate startup tab captures when hooks or Swing walks observe the same tab again.
Findings / Sitemap periodic seen keys	Single export run	Thirty-second polls skip rows already exported this session; finding keys include issue detail so distinct issues are not collapsed.
Proxy WebSocket live callbacks	Single export run	Exports final frames observed while the Proxy source and export run are active.

Files integrity and manual destination upload responsibilities: Config Panel - Formats and Manual Bulk API upload. Stats counter and _count interpretation: Stats Panel - Export accounting.

Live Proxy History-field correlation

Live Proxy HTTP traffic has several useful moments in Burp:

1. Proxy request/response callbacks, which expose received and about-to-be-sent representations;
2. the global HTTP response callback, where the exporter can build the live exchange document; and
3. the Proxy History row, where Burp exposes History identity, listener, edited status, and timing metadata.

Those callbacks use different ID namespaces. The exporter therefore does not join them by messageId, URL, method, timestamp proximity, or FIFO order. At the global HTTP request callback it appends a namespaced UUID marker to Burp annotations. Proxy-specific callbacks carry the same annotations through the Proxy pipeline. The background reconciler first searches token-bearing History rows and can fall back to a uniquely matching final-request/listener/time candidate. It never chooses arbitrarily between multiple candidates.

Field contract

The exported request and response are the furthest downstream representations observed: request-to-be-sent and response-to-be-sent when Burp exposes them. Originals are not emitted as extra documents because they were not necessarily sent or delivered. The burp.proxy fields describe what metadata and comparisons were actually observable:

Field	Meaning
burp.proxy.history_id	ProxyHttpRequestResponse.id() after a unique History bind; otherwise null
burp.proxy.listener_port	Proxy listener port when observed, including before History enrichment
burp.proxy.history_is_edited	Burp's edited flag on the bound History row; null when no History row was bound
burp.proxy.request_change_stages	RECEIVED_TO_SENT when both stages were observed and differ; empty when observed and equal; null when comparison was unavailable
burp.proxy.response_change_stages	Ordered UPSTREAM_TO_RECEIVED and/or RECEIVED_TO_SENT changes; empty when all observed stages were equal; null when comparison was unavailable
burp.timing.req_sent	History timing when bound; otherwise unavailable
burp.timing.end	History request-sent time plus request-to-response-end duration when bound
burp.timing.req_sent_to_res_start	History request-to-response-start duration when bound
burp.timing.req_sent_to_res_end	History request-to-response-end duration when bound

burp.message_id remains the live HTTP handler's source-local ID. It is useful metadata, but it is not the Proxy History join key.

The marker is private transport metadata. The exporter removes only its generated marker from the latest live and History annotation values. User notes remain in Burp and burp.notes; the generated marker is independently redacted from any document built before cleanup. Proxy WebSocket live frames follow the same downstream principle and record observable changes in websocket.change_stages; History WebSocket snapshots retain Burp's History edit metadata but do not invent unavailable live stages.

Asynchronous path and Burp latency

The correlation path is deliberately outside Burp's traffic-critical work:

• request/response callbacks perform bounded in-memory marker and admission bookkeeping;
• callbacks never scan Proxy History, estimate the full document size, write spool files, or wait for OpenSearch/Files;
• one coalesced background worker reads newly appended History rows in the normal case;
• defensive all-History scans are limited to fresh misses and the slower disk-backed lane;
• destination queue handoff occurs after exact enrichment or after the bounded History-unavailable fallback and marker cleanup.

The request and response passing through Burp are not delayed for correlation. Only visibility in the export destination can lag while the matching History row is joined.

Callback boundaries are fail-open for Burp: an exporter runtime exception is logged without propagating into Burp traffic processing. Correlation never delays or blocks the proxied exchange.

Memory, temporary spooling, and restart

Fresh completed responses wait for History enrichment within a 15-second correlation window. That duration begins after the response document is admitted; it is not measured from request send. Heap pressure can cause earlier temporary spooling. Once spooled, the heavy document map is released from the in-memory pending entry and loaded from disk only when binding or handoff needs it.

Spool files are current-run JSON envelopes below the active extension instance and Burp project: {java.io.tmpdir}/burp-exporter/instances/<instance-id>/projects/<project-id>/proxy-correlation. The root and all managed children must be non-symlink, owned by the running principal, and restricted to that owner. Every extension instance holds an operating-system ownership lock; startup removes only abandoned unlocked instance trees.

The exporter-managed 1 GiB per-project storage ceiling is partitioned across concurrent instances:

• 768 MiB for ordinary live-traffic spill;
• 256 MiB for unresolved Proxy correlation.

The free-space guard separately preserves 1 GiB of free disk on the volume. Cross-process quota files serialize reservations. A rejected spool write remains pending within the active run and retries with backoff; Spool / Explicit Failures and Log identify the condition.

Stop, restart, and delivery ownership

The first Stop click closes new correlation intake, then waits within the shared shutdown budget for response callbacks admitted before Stop. The active export run token remains valid while already accepted destination work drains. Stop then discards unresolved memory and spool state.

The next Start always opens a clean correlation generation after discarding stale in-memory and on-disk state. Interrupted or crashed-run files are never replayed. Generation fencing prevents old or blocked workers from offering into or mutating a later run. Successful handoff retires the spool file; unload releases the instance lock and removes its disposable tree on a best-effort basis.

Accuracy and troubleshooting rules

• Preserve downstream traffic evidence: after the correlation window, an otherwise complete final exchange is exported even when History-only identity/timing/edit metadata is unavailable.
• Never guess a History row: a unique token or fallback candidate enriches the document; zero or multiple candidates leave History-only fields unavailable rather than selecting arbitrarily.
• Listener validation: a token row from another Proxy listener is rejected.
• Generation fencing: responses from an inactive generation cannot enter the next run.
• No cross-run replay: Stop, unload, and startup cleanup discard unresolved spill/correlation state instead of exporting it during a later run.
• No token disclosure: normal Log lines identify generation, callback messageId, counts, and failure reason without printing the private marker or spool filename.

Operational surfaces:

• Log Panel — Live Proxy correlation lines
• Stats Panel — Proxy Correlation
• event.data.traffic.proxy_correlation in exporter stats_snapshot documents

For a healthy controlled run, HTTP Marked / Responses should balance, Bound / Eligible should converge, pending counts should return to zero, and lookup/cleanup/spool/explicit failures should remain zero. A non-zero durable count means current-run work temporarily moved out of heap; it is not persistent storage and does not survive the run boundary.

Parameter integrity messages

In-app Stats Parameter Integrity tooltips and DEBUG [ParameterIntegrity] log lines point at anchors on this page (Log lines show Wiki: Reference#…).

Category anchor	Typical Stats row
misgate_binary	Mis-gate Suspects
skip_path_rescued	Skip-path Rescued
wire_replaced	Wire BODY Replaced
supplemental_added	Supplemental BODY Used
supplemental_rejected_non_form	Supplemental Rejected (non-form)
wire_dropped	Wire BODY Dropped (entries)
body_params_truncated	BODY cap (pairs with Wire BODY Dropped)
compressed_wire_body_params	Rollup for wire replace / drop / supplement family

Integrity-related LogPanel lines are intentionally concise: they give the outcome first, include an exact wiki anchor, and mention OpenSearch drill-down only when the affected endpoint or derived field matters.

Burp-first triage:

1. Is the host/endpoint in scope?
2. Does the test depend on request.parameters, URL/query helpers, or another derived helper field?
3. If no, stop; no OpenSearch query is needed.
4. If yes, check the Stats row and the exact wiki anchor from the log line.
5. Use OpenSearch drill-down only when the expected derived data may affect a finding or validation set.

Severity policy:

• DEBUG covers common derived-field hygiene, routine corrections, and optional drill-down pointers. Keep a single issue family at one LogPanel level so level filtering does not split the summary from its category details.
• WARN is reserved for unexpected live events or integrity failures that may deserve attention in the current Burp session.
• Strong investigation language is reserved for OpenSearch write integrity issues where indexed documents may be missing or delayed.

parameter_integrity_detail documents use event.type=parameter_integrity_detail with event.level=DEBUG and include event.data.category, event.data.phase, event.data.doc_count, event.data.unique_url_count, event.data.sample_urls, event.data.impact, and Lucene query hints. Categories with useful full URL review also include chunked event.data.urls lists (chunk / chunk_count). supplemental_added records representative sample URLs only because the full list is usually not useful for manual validation.

Lucene in OpenSearch Dashboards stays consistent with API query-string searches. The _search API can add detail when the Wiki or Stats answer is not enough and the derived field matters.

Derived-field limitations / review when relevant

These entries say the exporter preserved raw evidence while adjusting a derived helper field. They usually do not require action during a Burp workflow.

body_params_truncated

Definition: more than 1,000 BODY parameters; first 1,000 kept in request.parameters. Impact: No raw loss; wire body.b64, decoded body.decoded_b64 when present, and eligible body.text stay available. When it matters: in-scope endpoints where parameter count or names after the cap matter. Validate: event.type:parameter_integrity_detail AND event.data.category:body_params_truncated | Stats: BODY cap rows | Detail: event.data.urls

url_params_truncated

Definition: more than 1,000 URL parameters; path/query helpers rebuilt from retained set. Impact: No body loss; URL/query helpers capped. When it matters: signed URLs, gateway behavior, or query-completeness tests. Validate: event.type:parameter_integrity_detail AND event.data.category:url_params_truncated | Stats: URL cap rows

misgate_binary

Definition: declared form/multipart but logical body sniffed binary; BODY enumeration skipped. Impact: No raw loss; request.parameters BODY rows may be absent by design. When it matters: in-scope hosts expected to contain real form fields. Validate: event.type:parameter_integrity_detail AND event.data.category:misgate_binary | Stats: Mis-gate Suspects

wire_dropped

Definition: suspect Burp-derived BODY rows removed after wire vs logical comparison. Impact: No raw loss; dropped entries removed from request.parameters. When it matters: compressed forms where BODY parameter search helpers matter. Validate: event.type:parameter_integrity_detail AND event.data.category:wire_dropped | Stats: Wire BODY Dropped (entries)

supplemental_rejected_non_form

Definition: logical bytes looked like JSON/protobuf/non-form; exporter refused fake form rows. Impact: No raw loss; no synthetic BODY parameters. When it matters: endpoints expected to be urlencoded forms after decompress. Validate: event.type:parameter_integrity_detail AND event.data.category:supplemental_rejected_non_form | Stats: Supplemental Rejected (non-form)

unexpected_body_enumeration_gate

Definition: WARN — BODY enumeration skipped for declared form/multipart though inference was not binary. Impact: The raw body remains available; this is unusual on application traffic. When it matters: any occurrence on in-scope application traffic. Validate: Log search BODY enumeration skipped despite declared form; compare wire body.b64, decoded body.decoded_b64, chain metadata, and body.text.

No raw data lost; derived field adjusted

These entries are usually informational.

compressed_wire_body_params

Definition: DEBUG rollup grouping wire replace/drop/supplement categories. Impact: No raw loss. Drill into: wire_replaced, wire_dropped, supplemental_added, supplemental_rejected_non_form, skip_path_rescued.

wire_replaced

Definition: compressed wire; BODY rows replaced from parseable logical urlencoded bytes. Impact: No raw loss; corrected request.parameters. Validate: event.type:parameter_integrity_detail AND event.data.category:wire_replaced | Stats: Wire BODY Replaced

supplemental_added

Definition: BODY rows added from safe logical urlencoded bytes when Burp supplied none. Impact: No raw loss; detail docs use sample URLs only. Validate: event.type:parameter_integrity_detail AND event.data.category:supplemental_added | Stats: Supplemental BODY Used

skip_path_rescued

Definition: skip-path ran but parseable form fields recovered into request.parameters. Impact: No raw loss. Validate: event.type:parameter_integrity_detail AND event.data.category:skip_path_rescued | Stats: Skip-path Rescued

synthetic_body_parameter_drops

Definition: synthetic/nonsensical Burp BODY rows dropped before indexing. Impact: No raw loss. Validate: session stop Stats JSON / stats_snapshot synthesized-body counters.

Export efficiency, stability, and observability

User-visible outcomes from export-efficiency work. Log signals: Log Panel. Metrics: Stats Panel. stats_snapshot queries: Stats Panel - Exporter stats_snapshot.

• Queue-backed traffic export with bounded memory and explicit backpressure
• Live Proxy correlation performs only bounded admission on Burp callbacks; History lookup, document sizing, temporary spooling, and destination handoff run asynchronously
• Prepare-once documents reused for OpenSearch, file, Traffic Spill, and retry paths
• If the live byte budget shrinks while a bulk waits for pacing, derived search copies are re-fitted and re-split before HTTP; original prepared documents and Files output stay unchanged.
• Soft Outage paces the destination; Traffic Spill absorbs live-traffic overflow until Full (then reject new live traffic — preserve earliest; not drop-oldest). Snapshot indexes do not disk-spill.
• Exporter-managed storage is owner-restricted, instance-locked, and capped per Burp project at 1 GiB across concurrent instances: 768 MiB for Traffic Spill and 256 MiB for live Proxy correlation
• Run-scoped spill/correlation artifacts are disposable: Stop/unload clears current-run state and startup discards abandoned crash artifacts instead of replaying them
• Search-destination diagnostics redact known credentials and echoed secret values, cap response-body previews at 4 KiB, cap assembled response diagnostics at 8 KiB, and summarize exceptions without unbounded stack traces
• Adaptive batch sizing under cluster stress; shared soft outage + cooldown (5s → 15s → 30s → 60s) and shared AIMD bulk byte budget across OpenSearch / Amazon OpenSearch / Elasticsearch; bounded retry queues and outage mode
• Parallel snapshot export with shared chunk tuning, per-lane prepared-byte/throughput history for the next startup slice allowance, and periodic deduplication

Implementation detail (orchestration, traffic offload/batching, batch controller API): Contributor Guide — Export orchestration · Traffic export path · Adaptive batch size.

Troubleshooting

Symptom	Look here
No cluster / connection refused	OpenSearch Home; Config Test Connection tooltip
Base URL rejected before connection	Supply only an absolute HTTP(S) root endpoint: scheme, host, and optional port. Remove paths, query strings, fragments, and embedded credentials. Config — Destinations
Start fails on existing index settings	When Log names a confirmed mismatch, the index uses shard, replica-allocation, or refresh settings that do not match the selected deployment. Start does not mutate or delete it; correct the settings or delete/recreate the index. HTTP 401/403 on the optional _settings inspection now warns and continues unverified for least-privilege principals. Config - Control actions
Authorization disappears during a run	Database enters Authorization Recovery instead of stopping. Queued retries remain retained; automatic probes use increasing backoff. After access returns, selected index settings/mappings are revalidated and reproducible snapshots are replayed to Database only when the cluster changed or indexes were recreated. Config - Credential lifetime; Log - Export retries
Backlog done but database not caught up	Log — Run health; Stats — Failures vs Recovered
Live Proxy documents absent or delayed	Check Stats HTTP Marked / Responses, Bound / Eligible, pending memory/durable counts, and all correlation failure rows. Exact History enrichment is attempted for 15 seconds; a complete downstream exchange can then export with History-only metadata unavailable. Live Proxy correlation
Live Proxy durable count keeps growing	Check Log [ProxyCorrelation] warnings/errors, Proxy History availability, listener mismatches, the 256 MiB per-project correlation-spool share, and free disk. The count is current-run temporary storage and will not replay after Stop. Stats — Proxy Correlation
No [SnapshotExport] after Start	Log — Run checklist
Amazon throughput gap / chart lull	Log — Amazon signals; Stats — Database Capacity; Stats FAQ
Soft outage / shared cooldown / "export stopped?"	Stats FAQ; capacity pressure uses pacing, while repeated AUTH pauses for automatic recovery without disabling Database
Remote Database: chart lull / Failures ahead of Recovered	Config — Monitoring remote database exports; wait for backlog + recovery (often >> 5 minutes); Stats — Database Capacity
Soft Outage + Traffic Spill Ready/In use/Full	Soft Outage paces; Traffic Spill absorbs live overflow until Full (reject new). Stats — Overview, Stats — Traffic Spill, Config — Monitoring
Stop took too long / Permanent Drops on Stop	Graceful Stop ~20s wall-clock; undrained retries become Permanent Drops; second Stop is force-abort. Config — Control actions
Failures > Recovered after Stop	Review Permanent Drops and Stop logs. Undrained retry work is discarded at the run boundary; a later Start does not resume it. Use Files alone for a separately importable fallback rather than Files+Database under pressure.
Credentials or destination values appear in diagnostics	Current destination paths redact known session credentials, sensitive headers/parameters, and common echoed-field patterns. Preserve the log and treat any remaining secret exposure as a bug. Log — What to expect
Temporary / session credentials mid-run	Prefer Amazon IAM SigV4 - Profile; Config — Credential lifetime
DEBUG [ParameterIntegrity]	Hover Stats row; Parameter integrity messages
ERROR / [PERMANENT]	Log — Indexing failures
Exported vs _count mismatch	Stats — Stop reconciliation; OpenSearch API; remember cumulative _count and nested Lucene inflation (Stats FAQ)
Discover empty	OpenSearch Dashboards — pattern, time field, range
Discover Bad Request (large body)	Dashboards — Known limits