Skip to content

feat: response/request examples, enhanced path params, OPTIONS filter, redaction#49

Merged
Arkptz merged 16 commits into
mainfrom
feat/schema-examples-params-redaction
May 27, 2026
Merged

feat: response/request examples, enhanced path params, OPTIONS filter, redaction#49
Arkptz merged 16 commits into
mainfrom
feat/schema-examples-params-redaction

Conversation

@Arkptz
Copy link
Copy Markdown
Owner

@Arkptz Arkptz commented May 27, 2026

Summary

Four new capabilities added to the generate and discover pipeline, plus a fix for large mitmproxy flows.

Path parameterizationis_param_segment() now detects UPPER_CASE slugs (BTC_USDT), hex strings (0xabcdef...), and base58 identifiers in addition to numeric/UUID. Cross-request variability detection: when ≥3 distinct values appear at the same path position, that segment is parameterized automatically. --param-regex re-introduced with a working implementation (was previously accepted but silently ignored).

Examples collection — all captured response and request bodies are stored as named examples in the generated OpenAPI spec under MediaType::examples. Multiple captures of the same endpoint produce multiple named examples. Request body schemas are merged across captures via oneOf (same strategy already used for responses).

OPTIONS filtering--skip-options flag on both discover and generate drops OPTIONS requests from templates and spec. Off by default.

Redaction--redact-fields and --redact-patterns apply to stored examples only, never to schema inference. Example names are re-derived from the redacted value so sensitive data does not leak into YAML keys. --max-examples N (default 5) caps examples per endpoint per status code.

Large flow handling--max-payload-size wired through to the tnetstring parser (was a dead flag). Oversized entries are now skipped instead of halting the iterator, so flows after a large WebSocket/streaming entry are no longer silently dropped.

Test evidence

  • 303 unit + integration tests, 0 failures
  • cargo clippy --all-targets -- -D warnings: clean
  • End-to-end pipeline test in tests/full_pipeline.rs exercises all features together
  • Manual QA with 3.3GB mitmproxy capture: --max-payload-size 1GiB recovers fair_price/index_price/funding_rate paths previously lost behind an 800MB WebSocket entry (54734 → 82861 flows, paths=1 → paths=4)

Files changed

  • src/path_matching.rs, src/type_hints.rs: UPPER_CASE/hex/base58 detection, variability algorithm
  • src/builder.rs: examples accumulator, request body merge, redaction integration, max-examples cap
  • src/redact.rs (new): standalone redaction module
  • src/tnetstring.rs: skip oversized entries instead of halting iterator
  • src/mitmproxy_reader.rs: accept max_payload_size parameter
  • src/cli.rs, src/types.rs, src/main.rs: new flags wired through
  • book/src/usage/cli-reference.md, pipeline.md, introduction.md, README.md, CHANGELOG.md: docs updated

@Arkptz Arkptz merged commit 242c0a8 into main May 27, 2026
14 checks passed
@Arkptz Arkptz deleted the feat/schema-examples-params-redaction branch May 27, 2026 12:00
@Arkptz Arkptz mentioned this pull request May 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant