Skip to content

Custom patterns + configurable guardrails#318

Merged
martsokha merged 3 commits into
mainfrom
feature/custom-patterns
Jul 6, 2026
Merged

Custom patterns + configurable guardrails#318
martsokha merged 3 commits into
mainfrom
feature/custom-patterns

Conversation

@martsokha

Copy link
Copy Markdown
Member

Summary

Adds caller-inlined regex + dictionary rules to the pattern recognizer, with request-scoped ReDoS/compile-cost bounds configurable per deployment.

Wire (nvisy-schema)

PatternRecognizerParams gets two new slots:

```rust
pub struct PatternRecognizerParams {
pub builtins: bool,
pub context_enhanced: bool,
pub custom: Vec, // NEW
pub custom_dictionaries: Vec, // NEW
}
```

New wire types under `nvisy_schema::plan`:

  • `CustomPatternRule`, `CustomPatternVariant`, `CustomPatternContext` — regex-based rules mirroring `elide_pattern::Regex`, including optional `validator` names resolved against elide's `ValidatorRegistry`.
  • `CustomDictionary`, `CustomDictionaryTerm` — literal-term rules mirroring `elide_pattern::Dictionary`.
  • `MAX_REGEX_SOURCE_LEN` — 512-byte cap enforced at deserialize, the wire-side ReDoS anchor.

Types reuse elide's `LabelRef` and `Confidence` directly (both derive `JsonSchema` under `elide-core`'s `schema` feature). Re-exports `LabelRef` from `nvisy_schema::entity` and `Confidence` from `nvisy_schema::primitive` so SDK callers don't need `elide-core` as a separate dep.

Engine (`nvisy-engine`)

```rust
pub struct PatternGuardrails {
pub max_regex_source_len: usize, // default 512
pub max_custom_rules: usize, // default 32
pub max_dictionary_term_count: usize, // default 100_000
pub max_dictionary_term_bytes: usize, // default 8 MiB
}

Engine::new().with_pattern_guardrails(guardrails);
```

  • `max_regex_source_len` is enforced twice: at deserialize in `nvisy-schema` as a hard ceiling, and at engine compile time against the deployment value. `with_pattern_guardrails` clamps values above the ceiling on the way in — deployments can only tighten below the wire limit.
  • `max_custom_rules` — engine-side cap on `custom.len() + custom_dictionaries.len()` per request.
  • `max_dictionary_term_count` / `max_dictionary_term_bytes` — passed to elide's `PatternRecognizerBuilder::with_term_count_limit` / `with_term_bytes_limit` (aggregate across every dictionary in the recognizer, builtin + custom).

Not wired: elide's `with_size_limit` / `with_dfa_size_limit` on the regex automaton. Elide's single-budget API can't separate per-regex NFA (small target) from the shared `RegexSet` union (must fit dozens of shipped builtins). See #317 for the split-budget follow-up.

Analyzer module reorg

Sibling to the feature: `analyzer/common.rs` split by role, matching the wire schema's shape.

Before:

```
analyzer/
mod.rs, catalog.rs, common.rs, guardrails.rs
ner.rs, llm.rs
text.rs, tabular.rs, image.rs, audio.rs
```

After:

```
analyzer/
mod.rs, catalog.rs
text.rs, tabular.rs, image.rs, audio.rs (~30–50 lines each, pure orchestration)
recognizer/{mod, pattern, guardrails, ner, llm}
enricher/{mod, language, ocr, stt}
layer/{mod, dedup}
```

OCR and STT were inline in `image.rs` / `audio.rs`; they now live next to `language` under `enricher/`. `PatternGuardrails` sits next to its owner `pattern.rs`. Public API (`nvisy_engine::PatternGuardrails`) unchanged.

Tests

`custom_patterns.rs` — 7 tests:

  • `custom_regex_produces_entity_with_caller_label` — regex matching the fixture surfaces with the caller-supplied label.
  • `custom_dictionary_produces_entity_with_caller_label` — same for the literal-term dictionary path.
  • `overlong_regex_source_rejects_at_deserialize` — 513-byte regex fails at wire boundary, naming the cap.
  • `too_many_custom_rules_rejects_at_analyze` — 33 rules fail at engine compile, naming the cap.
  • `tightened_rule_count_guardrail_takes_effect` — deployment sets `max_custom_rules: 4`, 5 rules fail with the tightened cap in the error.
  • `tightened_regex_source_len_guardrail_takes_effect` — deployment sets `max_regex_source_len: 16`, 60-byte source fails.
  • `regex_source_len_config_above_ceiling_is_clamped` — deployment sets `4096`, gets clamped to 512; in-code source over the ceiling still rejects.

Existing `multimodal.rs` tests unchanged; `PatternRecognizerParams` literal gets `..Default::default()` for the two new fields.

Verified locally

`make rust-ci` passes (`lint`, `test`, `doc`, `deny`).

Related

  • Elide bump to `b0fbfbad`: shipped `with_size_limit` / `with_dfa_size_limit` / `with_term_count_limit` / `with_term_bytes_limit` on the pattern recognizer builder. This PR consumes the last two.
  • Follow-up guardrails for custom pattern recognizer #317 — follow-up guardrails (split regex budget, AST refuse-list, per-match timeout, per-document budget).

🤖 Generated with Claude Code

**Wire**: `PatternRecognizerParams` gains `custom` +
`customDictionaries` slots for caller-inlined regex rules and
literal-term dictionaries. New wire types in `nvisy-schema`:

- `CustomPatternRule`, `CustomPatternVariant`,
  `CustomPatternContext` — regex-based rules mirroring
  `elide_pattern::Regex`.
- `CustomDictionary`, `CustomDictionaryTerm` — literal-term rules
  mirroring `elide_pattern::Dictionary`.

Types reuse elide's `LabelRef` and `Confidence` (both derive
`JsonSchema` under elide-core's `schema` feature) — no
lossy string/f32 mirror types. Re-exports `LabelRef` from
`nvisy_schema::entity` and `Confidence` from
`nvisy_schema::primitive` so SDK callers don't need `elide-core`.

**Engine**: new `Engine::with_pattern_guardrails` sets a
`PatternGuardrails` config that bounds runaway compile cost.
Four knobs with conservative defaults:

- `max_regex_source_len` (default 512) — enforced twice: at
  deserialize in `nvisy-schema` (`MAX_REGEX_SOURCE_LEN`) as a
  hard ceiling, and at engine compile time against the
  potentially-tightened deployment value. `with_pattern_guardrails`
  clamps to the wire ceiling on the way in.
- `max_custom_rules` (default 32) — cap on total `custom.len() +
  customDictionaries.len()` per request.
- `max_dictionary_term_count` (default 100 000) —
  aggregate across every dictionary in the recognizer, enforced
  by elide's `PatternRecognizerBuilder::with_term_count_limit`.
- `max_dictionary_term_bytes` (default 8 MiB) — same aggregate,
  enforced by `with_term_bytes_limit`.

Regex NFA/DFA size limits are deliberately NOT wired: elide's
single-budget API can't separate per-regex from shared
`RegexSet` union, and the two want very different bounds. See
#317 for the split-budget follow-up.

**Analyzer module reorg**: `analyzer/common.rs` split into
three sibling subfolders that mirror the wire schema's shape:

- `analyzer/recognizer/` — pattern, guardrails, ner, llm.
- `analyzer/enricher/` — language (was in `common.rs`), ocr
  (was inline in `image.rs`), stt (was inline in `audio.rs`).
- `analyzer/layer/` — dedup (was in `common.rs`).

Per-modality entry files (text/tabular/image/audio) shrink to
just orchestration. `PatternGuardrails` sits next to the
pattern module in `recognizer/`.

**Tests** (`custom_patterns.rs`): 7 covering the happy path
(regex + dictionary matching the docx fixture), wire guardrails
(deserialize rejects overlong source, analyze rejects too many
rules), and configurability (tightened rule-count / source-len
take effect, over-ceiling source-len clamps).
@martsokha martsokha self-assigned this Jul 6, 2026
@martsokha martsokha added feat request for or implementation of a new feature engine redaction engine, pipeline runtime, orchestration, configuration recognition pattern, NER, LLM, and OCR backends (elide::recognition::*) refactor code restructuring without behavior change labels Jul 6, 2026
martsokha added 2 commits July 7, 2026 00:32
Split anonymizer/ from 8 flat files into 3 role-based groups
mirroring the analyzer/ shape:

- anonymizer/compile/{dispatch,selector} — modality-generic
  dispatch machinery: Target enum + attach_policies +
  attach_one_override + attribution / predicate builders.
- anonymizer/operator/{text,image,audio} — per-modality
  operator vocabularies: each Op enum + attach_to + build.
- anonymizer/{text,tabular,image,audio}.rs unchanged in role:
  entry files that wire the compile dispatcher with the operator
  builder.

Image and audio's local `Op` enums lift out of the per-modality
entry files into siblings of the existing text_op.rs (renamed
operator/text.rs). Consistent shape across modalities: reader
knows for any modality that entry file dispatches, operator/
holds the vocabulary. Text keeps its shared-with-tabular
character (cells are TextBacked in elide).

Bump crossbeam-epoch 0.9.18 → 0.9.20 for RUSTSEC-2026-0204.
- ImageOp / AudioOp: `impl From<&{Image,Audio}Redaction>` —
  infallible wire → runtime conversion. Call sites use
  `ImageOp::from(spec).attach_to(target)`.
- TextOp: `impl TryFrom<&TextRedaction>` — fallible because
  `Pseudonymize` / `Encrypt` need engine-side infrastructure
  that isn't wired yet. Call sites use
  `TextOp::try_from(spec)?.attach_to(target)`.

Per-modality entry files import the Op enum directly
(`use super::operator::text::TextOp`) instead of the module
(`use super::operator::text as text_op`) — one less alias per
file, matches the enum's role as the module's primary export.
@martsokha martsokha merged commit b0e1c32 into main Jul 6, 2026
6 checks passed
@martsokha martsokha deleted the feature/custom-patterns branch July 6, 2026 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engine redaction engine, pipeline runtime, orchestration, configuration feat request for or implementation of a new feature recognition pattern, NER, LLM, and OCR backends (elide::recognition::*) refactor code restructuring without behavior change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant