Custom patterns + configurable guardrails#318
Merged
Conversation
**Wire**: `PatternRecognizerParams` gains `custom` + `customDictionaries` slots for caller-inlined regex rules and literal-term dictionaries. New wire types in `nvisy-schema`: - `CustomPatternRule`, `CustomPatternVariant`, `CustomPatternContext` — regex-based rules mirroring `elide_pattern::Regex`. - `CustomDictionary`, `CustomDictionaryTerm` — literal-term rules mirroring `elide_pattern::Dictionary`. Types reuse elide's `LabelRef` and `Confidence` (both derive `JsonSchema` under elide-core's `schema` feature) — no lossy string/f32 mirror types. Re-exports `LabelRef` from `nvisy_schema::entity` and `Confidence` from `nvisy_schema::primitive` so SDK callers don't need `elide-core`. **Engine**: new `Engine::with_pattern_guardrails` sets a `PatternGuardrails` config that bounds runaway compile cost. Four knobs with conservative defaults: - `max_regex_source_len` (default 512) — enforced twice: at deserialize in `nvisy-schema` (`MAX_REGEX_SOURCE_LEN`) as a hard ceiling, and at engine compile time against the potentially-tightened deployment value. `with_pattern_guardrails` clamps to the wire ceiling on the way in. - `max_custom_rules` (default 32) — cap on total `custom.len() + customDictionaries.len()` per request. - `max_dictionary_term_count` (default 100 000) — aggregate across every dictionary in the recognizer, enforced by elide's `PatternRecognizerBuilder::with_term_count_limit`. - `max_dictionary_term_bytes` (default 8 MiB) — same aggregate, enforced by `with_term_bytes_limit`. Regex NFA/DFA size limits are deliberately NOT wired: elide's single-budget API can't separate per-regex from shared `RegexSet` union, and the two want very different bounds. See #317 for the split-budget follow-up. **Analyzer module reorg**: `analyzer/common.rs` split into three sibling subfolders that mirror the wire schema's shape: - `analyzer/recognizer/` — pattern, guardrails, ner, llm. - `analyzer/enricher/` — language (was in `common.rs`), ocr (was inline in `image.rs`), stt (was inline in `audio.rs`). - `analyzer/layer/` — dedup (was in `common.rs`). Per-modality entry files (text/tabular/image/audio) shrink to just orchestration. `PatternGuardrails` sits next to the pattern module in `recognizer/`. **Tests** (`custom_patterns.rs`): 7 covering the happy path (regex + dictionary matching the docx fixture), wire guardrails (deserialize rejects overlong source, analyze rejects too many rules), and configurability (tightened rule-count / source-len take effect, over-ceiling source-len clamps).
Split anonymizer/ from 8 flat files into 3 role-based groups
mirroring the analyzer/ shape:
- anonymizer/compile/{dispatch,selector} — modality-generic
dispatch machinery: Target enum + attach_policies +
attach_one_override + attribution / predicate builders.
- anonymizer/operator/{text,image,audio} — per-modality
operator vocabularies: each Op enum + attach_to + build.
- anonymizer/{text,tabular,image,audio}.rs unchanged in role:
entry files that wire the compile dispatcher with the operator
builder.
Image and audio's local `Op` enums lift out of the per-modality
entry files into siblings of the existing text_op.rs (renamed
operator/text.rs). Consistent shape across modalities: reader
knows for any modality that entry file dispatches, operator/
holds the vocabulary. Text keeps its shared-with-tabular
character (cells are TextBacked in elide).
Bump crossbeam-epoch 0.9.18 → 0.9.20 for RUSTSEC-2026-0204.
- ImageOp / AudioOp: `impl From<&{Image,Audio}Redaction>` —
infallible wire → runtime conversion. Call sites use
`ImageOp::from(spec).attach_to(target)`.
- TextOp: `impl TryFrom<&TextRedaction>` — fallible because
`Pseudonymize` / `Encrypt` need engine-side infrastructure
that isn't wired yet. Call sites use
`TextOp::try_from(spec)?.attach_to(target)`.
Per-modality entry files import the Op enum directly
(`use super::operator::text::TextOp`) instead of the module
(`use super::operator::text as text_op`) — one less alias per
file, matches the enum's role as the module's primary export.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds caller-inlined regex + dictionary rules to the pattern recognizer, with request-scoped ReDoS/compile-cost bounds configurable per deployment.
Wire (
nvisy-schema)PatternRecognizerParamsgets two new slots:```rust
pub struct PatternRecognizerParams {
pub builtins: bool,
pub context_enhanced: bool,
pub custom: Vec, // NEW
pub custom_dictionaries: Vec, // NEW
}
```
New wire types under `nvisy_schema::plan`:
Types reuse elide's `LabelRef` and `Confidence` directly (both derive `JsonSchema` under `elide-core`'s `schema` feature). Re-exports `LabelRef` from `nvisy_schema::entity` and `Confidence` from `nvisy_schema::primitive` so SDK callers don't need `elide-core` as a separate dep.
Engine (`nvisy-engine`)
```rust
pub struct PatternGuardrails {
pub max_regex_source_len: usize, // default 512
pub max_custom_rules: usize, // default 32
pub max_dictionary_term_count: usize, // default 100_000
pub max_dictionary_term_bytes: usize, // default 8 MiB
}
Engine::new().with_pattern_guardrails(guardrails);
```
Not wired: elide's `with_size_limit` / `with_dfa_size_limit` on the regex automaton. Elide's single-budget API can't separate per-regex NFA (small target) from the shared `RegexSet` union (must fit dozens of shipped builtins). See #317 for the split-budget follow-up.
Analyzer module reorg
Sibling to the feature: `analyzer/common.rs` split by role, matching the wire schema's shape.
Before:
```
analyzer/
mod.rs, catalog.rs, common.rs, guardrails.rs
ner.rs, llm.rs
text.rs, tabular.rs, image.rs, audio.rs
```
After:
```
analyzer/
mod.rs, catalog.rs
text.rs, tabular.rs, image.rs, audio.rs (~30–50 lines each, pure orchestration)
recognizer/{mod, pattern, guardrails, ner, llm}
enricher/{mod, language, ocr, stt}
layer/{mod, dedup}
```
OCR and STT were inline in `image.rs` / `audio.rs`; they now live next to `language` under `enricher/`. `PatternGuardrails` sits next to its owner `pattern.rs`. Public API (`nvisy_engine::PatternGuardrails`) unchanged.
Tests
`custom_patterns.rs` — 7 tests:
Existing `multimodal.rs` tests unchanged; `PatternRecognizerParams` literal gets `..Default::default()` for the two new fields.
Verified locally
`make rust-ci` passes (`lint`, `test`, `doc`, `deny`).
Related
🤖 Generated with Claude Code