primitives smart replacement map

Smart replacement map

Active contributors: Douwe de Vries

Purpose

Smart replacement map is the core primitive that stores accepted Local AI replacements for selected columns. It validates model output, reuses replacements for repeated source values, records rejection reasons, and lets preview replacements carry into final output when they cover selected values.

Directory layout

Path	Role
crates/csv-anonymizer-core/src/smart.rs	Defines Smart replacement requests, provider trait, replacement map, validation, batching, missing-value checks, and CSV or row collection.
src-tauri/src/local_ai/provider.rs	Implements SmartReplacementProvider for Ollama and converts provider requests into prompt calls.

Key abstractions

• SmartReplacementProvider is a core trait with generate_replacements.
• SmartReplacementRequest includes a column and a batch of source values.
• SmartReplacement is a provider-returned original and replacement pair.
• SmartReplacementEntry is the serializable DTO shared with the frontend.
• SmartReplacementMap stores replacements by column index and normalized source value.
• SmartReplacementRejectionReason records invalid outputs such as unexpected source, missing output, empty output, same-as-original, contains-original, control character, duplicate source, and duplicate output.
• SMART_REPLACEMENT_VALUE_CAP_PER_COLUMN limits collected unique values per selected Smart column.

How it works

sequenceDiagram
    participant Rows as Rows or CSV
    participant Smart as smart.rs
    participant Provider as SmartReplacementProvider
    participant Map as SmartReplacementMap
    participant Strategy as TransformState

    Rows->>Smart: collect unique selected Local AI values
    Smart->>Map: reuse existing preview entries
    Smart->>Provider: generate missing replacements in batches
    Provider-->>Smart: original/replacement pairs
    Smart->>Smart: validate replacements
    Smart->>Map: insert accepted replacements and counts
    Map-->>Strategy: replacement lookup during transform

Loading

Preview builds a replacement map from sampled rows and returns SmartReplacementEntry values to the frontend. Final CSV output receives those preview entries and checks whether they cover all selected Smart values. If values are missing, the backend needs a ready provider and requests only missing values. During transformation, TransformState looks up a replacement by column index and normalized original; a miss records a fallback and uses rule-based pseudonymization.

Integration points

• Transform strategy uses the map for the localAi strategy and records fallback counts.
• Privacy report includes accepted values, rejected candidates, rejection reasons, and fallback counts.
• Tauri command shell creates the provider for preview, paste, quick, and anonymize requests.
• Background jobs prepares Smart replacements before streaming full CSV output.
• src-tauri/src/local_ai/provider.rs sends prompts to Ollama through /api/generate and parses structured response JSON.

Entry points for modification

• Change provider trait shape, batching, value collection, value caps, or missing-value detection in crates/csv-anonymizer-core/src/smart.rs.
• Change validation rules or rejection reasons in crates/csv-anonymizer-core/src/smart.rs and mirror DTO changes in frontend/src/types.ts.
• Change Ollama request construction, response parsing, or provider enablement in src-tauri/src/local_ai/provider.rs.
• Change prompt schema or prompt text in src-tauri/src/local_ai/prompt.rs.
• Change Local AI status or model download behavior in Background jobs and related Local AI modules.

Key source files

• crates/csv-anonymizer-core/src/smart.rs
• src-tauri/src/local_ai/provider.rs
• src-tauri/src/local_ai/prompt.rs