v0.1.0

v0.1.0 (2026-04-09)

Bug Fixes

• fix(ci): use pip instead of uv in semantic-release build command (#14)

fix(ci): use pip install build in semantic-release build command

The python-semantic-release action runs in its own Docker container
which doesn't have uv installed. Changed the build_command to use
standard pip and python instead of uv.

Fixes GitHub Actions Release workflow failure where 'uv: command not found'
was causing the build to fail with exit code 127. (4bc8b2c)

• fix: include database name in URI normalization for validation

Updated _normalize_uri to include database name in comparison. This ensures
that connections to the same MongoDB cluster but different databases are
allowed, while still preventing replication to the same database. (ca10e65)

• fix: rename parent_collection and child_collection to parent and child in scan command (2697b5a)

• fix: remove incorrect parent_collection/child_collection fallback

SchemaRelationshipConfig uses 'parent' and 'child' attributes, not 'parent_collection' and 'child_collection'. The hasattr check was unnecessary and the fallback was causing AttributeError. (436d8b6)

• fix: add python-dotenv dependency and fix exception handling

Issues fixed:

1. .env file not being loaded because python-dotenv was not installed

   • Add python-dotenv>=1.0.0 to dependencies in pyproject.toml
   • Improve error handling in main.py to warn if dotenv import fails
   • Explicitly check for .env file existence and load from cwd

2. Wrong exception type caught in scan and run commands

   • JobManager.get_job() raises ValueError, not KeyError
   • Fix scan.py to catch ValueError instead of KeyError
   • Fix run.py to catch ValueError instead of KeyError
   • Display actual error message from ValueError for better UX

All 286 tests passing. (c88d76f)

• fix: make CollectionsConfig subscriptable

Add getitem, contains, get(), items(), keys(), and values() methods
to CollectionsConfig RootModel to enable dictionary-style access.

This fixes the test failure where config.replication.collections['users']
was not working. (425afb8)

• fix: fix scan command to sample docs (b64b363)

• fix: update Presidio tests to match new YAML structure and caching

• Update test YAML configs to use recognizer_registry.recognizers hierarchy
• Fix test_get_analyzer_lazy_initialization to check 'default' cache key
• Update test_load_invalid_yaml_raises_error to use truly invalid structure
• All 270 tests now passing (4363f61)

Chores

• chore: update to use PAT for release workflow (d96dc03)

• chore: minor change (ffa869f)

• chore: minor change (640e698)

• chore: fix some run issues (8c312e3)

• chore: update jinja template (5ac3357)

• chore: update sacn command (a20de90)

• chore: update jinja2 template and defaults for cursor detection (23b2b3c)

• chore: change cursor_initial_value (7ccfac9)

• chore: minor change to init.py (0621fa7)

• chore: minor change to jinja2 template (fc5db25)

• chore: refactor config with pydantic models (4498c75)

• chore: remove unrelated test file from PR (6ac6886)

• chore: update config variables from REP_ to MONGOREP_ (f423191)

• chore: set _id and meta.* fields in pii allowlist (skip) (7777e21)

• chore: update reference to cli (c2703e5)

• chore: update cascade_filter.py to support query based selection

Rename --select option to --ids
Introduce --query option (d400f61)

• chore: Update logic to detect cursor field automatically on scan (d8a15b0)

• chore: Rename loader.py to manager.py (9192820)

• chore: Move config models in models.py (7d2f3fc)

• chore: Homogenize replication defaults config with generated config (f0ec9cf)

Documentation

• docs: add banner image to README (4a64901)

• docs: add comprehensive Presidio documentation and update references

• Add new docs/presidio.md with complete Presidio operator documentation

  • All 16 operators documented (built-in + custom)
  • Default entity-to-operator mappings
  • Strategy aliases
  • Custom YAML configuration examples (healthcare, finance, e-commerce)
  • Architecture overview and usage examples

• Update docs/configuration.md

  • Replace outdated default strategies with correct mappings
  • Reference presidio.md for detailed operator documentation
  • Simplify Custom Presidio Configuration section
  • Update PII analysis settings with correct operator names

• Update README.md files

  • Replace outdated strategy mappings with current operators
  • Reference presidio.md for comprehensive documentation
  • Update examples to use correct operator names

• Update docs/technical-design.md

  • Add note referencing presidio.md for implementation details
  • Keep high-level architecture overview (0ebe4c4)

• docs: add schema relationship analysis documentation

• Add Schema Relationship Analysis section to configuration.md with detailed explanation
• Add Cursor Detection Settings section to configuration.md
• Update README.md to mention schema relationship inference in features
• Update scan description to include relationship detection
• Document configuration options, patterns, and usage examples (a326cde)

• docs: fix docs (c54025e)

• docs: update documentation to reflect new config structure

• Update README.md with new scan and replication structure
• Update docs/configuration.md with detailed new config sections
• Rename schema -> schema_relationships in examples
• Split scan.pii into scan.sampling and scan.pii_analysis
• Reorganize replication into discovery, state_management, performance, and defaults
• Update field names: fallback_cursor -> cursor_fallback_field, initial_value -> cursor_initial_value
• Fix config_template.yaml.j2 to use new structure
• Update defaults.yaml to remove duplicate entries (3bb7edb)

• docs: update README.md (9b2d3a9)

• docs: update README.md (12d5cd3)

• docs: update documentation (d709608)

• docs: update documentation (e70b5a5)

• docs: update configuration documentation (b4a9ab7)

• docs: update README.md (c937959)

• docs: update documentation (b59614c)

Features

• feat(cli): improve signal handling for graceful interruption

Add comprehensive signal handling to gracefully handle SIGINT (Ctrl+C),
SIGTERM, and EOFError (Ctrl+D) across all CLI commands.

Changes:

• Add signal_handler.py utility with SignalHandler context manager
• Register global signal handlers in main.py for SIGINT and SIGTERM
• Wrap init command with exception handling for KeyboardInterrupt and EOFError
• Update scan and run commands to handle EOFError in addition to KeyboardInterrupt
• Extract init wizard logic into _run_init_wizard() for cleaner error handling

Benefits:

• Graceful shutdown on Ctrl+C with proper cleanup
• Handles Ctrl+D (EOFError) in interactive prompts
• Consistent exit code 130 for user interruptions (standard for SIGINT)
• Clean error messages without stack traces
• Signal handlers registered at application level for global coverage

Exit codes:

• 0: Success
• 1: Error/failure
• 130: User interruption (SIGINT/Ctrl+C)

Addresses issue with signals not being trapped properly when running commands. (a62302c)

• feat(cli): show default anonymization strategies in init wizard

Enhance the 'mongorep init' command UX to display default anonymization
strategies for each PII entity type, making it clearer for users what
will happen to detected PII data.

Changes:

• Step 5: Entity type selection now shows default strategy next to each
  type (e.g., 'EMAIL_ADDRESS → smart_redact')
• Step 6: Replaced hardcoded outdated strategies with actual defaults
  from DEFAULT_ENTITY_STRATEGIES imported from presidio_anonymizer
• Updated available operators description to reflect both built-in and
  custom operators from presidio.yaml
• Expanded strategy selection choices from 3 to 10 common operators
  including custom ones (fake_email, fake_name, smart_redact, etc.)

This aligns the CLI UX with the recent Presidio operator refactoring
and makes the configuration process more transparent and user-friendly. (f8a6579)

• feat: refactor PII anonymization to use Presidio AnonymizerEngine with YAML-configured operators

• Add YAML configuration for anonymization operators (presidio.yaml)
• Create custom Presidio operators: fake_email, fake_name, fake_phone, fake_address, fake_ssn, fake_credit_card, fake_iban, fake_us_bank_account, stripe_testing_cc, smart_redact
• Implement PresidioConfig parser to convert YAML to OperatorConfig objects
• Refactor presidio_anonymizer.py to properly use Presidio's AnonymizerEngine.anonymize()
• Update PIIHandler to use new PresidioAnonymizer instead of pii_redaction module
• Remove deprecated PIIRedactor and redact_document exports
• Rewrite test_presidio_anonymizer.py to test new implementation (29 tests, all passing)
• Remove deprecated test_pii_redaction.py
• All 271 tests passing

This change moves from a fragile custom implementation to proper Presidio integration with operator-based configuration, improving maintainability and extensibility. (8e99af2)

• feat: validate source and destination databases are different

Adds validation to prevent source and destination MongoDB connections from
pointing to the same database, which could cause data corruption.

Changes:

• Add validation in ConnectionManager.init() that compares normalized
  URIs and database names
• Implement URI normalization to extract host:port for comparison
• Update init, run, and scan commands to catch and handle ValueError
• Raise clear error message if source and destination match

All 286 existing tests continue to pass. (f363a25)

• feat: enhance scan report with Jinja2 template and additional sections

• Convert PII report to comprehensive scan report using Jinja2
• Add cursor field detection section with sample values
• Add schema relationships section with Mermaid diagram
• Rename report from pii_report.md to scan_report.md
• Track cursor field info during detection for reporting
• All 286 tests passing (9ddc211)

• feat: sync discovery config between scan and replication

• Update init command to set both scan.discovery and replication.discovery with same include/exclude patterns
• Update run command interactive mode to filter collections by discovery patterns
• Scan command already respects discovery patterns via CollectionDiscovery
• All 286 tests passing (ca0e6e1)

• feat: add automatic schema relationship inference during scan

• Add SchemaRelationshipAnalyzer class to detect parent-child relationships between collections based on field name patterns (e.g., customer_id -> customers)
• Integrate analyzer into scan command as Step 6, running after PII analysis when scan.schema_relationships.enabled is true
• Add configuration option in init command (Step 9) to enable/disable relationship analysis
• Update config models with ScanSchemaRelationshipsConfig and add scan.schema_relationships section to template
• Inferred relationships are saved to root-level schema_relationships in generated config for cascade replication
• Support snake_case (customer_id), camelCase (customerId), nested fields (meta.order_id), and plural/singular conversions
• Add 16 comprehensive unit tests for analyzer (all passing, 286 total tests) (163f5af)

• feat: add cursor_detection config and improve init command flow

• Add scan.cursor_detection.cursor_fields to config model and defaults
• Update Jinja template to include cursor_detection section
• Add cursor_fields question in init command (Step 4 of 9)
• Reorganize init command steps to group all PII-related questions (Steps 5-8)
• Update scan.py to read cursor_fields from scan.cursor_detection config
• All 270 tests passing

This allows users to customize which cursor field candidates are checked
during scan for incremental replication, with sensible defaults:

• updated_at
• updatedAt
• meta.updated_at
• meta.updatedAt (e1f1afe)

• feat: make PII analysis respect scan.pii.enabled config flag

• Read scan.pii.enabled from existing config if present
• Precedence: --no-pii CLI flag > scan.pii.enabled config > default (enabled)
• Update banner to show PII status source (--no-pii, config, or enabled)
• Add informative messages when PII is skipped
• Always save pii config section (not conditionally based on --no-pii)
• Save enabled=false when --no-pii is passed, enabled=true otherwise

This allows users to disable PII analysis by setting enabled: false in
their config file, without having to pass --no-pii on every scan. (5805d82)

• feat: add Presidio YAML configuration support for custom PII recognizers

Add comprehensive YAML-based configuration system for Presidio PII detection, allowing users to define custom PII recognizers without writing Python code.

Key Features:

• Default presidio.yaml with common recognizers (EMAIL, PHONE, SSN, CREDIT_CARD, etc.)
• Custom recognizer examples (Bank Account, Employee ID, Patient ID)
• Smart path resolution (absolute → cwd → config/ → default)
• Fail-fast validation with clear error messages
• Analyzer caching by config path for performance
• Support for regex patterns and deny-lists
• Context-aware detection with context words

Changes:

• Add presidio_config field to ScanPIIConfig model
• Update PresidioAnalyzer to load from YAML using AnalyzerEngineProvider
• Update PIIAnalyzer to pass config path through chain
• Update scan command to read/save presidio_config
• Update init command to ask about custom Presidio config (now 8 steps)
• Update config manager to load/save presidio_config field
• Update defaults.yaml with presidio_config documentation
• Add 300+ lines of documentation with examples in docs/configuration.md
• Add 15 unit tests covering path resolution, YAML loading, and caching

Files Modified:

• src/mongo_replication/config/models.py
• src/mongo_replication/engine/pii/presidio_analyzer.py
• src/mongo_replication/engine/pii/pii_analyzer.py
• src/mongo_replication/cli/commands/scan.py
• src/mongo_replication/cli/commands/init.py
• src/mongo_replication/config/manager.py
• src/mongo_replication/config/defaults.yaml
• docs/configuration.md

Files Added:

• src/mongo_replication/config/presidio.yaml (392 lines)
• tests/unit/test_presidio_config.py (414 lines)

Total: 1589 insertions, 61 deletions across 10 files
All tests passing ✓ (a20b7df)

Refactoring

• refactor(cli): load anonymization strategies from presidio.yaml config

Remove hardcoded DEFAULT_ENTITY_STRATEGIES and load strategies dynamically
from the presidio.yaml configuration file (default or user-provided).

Changes:

• Move Presidio config question earlier in init wizard (right after 'Enable PII Analysis?')
• Add load_entity_strategies_from_config() helper to read strategies from YAML
• Entity type selection now shows strategies from the loaded config
• Strategy configuration uses loaded config instead of hardcoded dictionary
• Remove DEFAULT_ENTITY_STRATEGIES from presidio_anonymizer.py
• Add TEST_ENTITY_STRATEGIES in test_presidio_anonymizer.py for unit tests
• Remove DEFAULT_ENTITY_STRATEGIES from init.py exports

Benefits:

• Single source of truth: presidio.yaml is the only place to define strategies
• Respects user's custom config when provided
• No need to maintain duplicate hardcoded mappings
• Tests use TEST_ENTITY_STRATEGIES to avoid config file dependency

This addresses feedback to read the actual configuration file rather than
maintaining a separate hardcoded dictionary. (585ebd2)

• refactor: rename pii_report.py to scan_report.py and update function name

• Rename pii_report.py to scan_report.py to better reflect comprehensive nature
• Rename generate_pii_report() to generate_scan_report()
• Update import in scan.py
• All 286 tests passing (3392f9a)

• refactor: rename 'Schema Relationship Analysis' to 'Schema Relationship Inference'

Update terminology throughout codebase and documentation for clarity:

• Update init.py: Step 9 now says 'Schema Relationship Inference'
• Update scan.py: Step 6 now says 'Infer Schema Relationships'
• Update configuration.md: Section renamed to 'Schema Relationship Inference'
• Update config templates and defaults.yaml
• Update models.py docstrings
• All 286 tests still passing (2987af6)

• refactor: rename RelationshipConfig to SchemaRelationshipConfig and move the config at root-level (7dc1875)

• refactor: make CollectionConfig inherit from ReplicationDefaultsConfig

This eliminates field duplication between CollectionConfig and
ReplicationDefaultsConfig by using inheritance. Now CollectionConfig
automatically inherits all default fields (cursor_field, cursor_fallback_field,
cursor_initial_value, primary_key, write_disposition, transform_error_mode)
from ReplicationDefaultsConfig, making the code more DRY.

Benefits:

• Reduces code duplication
• Ensures consistency between defaults and collection overrides
• Makes it easier to add new default fields in the future
• All 270 tests still passing (2450b03)

• refactor: reorganize config structure for better separation of concerns

Restructure the configuration models to better organize settings:

SCAN configuration:

• Split 'scan.pii' into 'scan.sampling' and 'scan.pii_analysis'
• Move sample_size and sample_strategy to scan.sampling
• Move PII analysis settings to scan.pii_analysis
• Add scan.cursor_detection for cursor field detection

REPLICATION configuration:

• Separate concerns into logical sub-sections:
  • replication.discovery: replicate_all, include/exclude patterns
  • replication.state_management: runs_collection, state_collection
  • replication.performance: max_parallel_collections, batch_size
  • replication.defaults: cursor defaults, write disposition, etc.
• Remove cursor_fields from defaults (determined by scan)
• Rename fallback_cursor to cursor_fallback_field
• Rename initial_value to cursor_initial_value

COLLECTIONS configuration:

• Rename 'pii_fields' to 'pii_anonymized_fields' for clarity

Updated all code references:

• scan.py: Use new scan.sampling.* and scan.pii_analysis.* paths
• run.py: Use new replication.discovery., performance. paths
• orchestrator.py: Use new nested config structure
• init.py: Create separate sampling and pii_analysis configs
• config_template.yaml.j2: Update Jinja2 template to new structure

All 270 unit tests pass. (01ebb51)

• refactor: convert remaining dataclasses to Pydantic models

• Convert IndexInfo in indexes.py to Pydantic BaseModel
• Convert ReplicationResult in replicator.py to Pydantic BaseModel
• Convert OrchestrationResult in orchestrator.py to Pydantic BaseModel
• Convert SamplingResult in pii/sampler.py to Pydantic BaseModel
• Convert FieldPIIStats and CollectionPIIAnalysis in pii/pii_analyzer.py to Pydantic BaseModel
• Update test_cascade_replication.py to use keyword arguments for Relationship instances

All 268 unit tests passing (41868ce)

• refactor: remove legacy code to load config (5fefb2d)

• refactor: migrate from dataclasses to Pydantic models

• Replace dataclass with Pydantic BaseModel for all config models
• Add Pydantic validation using field_validator and model_validator
• Simplify load_config with deep merge and Pydantic validation
• Add _save_defaults function to generate defaults.yaml from models
• Add main() function to allow saving defaults via CLI
• Update tests to handle Pydantic ValidationError
• Add pydantic>=2.0.0 dependency

This refactoring provides:

• Better validation with clearer error messages
• Automatic type coercion and validation
• Simplified config loading logic
• Ability to regenerate defaults.yaml from code (625c594)

• refactor: replace fragile _write_yaml_with_comments with Jinja2 template

Replace the 290-line fragile _write_yaml_with_comments function that manually
concatenates YAML strings with a clean Jinja2 template-based approach.

Changes:

• Add jinja2>=3.0.0 to project dependencies
• Create config_template.yaml.j2 template for config serialization
• Add _get_jinja_env() to initialize Jinja2 environment
• Add custom filters: toyaml and tojson for YAML formatting
• Replace _write_yaml_with_comments() with _render_config_template()
• Remove _format_yaml_value() helper function (no longer needed)
• Update save_config() to use template rendering

Benefits:

• Cleaner separation of logic and presentation
• Easier to maintain and modify YAML structure
• No manual string concatenation
• Better whitespace handling
• All 270 tests passing

The template properly handles:

• Scan configuration (discovery patterns, PII settings)
• Replication configuration (defaults, collections)
• Schema/relationships
• Comments and documentation
• Empty lists vs populated lists
• Proper YAML indentation (daa94ff)

• refactor: remove hardcoded defaults from scan command, use centralized defaults.yaml

• Load scan defaults from defaults.yaml at the start of scan_command()
• Replace all hardcoded default values with lookups from loaded defaults
• Maintain proper precedence: CLI options > Config file > defaults.yaml
• Remove duplicate load_defaults() call (was loading twice)
• All 270 tests passing

Key changes:

• Lines 201-205: Load system_defaults, scan_defaults, discovery_defaults, pii_defaults early
• Line 234: Get pii.enabled from defaults instead of hardcoding True
• Lines 249-251: Get sample_size, confidence_threshold from pii_defaults
• Line 294: Get exclude_patterns from discovery_defaults
• Lines 377-380: Get entity_types, default_strategies, allowlist, sample_strategy from pii_defaults
• Line 488: Remove duplicate load_defaults() call

This ensures both init and scan commands use the same centralized defaults.yaml as the source of truth. (1e8197f)

Testing

• test: revert test assertion for test_load_replication_config test (6979bb9)

Unknown

• Merge pull request #13 from nhuray/docs/improve-documentation

docs: comprehensive documentation improvements (949cd3d)

• Update technical-design.md (61dc832)

• Update technical-design.md (c64df53)

• Merge pull request #12 from nhuray/feat/improve-signal-handling

feat(cli): Improve signal handling for graceful interruption (08ddc4c)

• Merge pull request #12 from nhuray/feat/improve-signal-handling

feat(cli): Improve signal handling for graceful interruption (cbf73d1)

• Merge pull request #11 from nhuray/feat/show-anonymization-strategies-in-init

feat(cli): Show default anonymization strategies in init wizard (dd436c8)

• Merge pull request #10 from nhuray/feat/presidio-operators-yaml-config

feat: Refactor PII anonymization to use Presidio AnonymizerEngine with YAML-configured operators (f81bb11)

• Merge pull request #9 from nhuray/feat/validate-source-dest-urls

feat: validate source and destination databases are different (00a3f49)

• Merge pull request #8 from nhuray/feat/enhanced-scan-report

feat: Enhanced scan report with Jinja2 template and comprehensive sections (adb8c8a)

• Merge pull request #7 from nhuray/feat/sync-discovery-config

Sync discovery config between scan and replication (e69e3c0)

• Merge pull request #6 from nhuray/fix/dotenv-loading-and-exception-handling

fix: Add python-dotenv dependency and fix exception handling (636022c)

• Merge pull request #5 from nhuray/chore/infer-schema-relationships

feat: Add automatic schema relationship inference during scan (73defcb)

• Merge pull request #4 from nhuray/refactor/config

refactor: reorganize config structure for better separation of concerns (c26d741)

• Merge pull request #3 from nhuray/chore/pydantic-config

refactor: migrate from dataclasses to Pydantic models (90d2fa2)

• Merge pull request #2 from nhuray/feat/jinja-config-template

refactor: replace fragile YAML writer with Jinja2 template (a945d4a)

• Merge pull request #1 from nhuray/feat/presidio-config

feat: Add Presidio YAML configuration support for custom PII recognizers (d2782de)

• minor change (6da5661)

• Add semantic-release and github actions workflows (42bb44f)

• Format lint and fix tests (ef6bc0e)

• Remove black in favor of ruff

Update pre-commit hooks and install it (ade762a)

• Fix lint errors (9574db1)

• Add pre-commit hooks for code formatting and linting (ee2190a)

• Add .env.example file (6a50a52)

• Review README.md for replication CLI (f9a3354)

• Configure uv to install dev dependencies (aaaf6ee)

• Fix tests and deprecate legacy code (c333a0d)

• Add test coverage documentation (fa92767)

• Add comprehensive unit test suite

• Copied 13 test modules from analytics-pipelines
• Updated all imports to use mongo_replication package
• Added pytest configuration with coverage settings
• Added mongomock as dev dependency
• 223 passing tests covering:
  • PII detection and anonymization (Presidio)
  • Field transformations and exclusions
  • Cascade replication and relationships
  • State management (loads and collection state)
  • Configuration models and loading
  • Index replication
  • Job management
  • Error summarization

Note: 13 tests fail due to deprecated API usage in legacy tests
These tests validate deprecated methods that still work with warnings (f08e30d)

• Initial commit: MongoDB Replication Tool

• Production-grade MongoDB replication with PII redaction
• Parallel processing with configurable worker pools
• Incremental loading with cursor-based state management
• Cascade filtering for related document replication
• Support for replace, append, and merge write strategies
• Field transformations and exclusions
• Comprehensive CLI with interactive mode
• Full documentation (README, configuration, technical design)
• MIT licensed and ready for PyPI publishing (6dab292)