Restructure Rust plugin monorepo and CI

[lucarlig]

Summary

• restructure the repo into a managed Rust/Python plugin workspace rooted at plugins/rust/python-package, add the top-level Cargo workspace, and move rate_limiter into the managed layout with the shared framework bridge
• import the pii_filter plugin as a managed package, including its Rust core, Python compatibility shims, manifests, stubs, benchmarks, and plugin tests
• replace the single-plugin workflows with shared catalog-driven CI/release workflows, add root plugin tooling (plugin_catalog.py, install_built_wheel.py, root Makefile), and add repo-contract tests for layout, release tags, and wheel selection

What Changed

• moved the legacy top-level rate_limiter/ package into plugins/rust/python-package/rate_limiter/ and updated its Rust/Python surface to fit the managed plugin contract
• added a top-level Cargo workspace and shared cpex_framework_bridge crate for PyO3 framework object construction
• added the new managed plugins/rust/python-package/pii_filter/ package with Rust detection/masking logic, manifests, generated stubs, and plugin tests
• introduced root-level catalog and wheel-install helpers used by CI/release automation to discover plugins, map release tags, and select compatible built wheels
• replaced the plugin-specific Actions workflows with shared CI/release workflows that validate the catalog, select affected plugins, and run per-plugin verification
• updated root docs (README.md, DEVELOPING.md, TESTING.md) to describe the managed plugin layout, testing flow, and release model

Validation

• python3 -m unittest tests/test_plugin_catalog.py tests/test_install_built_wheel.py
• python3 tools/plugin_catalog.py validate .
• make verify-stubs in plugins/rust/python-package/pii_filter

[jonpspri]

Interesting change, and better to make it now. How does this approach work from a release management standpoint? I'd wanted to keep each plugin in its own Python Package with its own versioning system. Is that still viable? It looks on first blush like all packages will get built/versioned/released in lockstep. (Maybe we can do something with tags...)

[lucarlig]

Interesting change, and better to make it now. How does this approach work from a release management standpoint? I'd wanted to keep each plugin in its own Python Package with its own versioning system. Is that still viable? It looks on first blush like all packages will get built/versioned/released in lockstep. (Maybe we can do something with tags...)

Current state: plugins are still independently versioned and released, not lockstepped. Each plugin keeps its own version in its own Cargo.toml and manifest, and the release workflow is already plugin-specific via tags like <plugin>-v<version>, so a release builds only the plugin named by that tag.

[gandhipratik203]

What's Good

Catalog-driven architecture with strong convention enforcement, efficient CI that only builds affected plugins, and a lightweight fail-fast gate before expensive release builds. The pii_filter plugin is production-quality with RegexSet parallel matching, Luhn/SSN structural validation, and deterministic overlap resolution. The repo-contract test suite is exceptional (47+ tests covering catalog validation, CI selection, release resolution, stub exports, and workflow content assertions). The rate limiter migration to Rust plugin core is complete and faithful, with double-layer fail-open semantics preserved.

---

Findings

F1 — Orphan stub file in python/pii_filter_rust/

Two .pyi stubs exist for the pii_filter extension module. cpex_pii_filter/pii_filter_rust/__init__.pyi is the authoritative one (curated by stub_gen.rs, includes both PIIDetectorRust and PIIFilterPluginCore). python/pii_filter_rust/__init__.pyi is an orphan from pyo3_stub_gen's default output path — it's missing PIIFilterPluginCore, isn't used by maturin (python-source = "."), and isn't checked by verify-stubs. It will confuse IDEs and silently diverge as the API evolves. Delete it and add python/ to .gitignore, or have stub_gen.rs clean it up after generation.

F2 — stub_gen.rs curates stubs via fragile string replacement

curate_extension_stub() injects PIIFilterPluginCore using content.replace("\"PIIDetectorRust\",\n]", ...) — a literal match that assumes exact formatting from pyo3_stub_gen. If the library changes its output formatting in a future version, the replacement silently fails and the stub becomes incomplete. The existing verify-stubs step (git diff --exit-code) won't catch a first-time silent failure. Add a post-curation assertion that the output contains both "PIIFilterPluginCore" in __all__ and class PIIFilterPluginCore:.

F3 — Benchmark function detect_pii() skips validation and whitelist

The standalone detect_pii() used by all benchmarks does simple RegexSet matching. The production detect_internal() adds whitelist filtering, structural validation (SSN format, Luhn), and overlap resolution (O(n log n) sort + sweep). Benchmark results are artificially fast and won't catch regressions in the validation/dedup path. The benchmark named "whitelist_filtering" configures whitelist patterns but calls a function that ignores them. Either unify the two functions or switch benchmarks to use detect_internal().

F4 — No .gitignore updates for restructured paths

Build artifacts moved to plugins/rust/python-package/*/target/, .venv/, dist/ but no .gitignore changes are included. Verify the existing .gitignore has global pattern coverage; if it uses path-specific entries like rate_limiter/target/, the new locations lack coverage.

---

Verdict

Well-architected restructure with strong tooling and thorough testing. F1 and F2 should be addressed before merge — both are straightforward fixes. F3 is worth fixing so benchmarks measure real behavior. F4 needs a quick verification. None of these are heavy lifts.

[lucarlig]

Addressed Pratik's findings on the current PR head (158372f).

• F1: removed the orphan python/pii_filter_rust/__init__.pyi stub and updated stub_gen to delete that path if it reappears, so only cpex_pii_filter/pii_filter_rust/__init__.pyi remains authoritative.
• F2: hardened stub_gen by curating through a dedicated helper with explicit assertions that PIIFilterPluginCore is present in both __all__ and the generated class body, and added unit coverage for that path.
• F3: updated the benchmark-facing detect_pii() path to share the same whitelist filtering, structural validation, and overlap resolution logic as production detection.
• F4: verified the root .gitignore already has global coverage for target/, .venv/, and dist/, so no repo ignore change was needed.

Validation run locally:

• cargo test --manifest-path plugins/rust/python-package/pii_filter/Cargo.toml
• make verify-stubs in plugins/rust/python-package/pii_filter
• focused python3 -m unittest checks for the authoritative extension stub layout

[gandhipratik203]

LGTM!