ENG-3084: Go shared library + CLI via ctypes

[thabofletcher]

Ticket ENG-3084

Description Of Changes

Adds libpbac — a Go shared library (.so/.dylib) that Python calls directly via ctypes. This eliminates all duplicated Python evaluation code. The Go library is the single source of truth for PBAC evaluation; Python handles only CLI argument parsing, SQL parsing (sqlglot), and JSON I/O.

Architecture:

Go library (single source of truth)
├── pkg/pbac/         EvaluatePurpose, EvaluatePolicies
├── pkg/pipeline/     full pipeline orchestration
├── pkg/fixtures/     YAML config loading
└── cmd/libpbac/      CGo exports → libpbac.so/.dylib
        │
        ├── ctypes ── fides CLI (Python: SQL parsing + JSON I/O)
        └── HTTP ──── fidesplus sidecar (separate repo)

Code Changes

Go — new packages:

• pkg/fixtures/ — YAML loaders for consumers/, purposes/, datasets/, policies/ config directories
• pkg/pipeline/ — full pipeline: identity resolution + dataset resolution + purpose eval + gap reclassification + policy filtering. 8 tests against the pbac/ fixture set.
• cmd/libpbac/ — CGo exports: EvaluatePipelineJSON, EvaluatePurposeJSON, EvaluatePoliciesJSON, LoadFixturesJSON, FreeString. Build with go build -buildmode=c-shared.

Go — type changes:

• PurposeViolation — add SuppressedByPolicy, SuppressedByAction fields (violations are kept for audit with suppression metadata inline, not dropped)
• AccessEvaluationRequest — add Identity field (carried through for future identity-aware policies)
• All policy types — yaml: + json: dual tags for YAML loading and HTTP transport
• GapUnconfiguredConsumer restored (needed by pipeline's gap reclassification step)
• data_use and control enriched on violations in the pipeline

Python — new:

• service/pbac/engine.py — ctypes wrapper that loads libpbac and exposes evaluate_pipeline(), evaluate_purpose(), evaluate_policies(), load_fixtures(). JSON in, JSON out, Go does all the work.

Python — rewritten:

• cli/commands/pbac.py — fides pbac evaluate --config DIR --identity EMAIL [SQL_FILE] now parses SQL with sqlglot then calls Go via engine.py. Lower-level evaluate-purpose and evaluate-policies primitives also call Go.
• service/pbac/service.py — InProcessPBACEvaluationService calls engine.evaluate_purpose() (Go via ctypes) instead of the deleted Python reimplementation.

Python — deleted (was duplicating Go):

• service/pbac/evaluate.py — Python EvaluatePurpose (135 lines)
• service/pbac/policies/evaluate.py — Python EvaluatePolicies + ParsedPolicy (349 lines)
• service/pbac/pipeline.py — Python pipeline (307 lines)
• service/pbac/fixtures.py — Python YAML loaders (220 lines)
• Tests for the above (~600 lines)

Fixtures — ported from POC #7941:

• pbac/ — consumers, purposes, datasets, policies, SQL entries for alice/bob/carol/dave

How to test

# Build the Go shared library (once)
cd policy-engine && go build -buildmode=c-shared -o libpbac.dylib ./cmd/libpbac/ && cd ..

# Run Go tests
cd policy-engine && go test ./... -v && cd ..

# Run the CLI (needs fides installed with [all] extras for sqlglot)
fides pbac evaluate --config pbac/ --identity alice@demo.example pbac/entries/alice.txt
fides pbac evaluate --config pbac/ --identity bob@demo.example pbac/entries/bob.txt
fides pbac evaluate --config pbac/ --identity carol@demo.example pbac/entries/carol.txt
fides pbac evaluate --config pbac/ --identity dave@demo.example pbac/entries/dave.txt

Expected outcomes

Identity	Result
alice	4 records: compliant, violation suppressed by allow-analytics-on-billing-data (with data_use + control + suppressed_by_action), compliant via collection-level purpose, violation stands
bob	unconfigured_dataset gap for cold_storage
carol	unresolved_identity gap
dave	unconfigured_consumer gap (exists but no purposes)

Companion PRs

• fidesplus pbac-cli-option-b-transport — sidecar endpoint POST /v1/evaluate-pipeline + Python client method, consuming the same Go library over HTTP

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
fides-plus-nightly	Ready	Preview, Comment	Apr 21, 2026 4:19pm

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
fides-privacy-center	Ignored		Apr 21, 2026 4:19pm

[codecov]

Codecov Report

❌ Patch coverage is 50.00000% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.94%. Comparing base (a52d50a) to head (5b9027e).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
src/fides/cli/commands/pbac.py	22.85%	27 Missing ⚠️
src/fides/service/pbac/service.py	20.00%	12 Missing ⚠️
src/fides/service/pbac/engine.py	87.87%	2 Missing and 2 partials ⚠️

❌ Your patch status has failed because the patch coverage (50.00%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage.
❌ Your project status has failed because the head coverage (84.94%) is below the target coverage (85.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7944      +/-   ##
==========================================
- Coverage   85.04%   84.94%   -0.11%     
==========================================
  Files         631      630       -1     
  Lines       41217    41082     -135     
  Branches     4807     4768      -39     
==========================================
- Hits        35053    34896     -157     
- Misses       5070     5103      +33     
+ Partials     1094     1083      -11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[thabofletcher]

/code-review

[claude]

Code Review: Go shared library (libpbac) + fides pbac CLI

This PR makes a significant architectural shift: the PBAC evaluation engine moves from pure Python to a Go shared library called via ctypes, with Python retained only for CLI argument parsing, SQL parsing (sqlglot), and JSON I/O. The overall design is sound — keeping evaluation logic in one place (Go) and using the shared library as the single source of truth is a good pattern. The fixture loading, pipeline orchestration, and demo data are well-structured.

That said, there are a few issues worth addressing before this lands.

Critical

Test coverage gap (deleted Python tests, not replaced in Go)
The most significant concern is that ~1,000 lines of precise Python test coverage was deleted — covering taxonomy prefix matching, all four consent requirement variants, geo not_in, data_flow none_of, AND logic in unless blocks, and several edge cases (empty match key, empty context, action only on DENY). These are the highest-stakes paths in an access-control system. The new Go pipeline tests cover the happy-path scenarios but don't reach the policy evaluation unit level. The detailed comment on the deleted test_evaluate_access.py lists the specific gaps. Recommend adding equivalent Go unit tests in policy_evaluate_test.go before merging.

Null pointer from Go not guarded in _call
ctypes.string_at(result_ptr) when result_ptr is None (null c_void_p) gives an unhelpful ValueError. Add an explicit None check with a clear error message.

Medium

• _find_library fragile path arithmetic — 5 chained .parent calls; silent breakage if the file moves.
• Library absence fails at evaluation time — If the Go .so isn't built, the service crashes on the first evaluate() call. A startup-time check or clearer documentation of the build prerequisite is needed.
• Path traversal in LoadFixturesJSON — config_dir is used unsanitized; a ../../ prefix can escape the intended directory.
• .yaml not supported — All four Load* functions only glob *.yml.

Minor / Nit

• __import__("threading").Lock() pattern is unexplained; normal top-level import threading is preferred.
• input parameter shadows the Python built-in in evaluate_pipeline.
• _qualified_name is defined below its call site in pbac.py.
• collections list per dataset key in service.py is not deduplicated before passing to Go (Go deduplicates, but the redundant data is unnecessary).
• The allSuppressed vacuously-true case for empty violations is correct but deserves a clarifying comment.
• The except Exception catch around sqlglot.parse needs a comment explaining what non-ParseError exceptions are expected.

---

🔬 Codegraph: connected (47107 nodes)

---

💡 Write /code-review in a comment to re-run this review.

[thabofletcher]

Thanks for the thorough review. Addressed in the latest commit:

Critical #1 — Test coverage gap

The coverage is actually present — it moved to Go. policy_evaluate_test.go covers taxonomy prefix matching, all four consent variants, geo not_in, data_flow none_of, AND logic in unless, empty match key, empty context, and DENY-only action. edge_cases_test.go adds 31 more focused tests covering the exact cases called out (data subject dimension, not_opt_in/not_opt_out, none_of, match dimension combinations). Total: 64 Go unit tests across pkg/pbac/. The section comments that said "Missing coverage" were artefacts of the drafting process — removed in this commit.

Critical #2 — Null pointer from Go not guarded ✅ Fixed — explicit None check with a clear error message before ctypes.string_at.

Medium #4 — .yaml not supported ✅ Fixed — extracted globYAML helper that globs both *.yml and *.yaml, used in all four loaders.

Minor — __import__("threading") pattern ✅ Fixed — normal module-level import threading.

Minor — input shadows built-in ✅ Fixed — renamed to query in evaluate_pipeline.

Minor — _qualified_name below call site ✅ Fixed — moved before the command definitions.

Medium #1 — Fragile path arithmetic — Left as-is for now; the five .parent calls are a standard pattern for package-relative paths and the locations they target (wheel bin/ and repo-root policy-engine/) are stable. Happy to revisit if the structure changes.

Medium #2 — Library absence fails at eval time — Intentional lazy loading; the error message already includes the build command. A startup check would require importing the service at init time which has other tradeoffs.

Medium #3 — Path traversal in LoadFixturesJSON — This is a CLI command running on the developer's own machine with their own filesystem access. Not treating as a security surface.