ENG-3630: Consent identity enrichment via DB graph traversal#8039
Merged
Conversation
…aversal Before consent propagation, resolve missing user identities by querying consent-enabled database integrations. When a consent request arrives with only a subset of required identities (e.g., external_id from a webhook), enrichment queries a DB users table to discover the missing identity (e.g., email), enabling consent to propagate to all configured connectors. Enrichment is opt-in: ConnectionConfig.enabled_actions must explicitly include ActionType.consent for non-SaaS DB integrations. Uses each connector's own QueryConfig for correct SQL generation across DB types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 2 Skipped Deployments
|
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| @@ -0,0 +1,283 @@ | |||
| from typing import Any, Dict, Generator | |||
Codecov Report❌ Patch coverage is
❌ Your patch status has failed because the patch coverage (83.12%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #8039 +/- ##
========================================
Coverage 84.96% 84.96%
========================================
Files 632 633 +1
Lines 41472 41632 +160
Branches 4836 4869 +33
========================================
+ Hits 35237 35374 +137
- Misses 5136 5149 +13
- Partials 1099 1109 +10 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
- Remove _determine_missing_identities check (always query reachable collections for robustness with bidirectional consent integrations) - Fix cache_identity to write directly to DSR cache store instead of going through Identity Pydantic model validation - Update doc to reference correct fidesplus system connection endpoint for setting enabled_actions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
galvana
requested review from
Linker44 and
nreyes-dev
and removed request for
a team and
nreyes-dev
- Document which QueryConfig attributes _EnrichmentNode must satisfy - Use connector.cursor_result_to_rows() instead of static call so MariaDB and other connectors use their own row conversion Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add cursor_result_to_rows to connector mock (instance method now) - Mock get_dsr_cache_store via autouse fixture instead of asserting on cache_identity (no longer called directly) - Remove test_updates_privacy_request_cache (tested implementation detail) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Writes to the activity timeline so enrichment is visible in the privacy request UI. Logs whether identities were discovered or not. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fast path resolves missing email/external_id from CurrentPrivacyPreference before falling back to DB connector queries. Gated behind FIDES__CONSENT__IDENTITY_ENRICHMENT. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
| mock_namespace_meta.return_value = None | ||
| mock_connector = create_autospec(SQLConnector, instance=True) | ||
| mock_connector.client.side_effect = Exception("Connection refused") | ||
| mock_connector.query_config = lambda node: SQLQueryConfig(node) |
The early return guard skips enrichment when both email and external_id are present, so phone_number was never populated from the DB. Pass all three identities in the input to actually test the no-overwrite behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ticket ENG-3630
Description Of Changes
Adds consent identity enrichment that resolves missing email or external_id before consent propagation. Gated behind
FIDES__CONSENT__IDENTITY_ENRICHMENTconfig flag (default: disabled).When a consent request arrives with only a subset of required identities (e.g.,
external_idfrom a Bloomreach webhook), Fides cannot propagate consent to connectors that need a different identity (e.g., Iterable needsemail). This change adds a two-tier enrichment strategy:CurrentPrivacyPreferencerecords using indexed hashed identity columns (single Postgres query)Key design decisions:
FIDES__CONSENT__IDENTITY_ENRICHMENT(defaultFalse) -- no behavior change until explicitly enabledCurrentPrivacyPreference.hashed_email/hashed_external_id(indexed) for O(1) lookupConnectionConfig.enabled_actionsmust explicitly includeActionType.consenton non-SaaS DB integrationsQueryConfigfor correct SQL generation (Postgres quoting, BigQuery schema prefix, etc.)Code Changes
FIDES__CONSENT__IDENTITY_ENRICHMENTconfig flag toConsentSettings_enrich_from_current_preferences()-- fast path lookup viaCurrentPrivacyPreferencehashed identity columnsenrich_identities_for_consent()-- config gate, preference fast path, DB connector fallback_cache_and_log_enrichment()and_enrich_from_db_connectors()helpersbuild_consent_identity_enrichment_graph()ingraph_task.py-- filters to non-SaaS connectors with consent inenabled_actionsenable_identity_enrichmentfixture and real DB sessionCurrentPrivacyPreferencerecordsSteps to Confirm
pytest --no-cov tests/ops/task/test_consent_identity_enrichment.pypytest --no-cov tests/ops/integration_tests/test_consent_identity_enrichment.pyPre-Merge Checklist
CHANGELOG.mdupdated