Add Foundation for "Manual Webhooks" [#1224] #1267
Conversation
…nType, and a new "requires_input" PrivacyRequestStatus. - Add endpoints to get/create/update/delete the single AccessManualWebhook associated with a ConnectionConfig of type "manual_webhook". The new AccessManualWebhook primarily stores the fields that we are going to need to upload. The values uploaded will be passed directly to the user.
…webhooks # Conflicts: # tests/ops/api/v1/endpoints/test_connection_config_endpoints.py Bump downrev.
…s a "manual_webhook" as a manual connector type.
…esponding webhook to the postman collection. Clean up some copy/paste issues with the email connector.
| class AccessManualWebhook(Base): | ||
| """Describes a manual datasource that will be used for access requests. | ||
|
|
||
| These data sources are not treated as part of the traversal. Data uploaded | ||
| for an AccessManualWebhook is passed on as-is to the end user and is | ||
| not consumed as part of the graph. | ||
| """ | ||
|
|
||
| connection_config_id = Column( | ||
| String, ForeignKey(ConnectionConfig.id_field_path), unique=True, nullable=False | ||
| ) | ||
| connection_config = relationship( | ||
| ConnectionConfig, backref=backref("access_manual_webhook", uselist=False) | ||
| ) | ||
|
|
||
| fields = Column(MutableList.as_mutable(JSONB), nullable=False) |
There was a problem hiding this comment.
manual_webhook ConnectionConfigs will not have DatasetConfigs associated with them because they are not part of the graph. Instead, we will define the fields that we need to collect in the AccessManualWebhook table. These AccessManualWebhooks will run first as part of privacy request execution, requesting manual input from a fidesops admin, and putting the privacy request in a requires_input state before data has been uploaded and then we'll proceed with building a graph and executing the remainder of the privacy request.
| if system_type == SystemType.manual or system_type is None: | ||
| manual_types: List[str] = sorted( | ||
| [ | ||
| manual_type.value | ||
| for manual_type in ConnectionType | ||
| if manual_type == ConnectionType.manual_webhook | ||
| and is_match(manual_type.value) | ||
| ] | ||
| ) | ||
| connection_system_types.extend( | ||
| [ | ||
| ConnectionSystemTypeMap(identifier=item, type=SystemType.manual) | ||
| for item in manual_types | ||
| ] | ||
| ) | ||
|
|
There was a problem hiding this comment.
This adds a new "manual" system type, which has one connector associated with it, the new manual_webhook construct.
| @router.post( | ||
| ACCESS_MANUAL_WEBHOOK, | ||
| status_code=HTTP_201_CREATED, | ||
| dependencies=[Security(verify_oauth_client, scopes=[WEBHOOK_CREATE_OR_UPDATE])], |
There was a problem hiding this comment.
I'm reusing existing webhook scopes - I thought they were similar enough here. We have PolicyWebhooks where a user can define a series of requests that we should make before and after PrivacyRequest execution, some of which are capable of taking in data and adding it to known identities. Similarly, these manual webhooks run outside of the privacy request execution.
| @@ -117,6 +117,7 @@ class PrivacyRequestStatus(str, EnumType): | |||
| """Enum for privacy request statuses, reflecting where they are in the Privacy Request Lifecycle""" | |||
|
|
|||
| identity_unverified = "identity_unverified" | |||
| requires_input = "requires_input" | |||
There was a problem hiding this comment.
Will be implemented in the followup PR when we actually wire up the ManualWebhook. Privacy requests that still need manual input from a manual_webhook will be placed in a "requires_input" state.
As an aside, we're getting a lot of privacy request statuses, and most statuses have specific statuses that they transition to next, and ones in which they should never transition to. It might be nice to formalize this at some point with some sort of state machine.
| pii_field: ManualWebhookFieldType | ||
| dsr_package_label: Optional[ManualWebhookFieldType] |
There was a problem hiding this comment.
The way I understand it, the pii_field will be what we display to the user who is manually uploading data for the given subject, and the dsr_package_label will be the key that is used in the json blob that we upload to the user. I am making the dsr_package_label optional and creating it from the pii field if it doesn't exist.
… to fetch secret types for this schema - we are not currently collecting secrets for this connector at this time but will in the futuer.
Co-authored-by: Paul Sanders <paul@ethyca.com>
…webhooks # Conflicts: # CHANGELOG.md
|
Thank you for your review @sanders41! ⭐ |
* Add a new AccessManualWebhook model, a new "manual_webhook" ConnectionType, and a new "requires_input" PrivacyRequestStatus. - Add endpoints to get/create/update/delete the single AccessManualWebhook associated with a ConnectionConfig of type "manual_webhook". The new AccessManualWebhook primarily stores the fields that we are going to need to upload. The values uploaded will be passed directly to the user. * Add requires_input privacy request status to the frontend, and exposes a "manual_webhook" as a manual connector type. * Add the endpoints to create the manual connection config and its corresponding webhook to the postman collection. Clean up some copy/paste issues with the email connector. * Run black. * Add an empty ManualWebhookSchema to prevent a KeyError if you attempt to fetch secret types for this schema - we are not currently collecting secrets for this connector at this time but will in the futuer. * Apply suggestions from code review Co-authored-by: Paul Sanders <paul@ethyca.com> * Catch possible integrity error. Co-authored-by: Paul Sanders <paul@ethyca.com>
❗ Contains migration. Verify downrev before merging.'
👉 Documentation will be added in follow-up #1228 once we've actually wired up manual webhooks
Purpose
Adds the foundation for a new connector type we're calling the "Manual Webhook". Manual Webhooks are a simple connector type that allows data to be manually uploaded for an access request. Erasure requests are out of scope here. Importantly, the manual webhook is not a part of the graph. Request execution will expect data for the manual webhooks to be uploaded before the graph traversal runs. Any manual data collected via a manual webhook will be returned as-is directly to the user. There is no filtering on data category, its outputs aren't used by any other connector, and the only inputs it can use are the original identity data.
Note that this differs from an existing "manual" connector type we have on the backend that plugs into the traversal. #554 #571
Changes
Checklist
CHANGELOG.mdfileCHANGELOG.mdfile is being appended toUnreleasedsection in an appropriate category. Add a new category from the list at the top of the file if the needed one isn't already there.Run Unsafe PR Checkslabel has been applied, and checks have passed, if this PR touches any external servicesTicket
Fixes #1224