Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read allowed_hosts from actor definitions yaml and store in database #21183

Closed
evantahler opened this issue Jan 10, 2023 · 2 comments · Fixed by #21363
Closed

Read allowed_hosts from actor definitions yaml and store in database #21183

evantahler opened this issue Jan 10, 2023 · 2 comments · Fixed by #21363
Assignees

Comments

@evantahler
Copy link
Contributor

evantahler commented Jan 10, 2023

From the tech spec:

ActorDefintions are updated so that connectors can declare which hosts/IPs they intend to connect to. This is a new optional field in source_definitions.yaml and destination_definitions.yaml.

- name: HubSpot
  sourceDefinitionId: 36c891d9-4bd9-43ac-bad2-10e12756272c
  dockerRepository: airbyte/source-hubspot
  dockerImageTag: 0.3.0
  documentationUrl: https://docs.airbyte.com/integrations/sources/hubspot
  icon: hubspot.svg
  sourceType: api
  releaseStage: generally_available
  # ~~~ NEW ~~~
  allowed_hosts:
    - *.hubspot.com 

The new “allowed_hosts” property is an array. It’s values can be URL matches (per the example above, e.g. "*.instagram.com"), IP address and/or CIDR ranges, or a reference to a value that will be completed by the user completing the connector’s configuration. The reference use-case is required for those connectors who accept a dynamic hostname or IP address to connect to (e.g. a database hostname or an API sub-domain). A reference specification could look like:

allowed_hosts:
 - “${#/definitions/config/hostname}”

Finally, we can combine the approaches for when there’s a customer-specific subdomain in use with interpolation:

allowed_hosts:
- “${#/definitions/config/subdomain}.snowflake.com”

This information should be loaded by the bootloader and stored in the actor_defintions table in the database in a new column as a JSON array.

@evantahler evantahler changed the title Read allowed_hosts from actor definitions and store in database Read allowed_hosts from actor definitions yaml and store in database Jan 10, 2023
@evantahler
Copy link
Contributor Author

evantahler commented Jan 10, 2023

Grooming:

  • we should do some validation. E.g. the values from configs should exist in the SPEC. We also might want to validate the URL strings?
  • Filling this info out in OSS will eventually propagate to cloud

@evantahler evantahler self-assigned this Jan 13, 2023
@evantahler
Copy link
Contributor Author

evantahler commented Jan 13, 2023

Following the work done in #18300 is a really good guide about how to add a migration and load data from the actor_definitions!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant