Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File CDK: S3 config adapter (top-level config options) #28131

Closed
clnoll opened this issue Jul 11, 2023 · 0 comments · Fixed by #29145
Closed

File CDK: S3 config adapter (top-level config options) #28131

clnoll opened this issue Jul 11, 2023 · 0 comments · Fixed by #29145

Comments

@clnoll
Copy link
Contributor

clnoll commented Jul 11, 2023

The existing S3 connector offers some config options that we do not want to provide for new connections. However, we still want to support them rather than making changes that will break users' syncs.

At the top-level, the options provided by the S3 connector include:

  • dataset: should be mapped to name.
  • path_pattern: should be mapped to globs. I don't think additional changes are required but we should double-check.
  • path_prefix: we should attempt to create the appropriate globs by combining this with path_pattern.
  • user_schema: should be mapped to input_schema.
  • start_date: this will be wired through the file-based CDK in File CDK: add start_date config option #28136

This will also include handling config options used by the existing S3 connector including:

  • bucket
  • aws_access_key_id
  • aws_secret_access_key
  • endpoint

These options are currently nested under a provider key (along with path_prefix and start_date), but for the new S3 connector we'll have them be at the top level.

This ticket will involve 1) adding a version to the new file-based CDK config object so that we can differentiate it from the existing S3 config and 2) creating a new S3-specific FileBasedConfig object, to be used by the new S3 connector. The new config object will include logic to adapt the old config to the new config format. In addition to top-level config options, this pattern will be used by the config options specific to each file type (described in separate tickets).

@clnoll clnoll changed the title File CDK: S3 config adapter File CDK: S3 config adapter (top-level config options) Jul 11, 2023
@clnoll clnoll changed the title File CDK: S3 config adapter (top-level config options) File CDK: S3 config adapter: handle top-level config options Jul 11, 2023
@clnoll clnoll changed the title File CDK: S3 config adapter: handle top-level config options File CDK: S3 config adapter (top-level config options) Jul 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants