You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The existing S3 connector offers some config options that we do not want to provide for new connections. However, we still want to support them rather than making changes that will break users' syncs.
At the top-level, the options provided by the S3 connector include:
dataset: should be mapped to name.
path_pattern: should be mapped to globs. I don't think additional changes are required but we should double-check.
path_prefix: we should attempt to create the appropriate globs by combining this with path_pattern.
These options are currently nested under a provider key (along with path_prefix and start_date), but for the new S3 connector we'll have them be at the top level.
This ticket will involve 1) adding a version to the new file-based CDK config object so that we can differentiate it from the existing S3 config and 2) creating a new S3-specific FileBasedConfig object, to be used by the new S3 connector. The new config object will include logic to adapt the old config to the new config format. In addition to top-level config options, this pattern will be used by the config options specific to each file type (described in separate tickets).
The text was updated successfully, but these errors were encountered:
The existing S3 connector offers some config options that we do not want to provide for new connections. However, we still want to support them rather than making changes that will break users' syncs.
At the top-level, the options provided by the S3 connector include:
dataset
: should be mapped toname
.path_pattern
: should be mapped toglobs
. I don't think additional changes are required but we should double-check.path_prefix
: we should attempt to create the appropriate globs by combining this withpath_pattern
.user_schema
: should be mapped toinput_schema
.start_date
: this will be wired through the file-based CDK in File CDK: add start_date config option #28136This will also include handling config options used by the existing S3 connector including:
bucket
aws_access_key_id
aws_secret_access_key
endpoint
These options are currently nested under a
provider
key (along withpath_prefix
andstart_date
), but for the new S3 connector we'll have them be at the top level.This ticket will involve 1) adding a version to the new file-based CDK config object so that we can differentiate it from the existing S3 config and 2) creating a new S3-specific FileBasedConfig object, to be used by the new S3 connector. The new config object will include logic to adapt the old config to the new config format. In addition to top-level config options, this pattern will be used by the config options specific to each file type (described in separate tickets).
The text was updated successfully, but these errors were encountered: