File CDK: S3 connector testing & slow rollout

[clnoll]

This ticket is for doing the testing and reconnaissance needed to ensure that the new S3 connector will be backwards-compatible with the existing syncs, and rolling out the new connector in a deliberate way to keep an eye on the syncs.

This ticket is blocked on the S3 config adapter tickets

• Top-level config adapter
• CSV config adapter
• JSONL config adapter
• Parquet config adapter
  (There are no Avro-specific options that we need to handle in the new file-based CDK.)

If there are any breaking changes that cannot be avoided using the config adapter, we should identify the connectors that will be impacted and communicate this info to support prior to the release of the S3 connector.

Acceptance Criteria

• A new S3 connector that uses the File CDK is published.
• The connector can be used to sync all file types (CSV, JSONL, Parquet, and Avro).
• Test coverage exists for the old and new versions of the config.
• Tests can be run using CATs.
• Any unavoidable breaking changes are communicated to Support & on Slack, and we have an understanding of which connectors will be affected.

[clnoll]

Grooming notes:

• We will have the following breaking changes:
  • Getting rid of columns for parquet file type; no cloud customers are using this, but some OS users might be.
  • newlines_in_values options for JSONL; these both have some adoption but supporting them would force us to use pyarrow for parsing both of these file types, which comes with memory-related drawbacks.
• Review existing S3 CATs to ensure test coverage
• Choose some (all?) existing S3 connectors to run syncs on to verify they succeed; swap out the old connector via launchdarkly
• The idea is that this will be a slow rollout of the new connector to a few workspaces at a time.