File CDK: S3 config adapter (Parquet options)

[clnoll]

The S3 connector offers a columns option for Parquet files. It does not appear to be in use by any connectors, but this should be verified. If any connectors are using it, we should update the file-based CDK to handle this option.

Acceptance Criteria

• The existing Parquet config options are mapped and handled appropriately by the file-based CDK.
• Any options that we cannot support are identified, along with the connectors that will be impacted.

[clnoll]

Grooming notes:

• Verify that column selection would be a good replacement for this.
• We'll deprecate the columns option (after verifying nobody from cloud is using it) and if any OS users write in we'll tell them to use column selection.
• To deprecate: go through the normal procedure (message to cloud users) + add a slack message for OS. If this option is set, we'll raise an exception stating that the field is deprecated & they should use column selection.

[maxi297]

Closing as there are only two customers on cloud using this option (one of which has never been synced and the other was a one time thing) and there is a path forward i.e. the customer can use column selection. This path forward could have a hit on performance but we can re-evaluate when it arises

[maxi297]

Comment on wrong issue. Re-opening

[maxi297]

It was the right issue... So closing because of :

Closing as there are only two customers on cloud using this option (one of which has never been synced and the other was a one time thing) and there is a path forward i.e. the customer can use column selection. This path forward could have a hit on performance but we can re-evaluate when it arises