1.0.0: Refactor config

[crowemi]

Major updates to configuration to allow for future expansion of formats and cloud providers.

• added config nodes for cloud_provider and format
• added aws_access_key_id and aws_secrect_access_key to pull from config (config credentials not used yet #3)
• updated format_parquet to include filesystem endpoint_override (Add a way to pass endpoint_url to target-s3 #7)
• moved s3 client creation to init, added endpoint_override (Add a way to pass endpoint_url to target-s3 #7)
• added sample-config.json
• linting flake8 -> black

[pnadolny13]

@crowemi this is cool, I'm planning to do a review of the code and clone and test it out myself locally but initially I had a question: what is your vision for this target? I see you mentioned this being multi cloud in the future, does that mean this would not longer be an S3 target but more of a generic cloud object storage target?

[pnadolny13]

I think this was discussed in an SDK (or Meltano) issue about pulling some of this cloud provider logic in, a challenge that came up was the fact that unneeded dependencies will always be install. If this target supports all major cloud providers but I only need AWS then I'm still constantly installing all dependencies needed for GCP/Azure/etc. even though they arent used. I wonder if using extras would be a good approach. Not a big deal but something to consider.

[crowemi]

@pnadolny13, thank you for reviewing!

what is your vision for this target?

My use-case is (and likely always will be) for S3. But, in this Singer Most Wanted we had talked about creating a cloud agnostic target.

I see you mentioned this being multi cloud in the future, does that mean this would not longer be an S3 target but more of a generic cloud object storage target?

Yes, I was thinking this target could become target-data-lake and serve as a multi-cloud storage/multi-format target.

---

I like the idea of extras, that will def be on my radar.

[pnadolny13]

My use-case is (and likely always will be) for S3. But, in this MeltanoLabs/Singer-Most-Wanted#70 we had talked about creating a cloud agnostic target.

@crowemi ah ok I didnt see that thread, sound good!