S3 to GCP

[PDEUXA]

Hi,

I am wondering if there is a better solution to do so. I have bunch of files in one cloud provider (lets says S3), then I need to run computation and save the results to another cloud provider (lets says Google).
Is there a way to do this properly?

For now, my process is the following:

• read from S3
• make computation and save to /tmp
• read back from /tmp
• change S3_KEYS and save to GCP

I work, but It is quite IO intensive on the local /tmp save.

[Tishj]

Using the secret manager, can't you define a scope for the different buckets, then copy directly to GCP instead of first dumping it in /tmp ?
https://duckdb.org/2024/02/13/announcing-duckdb-0100.html#secret-manager

[PDEUXA]

That's a fresh info, I will have a look to the secret manager, thanks!