Metadata: set cache control headers

[bnchrch]

What

Ensure we are not caching the metadata files, spec or registry at the CDN level.

While we had caching disabled, google caches public buckets by default.

More context here: https://airbytehq-team.slack.com/archives/C02U46QK5C3/p1683940962000049?thread_ts=1683925529.873469&cid=C02U46QK5C3

Notes for reviewers

Google does not have a bucket level setting for this 😢

So we will want to run this command after deploying

gsutil setmeta -r -h "Cache-Control: no-cache" gs://io-airbyte-cloud-spec-cache
gsutil setmeta -r -h "Cache-Control: no-cache" gs://prod-airbyte-cloud-connector-metadata-service

[alafanechere]

Cool, small change for greater peace of mind 👍

[alafanechere]

nit: shall we have a try/catch that make the function return false on error?

[bnchrch]

I think we want the loud error in this case.

But we could change this to

return blob_to_save.exists()

However that doesnt help much in an overwrite scenario

[alafanechere]

This is what uploads the metadata file right?

[bnchrch]

correct!

[alafanechere]

This is what upload the registry right?

[bnchrch]

correct!

[alafanechere]

So we will want to run this command after deploying
gsutil setmeta -r -h "Cache-Control: no-cache" gs://io-airbyte-cloud-spec-cache
gsutil setmeta -r -h "Cache-Control: no-cache" gs://prod-airbyte-cloud-connector-metadata-service

Can this be terraformed or re-executed after each metadata / registry upload?

[bnchrch]

@alafanechere thanks for the review.

So we will want to run this command after deploying
gsutil setmeta -r -h "Cache-Control: no-cache" gs://io-airbyte-cloud-spec-cache
gsutil setmeta -r -h "Cache-Control: no-cache" gs://prod-airbyte-cloud-connector-metadata-service

Can this be terraformed or re-executed after each metadata / registry upload?

Im talking with ops after the hackdays about this. Looking at some google help forms our best option is to add a serverless function to gcloud that reruns this on every upload.

if thats the option I would say lets avoid it and start ensuring the metadata_service library is the entry point to these buckets.