Rename s3 bucket

[bendnorman]

PR Overview

This PR:

• Updates nightly build script to write outputs to s3://pudl.catalyst.coop and gs://pudl.catalyst.coop. The build script still writes outputs to s3://intake.catalyst.coop for backward compatibility for folks still using intake.catalyst.coop. We'll deprecate it at a later date.
• Update references intake.catalyst.coop to pud.catalyst.coop in docs.
• Only copy outputs to distribution buckets if the build passed when kicked off by a tag or a scheduled build on dev. Previously, new unwanted directories for outputs were created in the distribution buckets when manually triggered builds succeeded.

Things that changed in the cloud:

• Created new buckets called s3://pudl.catalyst.coop and gs://pudl.catalyst.coop.
• Create a new log bucket for s3://pudl.catalyst.coop called pudl.catalyst.coop-logs. The AWS Open Data Program requires we have a 1 month retention policy on the log bucket. I created a GCP Storage Transfer job that copies the s3 logs daily to a GCS bucket called pudl-s3-logs.catalyst.coop so we can retain logs for longer than a month.
• We now have 1 TB of storage in s3!

PR Checklist

• Merge the most recent version of the branch you are merging into (probably dev).
• All CI checks are passing. Run tests locally to debug failures
• Make sure you've included good docstrings.
• For major data coverage & analysis changes, run data validation tests
• Include unit tests for new functions and classes.
• Defensive data quality/sanity checks in analyses & data processing functions.
• Update the release notes and reference reference the PR and related issues.
• Do your own explanatory review of the PR to help the reviewer understand what's going on and identify issues preemptively.

[codecov]

Codecov Report

Patch and project coverage have no change.

Comparison is base (b12967a) 88.5% compared to head (f86c4f0) 88.5%.
Report is 1 commits behind head on dev.

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #2793   +/-   ##
=====================================
  Coverage   88.5%   88.5%           
=====================================
  Files         90      90           
  Lines      10139   10139           
=====================================
  Hits        8982    8982           
  Misses      1157    1157           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bendnorman]

I keep getting this certificate error when I run the gsutil rsync command. I'm able to sync the two buckets on my local machine so I think it might be some Github Action authenticating weirdness:

Caught non-retryable exception while listing s3://pudl.catalyst.coop-logs/: Host pudl.catalyst.coop-logs.s3.amazonaws.com returned an invalid certificate (remote hostname "pudl.catalyst.coop-logs.s3.amazonaws.com" does not match certificate): ***'subject': ((('commonName', '*.s3.amazonaws.com'),),), 'issuer': ((('countryName', 'US'),), (('organizationName', 'Amazon'),), (('commonName', 'Amazon RSA 2048 M01'),)), 'version': 3, 'serialNumber': '013E1B83BC89D354ED4263B9FD2161BA', 'notBefore': 'Mar 21 00:00:00 2023 GMT', 'notAfter': 'Dec 19 23:59:59 2023 GMT', 'subjectAltName': (('DNS', '*.s3.amazonaws.com'), ('DNS', 's3.amazonaws.com')), 'OCSP': ('http://ocsp.r2m01.amazontrust.com/',), 'caIssuers': ('http://crt.r2m01.amazontrust.com/r2m01.cer',), 'crlDistributionPoints': ('http://crl.r2m01.amazontrust.com/r2m01.crl',)***
CommandException: Caught non-retryable exception - aborting rsync
Error: Process completed with exit code 1.

I decided to setup GCP Storage Transfer Service to copy the data from s3 to the GCS bucket.

[bendnorman]

Ah, running into the ogr failure getting debugged in #2849

[zaneselvans]

Is there a reason not to use a valid domain name in the pudl.catalyst.coop-logs bucket name? Do S3 and GCS treat bucket names that are domain names differently?

[bendnorman]

The AWS open data startup script creates log buckets using the {bucket-name}-logs format so I just stuck with it. I could update it to be pudl-logs.catalyst.coop if you'd prefer. I'm not sure if AWS and GCS handle domain named bucket differently. What are the benefits of domain-named buckets other than ensuring uniqueness?

[jdangerx]

Looks good to me, though I can't see the Storage Transfer job in the Google Cloud console due to permissions stuff. It would be nice to set that up through Terraform, too, instead of manually, but that shouldn't block this already long-suffering PR.

Looks like the storage transfer job is exposed through the Terraform provider: https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_transfer_job

[jdangerx]

"distribution" means "user-facing", right?

[bendnorman]

Yes!

[jdangerx]

nit: There's still a few references to "PUDL Intake catalogs" in the non-code sections of this doc, should we rename them?

[bendnorman]

Ah good point. I'll remove the references since we aren't supporting the intake catalogs.

[bendnorman]

Sounds good! Yeah, I wanted to use Terraform but thought it would drag the PR on for longer :/