Support passing user_project argument to GCS bucket to support GCS buckets with “Requester pays” enabled

[TomNaessens]

Motivation / Background

GCS has an option on buckets to make them a "Requester Pays" type of bucket: https://cloud.google.com/storage/docs/requester-pays. In short, this allows tracing of who/what is actually requesting an upload/download to be able to allocate the costs to the project used.

To query a bucket where "requester pays" is enabled, the userProject needs to be added in the URL. The GCS Ruby library provides this as an optional user_project keyword argument to Project#bucket: https://github.com/googleapis/google-cloud-ruby/blob/google-cloud-storage/v1.11.0/google-cloud-storage/lib/google/cloud/storage/project.rb#L87-L99 (took v1.11 explicitly here as this is the current version requirement on Rails.)

When this is not passed, an error is thrown when listing/uploading files in a requester-pays bucket: Bucket is a requester pays bucket but no user project provided. (Google::Cloud::InvalidArgumentError)

Detail

This commit adds support for "requester pays"-buckets by accepting a user_project parameter (defaults to nil) on the GCS service bucket call, set by a variable in the config config.fetch(:user_project, nil):

rails/activestorage/lib/active_storage/service/gcs_service.rb

Line 192 in cd31b16

@bucket ||= client.bucket(config.fetch(:bucket), skip_lookup: true)

Checklist

Before submitting the PR make sure the following are checked:

• This Pull Request is related to one change. Unrelated changes should be opened in separate PRs.
• Commit message has a detailed description of what changed and why. If this PR fixes a related issue include it in the commit message. Ex: [Fix #issue-number]
• Tests are added or updated if you fix a bug or add a feature.
• CHANGELOG files are updated for the changed libraries if there is a behavior change or additional feature. Minor bug fixes and documentation changes should not be included.

[TomNaessens]

I'm a bit confused by testing here: it appears the service files are only ran when a configuration file is presented. The repo contains an encrypted configurations.yml.enc file, which is decrypted when some env variable is set.

Looking at the CI output, they're not set in CI though?

Skipping GCS Service tests because no GCS configuration was supplied

Are all these services actually being tested somewhere?

[yahonda]

Are all these services actually being tested somewhere?

Rails CI does not run these tests. There is a open issue #39015

[TomNaessens]

Are all these services actually being tested somewhere?

Rails CI does not run these tests. There is a open issue #39015

Hmm, alright, thank you for the information! I'll draft this until I can come up with a manual test scenario then.

[TomNaessens]

Alright, testing this locally is pretty annoying. I've verified the behaviour following these steps, feel free to replicate them if you wish to do so:

1. Set up a GCS bucket where "requester pays" is enabled
2. Verifying current behaviour
   1. Set up a Rails project on master (or any stable version)
   2. Set up an ActiveStorage configuration to interface with the GCS bucket
   3. Add a model (Company) with an attachment called "logo"
   4. Try to attach a file: c = Company.first; c.logo.attach(io: File.open('/tmp/logo.png'), filename: 'logo.png')
   5. Verify an error is thrown due to the userProject param missing:

      Intiating resumable upload command to https://storage.googleapis.com/upload/storage/v1/b/redacted/o?name=redacted-key
      ...
      required: Bucket is a requester pays bucket but no user project provided. (Google::Apis::ClientError)

3. Test this branch:
   1. Base the Rails project on this branch
   2. In your active storage configuration, add user_project: 'a_user_project'
   3. Attach the file again: c = Company.first; c.logo.attach(io: File.open('/tmp/logo.png'), filename: 'logo.png')
   4. Assert this succeeds, note the userProject paramater:

   Intiating resumable upload command to https://storage.googleapis.com/upload/storage/v1/b/redacted/o?name=6820n9jj742aww9trns53lgiuqf1&userProject=a_user_project

   5. Get the URL for this file: c.logo.url and assert the userProject is present:

   https://storage.googleapis.com/redacted/redacted-key?GoogleAccessId=redacted-signed-content&response-content-disposition=inline%3B+filename%3D%22logo.png%22%3B+filename%2A%3DUTF-8%27%27logo.png&response-content-type=image%2Fpng&userProject=a_user_project

   6. Assert you can open the file using the URL
   7. To illustrate what happens if this wasn't present, assert an error occurs when trying to open the file without userProject parameter, with user-project instead of userProject, or with an inexisting user project:
      1. Inexisting user project: <Error><Code>UserProjectInvalid</Code><Message>User project specified in the request is invalid.</Message></Error>
      2. Without user project (or with user-project instead of userProject): <Error><Code>UserProjectMissing</Code><Message>Bucket is a requester pays bucket but no user project provided.</Message><Details>Bucket is a requester pays bucket but no user project provided.</Details></Error>
4. Verify regular buckets still work
   1. Convert your bucket to a regular non-"Requester Pays" bucket
   2. Remove the user_project setting from your storage config
   3. Attach a file and assert that it succeeds
   4. Retrieve the url for the attachment and assert you can open it in a browser