HA-524 - Fides should support Google Cloud Storage as a storage option

[vcruces]

Closes HA-524

Description Of Changes

Google Cloud Storage (GCS) can now be chosen as storage for privacy requests.

Code Changes

• Added GCS storage configuration UI component and related type definitions
• Extended storage schemas and endpoints to support GCS integration
• Created a new method to handle interactions with GCS: uploading files and generating signed URLs for secure access.
• Added authentication service for GCS.
• Updated storage upload service to handle GCS uploads
• Added test coverage for GCS functionality

Steps to Confirm

1. Create a default storage for GCS. Make a PUT request to /api/v1/storage/default with the following body:

{
  "type": "gcs",
  "details": {
    "naming": "request_id",
    "auth_method": "service_account_keys",
    "bucket": "prj-sandbox-55855-test-bucket",
    "max_retries": 0
  },
  "format": "json"
}

2. Make a PATCH request to /api/v1/config with the following body:

{
  "storage": {
    "active_default_storage_type": "gcs"
  }
}

3. Add secrets for the new storage. Make a PUT request to /api/v1/storage/default/{storage_type}/secret with storage_type "gcp" and the body of the request is in 1password under the name "GCS prj-sandbox-55855-test-bucket".

To test that the storage is working:

1. Add a system
2. Add an Integration
3. Add dataset
4. Make sure the new storage is set as the default storage.
5. The environment variable FIDES__SECURITY__SUBJECT_REQUEST_DOWNLOAD_UI_ENABLED must be true to be able to download the files.
6. Create a new privacy request ("access your data")
7. Access the bucket in https://console.cloud.google.com/storage/browser/prj-sandbox-55855-test-bucket and verify that the file is there with the correct content type
8. From the admin UI, access the privacy request and download the request results. It should open a new tab in the browser with the json.

Another way to configure storage and its secrets:

1. Go to /privacy-requests/configure/storage, select GCS and fill out the form. The secret details are in 1password under the name "GCS prj-sandbox-55855-test-bucket".

Pre-Merge Checklist

• Issue requirements met
• All CI pipelines succeeded
• CHANGELOG.md updated
  • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
  • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
• Followup issues:
  • Followup issues created (include link) -> https://ethyca.atlassian.net/browse/HA-622
  • No followup issues
• Database migrations:
  • Ensure that your downrev is up to date with the latest revision on main
  • Ensure that your downgrade() migration is correct and works
    • If a downgrade migration is not possible for this change, please call this out in the PR description!
  • No migrations
• Documentation:
  • Documentation complete, PR opened in fidesdocs
  • Documentation issue created in fidesdocs
  • If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  • No documentation updates required

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
fides-privacy-center	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 15, 2025 0:24am

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
fides-plus-nightly	⬜️ Ignored (Inspect)	Visit Preview		Apr 15, 2025 0:24am

[vcruces]

I think the ConfigurePrivacyRequestsStorage page (which uses this component) is not currently accessible by the user

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.90%. Comparing base (dd2cf79) to head (50a2a5c).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6006      +/-   ##
==========================================
+ Coverage   86.86%   86.90%   +0.03%     
==========================================
  Files         418      419       +1     
  Lines       25916    25995      +79     
  Branches     2826     2829       +3     
==========================================
+ Hits        22513    22591      +78     
  Misses       2785     2785              
- Partials      618      619       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[JadeCara]

This is an exciting piece of functionality! Thanks for adding so many tests! I have a couple change requests around testing, but mostly looks really good!

One thing I am trying to get better at myself, is making sure I Autospec on mocks. I called it out specifically on a few tests where a class is getting mocked, but its also great for functions.

• Signature enforcement: Your mock will only accept the same arguments as the real function.
• Early error detection: If you call the mock with the wrong arguments, the test fails.
• Cleaner, safer mocks: It prevents unexpected behavior due to missing or extra arguments.

[JadeCara]

Looks good! Great work!

[cypress]

fides    Run #12819

Run Properties:   Passed #12819  •  01c19e52c5: HA-524 - Fides should support Google Cloud Storage as a storage option (#6006)

Project	fides
Branch Review	main
Run status	Passed #12819
Run duration	00m 51s
Commit	01c19e52c5: HA-524 - Fides should support Google Cloud Storage as a storage option (#6006)
Committer	Victoria Cruces
View all properties for this run ↗︎

---

Test results
Failures	0
Flaky	0
Pending	0
Skipped	0
Passing	5
View all changes introduced in this branch ↗︎