[Ops] Fix GCS bucket access for future buildkite agents

[delanni]

Summary

Once we're moving to the elastic-wide buildkite agents, and away from the kibana-buildkite-managed ones, we won't have default access to the buckets we used to use, as the assumed service account will differ.

Note: Although this will only be required in the new infra, but this change can be merged and expected to work properly in the current infra as well.

Solution

We've set up a central service-account with rights to impersonate other service accounts that have controlled access to individual buckets to minimize the reach and influence of individual accounts. See: https://github.com/elastic/kibana-operations/pull/51

several of the changes weren't tested, as they're part of CI tasks outside the PR build - will merge with caution and monitor the stability afterwards

TODO: add access, and assume account before other GCS bucket usages

• storybook
• coverage (.buildkite/scripts/steps/code_coverage/reporting/uploadPrevSha.sh)
• upload static site (.buildkite/scripts/steps/code_coverage/reporting/uploadStaticSite.sh)
• SO object migration (.buildkite/scripts/steps/archive_so_migration_snapshot.sh)
• ES Snapshot manifest upload (.buildkite/scripts/steps/es_snapshots/create_manifest.ts)
• Scalability? (.buildkite/scripts/steps/functional/scalability_dataset_extraction.sh)
• Benchmarking (.buildkite/scripts/steps/scalability/benchmarking.sh)
• Webpack bundle analyzer (.buildkite/scripts/steps/webpack_bundle_analyzer/upload.ts)
• Build chromium (x-pack/build_chromium/build.py) Not needed, as it's manual, and not a CI task

TODO: others

• Remove manifest upload (.buildkite/scripts/steps/es_serverless/promote_es_serverless_image.sh)
• Decide if we should merge with the CDN access: no, SRE is managing that account
• Bazel remote cache seems to also rely on gcs - roles PR: https://github.com/elastic/kibana-operations/pull/56

Closes: https://github.com/elastic/kibana-operations/issues/29
Part of: https://github.com/elastic/kibana-operations/issues/15

[delanni]

In #173159 @jbudz set up a similar scheme of access to a bucket for CDN assets (just a few lines above).

What should our approach be here? Should we have specific service accounts for different usage sites, with well-controlled bucket access, and assume the specific ones right before the uploads? Or should we use a powerful service account for most of the GCS access throughout our build jobs?

The former solution allows for nicer access control over different buckets and gcloud features, the latter would allow us to activate the account in a central place, and not have to worry about locally assuming the role before usage.

[mistic]

IMHO we should go with the former one. Besides the nicer access control , its more explicit and a better operational/security practise

[delanni]

While I agree with the general sentiment, that granular access control is nicer, but wouldn't this get over-board with all these sites requiring 1-1 extra service account, with their own access to an individual bucket?

 coverage (.buildkite/scripts/steps/code_coverage/reporting/uploadPrevSha.sh)
 SO object migration (.buildkite/scripts/steps/archive_so_migration_snapshot.sh)
 Upload static site (.buildkite/scripts/steps/code_coverage/reporting/uploadStaticSite.sh)
 ES Snapshot manifest upload (.buildkite/scripts/steps/es_snapshots/create_manifest.ts)
 Scalability? (.buildkite/scripts/steps/functional/scalability_dataset_extraction.sh)
 Benchmarking (.buildkite/scripts/steps/scalability/benchmarking.sh)
 Webpack bundle analyzer (.buildkite/scripts/steps/webpack_bundle_analyzer/upload.ts)
 Build chromium (x-pack/build_chromium/build.py)
+ bazel cache

[jbudz]

Sharing in case there's any interest:

https://cloud.google.com/iam/docs/best-practices-service-accounts#choose-when-to-use suggests a combination of both. Default application credentials with permission to impersonate service accounts for specific areas.

https://cloud.google.com/docs/authentication/use-service-account-impersonation

[delanni]

I like this idea, but it's not far from managing and assuming roles manually. We'll still have to manage (probably through terraform as you suggested) the sevice accounts - we'd save on generating/storing the keys though.
I'll see how difficult is it to get the default account set up with rights to impersonate.

[delanni]

The gcloud service account we get by default in the new agents is this one: vault-gcp-authentication@elastic-ci-prod.iam.gserviceaccount.com
- this probably shouldn't have all the rights to impersonate any other service accounts, it looks like it's limited to accessing vault.

So if we'd like to have a power-user who can impersonate the bucket-limited service accounts, we'd need +1 account. Hopefully, I could set it up so that the account is activated in the pre-command hooks, so at the call sites, one would just need to use an impersonation flag, or command - then we wouldn't need to store/retrieve all the individual keys for the service accounts, just know their names.

@jbudz / @mistic - does this plan look OK? If so, I'll go ahead and create, setup the new account with rights.

[jbudz]

Agree with the plan, sounds good.

[delanni]

@jbudz - should I also move the CDN account's use to this form of impersonation, and include it to the set of accounts managed together by terraform?

[jbudz]

(We discussed on slack, we're holding off for now)

[delanni]

/ci

[delanni]

It seems the service-account proxy + impersonation seems to work, however, the usage is not without friction. Observe this positive and negative example:

➜  kibana git:(fix-gcs-access-in-elastic-buildkite) ✗ gcloud auth list --filter="status=ACTIVE" --format="value(account)"                                                                 
kibana-ci-sa-proxy@elastic-kibana-ci.iam.gserviceaccount.com
➜  kibana git:(fix-gcs-access-in-elastic-buildkite) ✗ gcloud config get auth/impersonate_service_account            
kibana-ci-access-es-snapshots@elastic-kibana-ci.iam.gserviceaccount.com
➜  kibana git:(fix-gcs-access-in-elastic-buildkite) ✗ gcloud storage cp test.txt gs://kibana-ci-es-snapshots-daily/test-alex.txt
WARNING: This command is using service account impersonation. All API calls will be executed as [kibana-ci-access-es-snapshots@elastic-kibana-ci.iam.gserviceaccount.com].
WARNING: This command is using service account impersonation. All API calls will be executed as [kibana-ci-access-es-snapshots@elastic-kibana-ci.iam.gserviceaccount.com].
Copying file://test.txt to gs://kibana-ci-es-snapshots-daily/test-alex.txt
⠼WARNING: This command is using service account impersonation. All API calls will be executed as [kibana-ci-access-es-snapshots@elastic-kibana-ci.iam.gserviceaccount.com].                               
⠧WARNING: This command is using service account impersonation. All API calls will be executed as [kibana-ci-access-es-snapshots@elastic-kibana-ci.iam.gserviceaccount.com].                               
⠛WARNING: This command is using service account impersonation. All API calls will be executed as [kibana-ci-access-es-snapshots@elastic-kibana-ci.iam.gserviceaccount.com].                               
⠼WARNING: This command is using service account impersonation. All API calls will be executed as [kibana-ci-access-es-snapshots@elastic-kibana-ci.iam.gserviceaccount.com].                               
  Completed files 1/1 | 16.0B/16.0B                                                                                                                                                                       
➜  kibana git:(fix-gcs-access-in-elastic-buildkite) ✗ gcloud storage cp test.txt gs://ci-artifacts.kibana.dev/test-alex.txt     
WARNING: This command is using service account impersonation. All API calls will be executed as [kibana-ci-access-es-snapshots@elastic-kibana-ci.iam.gserviceaccount.com].
WARNING: This command is using service account impersonation. All API calls will be executed as [kibana-ci-access-es-snapshots@elastic-kibana-ci.iam.gserviceaccount.com].
ERROR: (gcloud.storage.cp) User [kibana-ci-sa-proxy@elastic-kibana-ci.iam.gserviceaccount.com] does not have permission to access b instance [ci-artifacts.kibana.dev] (or it may not exist): kibana-ci-access-es-snapshots@elastic-kibana-ci.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket. Permission 'storage.objects.list' denied on resource (or it may not exist).

ℹ️ The first 2 commands show that my main identity was the sa-proxy and I was impersonating kibana-ci-access-es-snapshots to be able to copy the kibana-ci-es-snapshots-daily bucket.

⚠️ The 3rd command shows a successful copy to the bucket we granted access to through the impersonation. You can see, however, that the output is (even after several attempts) swarmed with warnings.

❌ The 4th command shows a failed copy, showing that the grant only applies to a select bucket. Once again, though, in the output the error shows the warnings and an error as if the call was being made with the sa-proxy account, which is a bit misleading.

[delanni]

/ci

[delanni]

/ci

[delanni]

I set up this script to make it easy to set up the impersonation before calling gcloud storage / gsutil commands. However, I didn't want the script caller to have to refer to e-mails that are associated with the service-accounts for the buckets.

Do you think it's okay to have this bucket name -> service account mapping (from line 49) to make it handier to use the script (with defaults/fallbacks for activating based on the email).

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: c274d87
• Storybooks Preview

Failed CI Steps

• FTR Configs #6

Test Failures

• [job] [logs] FTR Configs #6 / cases security and spaces enabled: basic Common bulk_create_attachments alerts security_solution "after each" hook for "should add the case ID to the alert schema when the user has read access only"

Metrics [docs]

✅ unchanged

History

• 💚 Build #191571 succeeded 9767b22
• 💔 Build #191554 failed 0c71f47
• 💚 Build #191434 succeeded 5a91a62
• 💛 Build #191378 was flaky b12ce66
• 💔 Build #191374 failed b12ce66
• 💔 Build #191370 failed 86c4729

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[delanni]

Merging this tomorrow (02-07) with more active time to monitor