Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: enable remote tests #5103

Merged
merged 33 commits into from
Oct 2, 2023
Merged

test: enable remote tests #5103

merged 33 commits into from
Oct 2, 2023

Conversation

stefreak
Copy link
Member

@stefreak stefreak commented Sep 18, 2023

What this PR does / why we need it:

We enable the remote tests to run in CI. This includes integration tests that are requiring some form of remote service access. In order to get these tests to work again, we have had to:

  • Replace the tests using a remote container registry. Previously we accessed docker hub with encrypted credentials in the repo. After this PR we instead use Google Artifact Registry. Access is based on a google cloud user group which retrieves the credentials needed for both local and remote (k8s container) registry operations.
  • Change test initialization to use the new credentials to be accessible both locally and in k8s containers
  • Make sure tests are independent(*) and can run with a clean initial state
  • Add/rewrite utility functions related to accessing and deleting images in the GAR and the local docker image cache
  • Remove dead code related k8s build status and running k8s builds

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

There are a couple of test suites that are disabled:

  • With the remote registry we have introduced shared state and cannot run tests concurrently.
  • Buildkit acquires a lock to ensure the container exists, this lock is sometimes causing the entire test suite to block for up to a minute.
  • Local mode tests that were disabled before we started this work

Tests in the disabled suites have been passing locally when we have been able to ensure there is no concurrent access to the registry. We plan to fix issues with concurrent and slow running tests in an upcoming PR. The affected tests are mainly related to local-mode and in-cluster building with kaniko and buildkit.

@stefreak stefreak changed the title test: WIP enable remote tests test: enable remote tests Sep 18, 2023
@stefreak stefreak marked this pull request as draft September 18, 2023 14:44
core/test/integ/helpers.ts Outdated Show resolved Hide resolved
core/test/integ/helpers.ts Show resolved Hide resolved
core/test/integ/helpers.ts Outdated Show resolved Hide resolved
core/test/integ/helpers.ts Outdated Show resolved Hide resolved
core/src/plugins/container/helpers.ts Show resolved Hide resolved
core/src/plugins/container/helpers.ts Outdated Show resolved Hide resolved
stefreak and others added 22 commits September 29, 2023 12:05
…til pod

Before this fix we sometimes synced to or from the old Pod

Co-authored-by: Mikael Högqvist Tabor <mkhq@users.noreply.github.com>
…ests

This allows google artifact registry access in the garden-ci project.
Using the service account impersonation has the benefit that we can allow any google group to impersonate this service account, so we do not need to manage access manually:
This only works because we granted the google group "dev@garden.io" permission to impersonate this service account with the following command:
`gcloud iam service-accounts add-iam-policy-binding gar-serviceaccount@garden-ci.iam.gserviceaccount.com --member=group:dev@garden.io --role=roles/iam.serviceAccountTokenCreator`
We could extend this in the future to allow access to the CI cluster in the same way, without the need to manage users manually.

Co-authored-by: twelvemo <anna@garden.io>
Co-authored-by: Steffen Neubauer <stefreak@users.noreply.github.com>
Co-authored-by: Steffen Neubauer <stefreak@users.noreply.github.com>
 Co-authored-by: Steffen Neubauer <stefreak@users.noreply.github.com>
@mkhq mkhq marked this pull request as ready for review October 2, 2023 07:07
mkhq and others added 2 commits October 2, 2023 15:35
- move error handling of impersonated client to point of creation

Co-authored-by: Steffen Neubauer <stefreak@users.noreply.github.com>
mkhq
mkhq previously approved these changes Oct 2, 2023
@mkhq mkhq enabled auto-merge October 2, 2023 15:19
@mkhq mkhq added this pull request to the merge queue Oct 2, 2023
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 2, 2023
@stefreak stefreak added this pull request to the merge queue Oct 2, 2023
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 2, 2023
@mkhq mkhq added this pull request to the merge queue Oct 2, 2023
Merged via the queue into main with commit d2c79de Oct 2, 2023
43 checks passed
@mkhq mkhq deleted the enable-remote-tests branch October 2, 2023 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants