[BREAKING CHANGES] Refactor of CI/CD, kustomize & skaffold #1221
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@bourgeoisor As discussed, PR is now ready for review (again 😜) - CI/CD pipelines are now service-scoped instead of team-scoped and e2e-tests are only run in CI-PR and when deploying to production to avoid flakiness from multiple CI-test instances running in parallel. |
|
@bourgeoisor anything I can do to help review/push this one? |
|
@aablsk Thank you! Would you be able to resolve the merge conflicts? |
…-anthos into private-clusters
Done 👍 |
|
@aablsk - Can you TAL at the conflicts on this PR? There seems to be more conflicts on the files changed in this PR! |
…-anthos into private-clusters
|
@Shabirmean Done! I merged current GoogleCloudPlatform/bank-of-anthos main into PR. Conflicts will continue to arise when there are changes to GoogleCloudPlatform/bank-of-anthos main. |
|
Thank you @aablsk. Can you TAL at the failing CI unit test? @bourgeoisor - is this change something we intend to have merged? Seems like conflicts continue to arise the longer we wait on it! |
|
@Shabirmean yes the plan is to do so this week. |
This was referenced Mar 29, 2023
Closed
This was referenced Mar 30, 2023
Closed
big-appled
pushed a commit
to big-appled/bank-of-anthos
that referenced
this pull request
Jun 9, 2023
…udPlatform#1221) * add ACM base * add ACM overlays for development, staging, production environments * add tf infrastructure definition * add cicd pipeline custom tf module * add shell scripts for easy setup * add README documenting how to setup multienv-cicd-anthos-autopilot * substitute projectId and region references in ACM config * substitute projectId and region references in ACM config * add configurable sync_branch * remove obsolete file * fix license header * replace hard-coded script names with variable * made development cluster private * make staging and production private clusters as well * make EOF consistent empty line * src split into teams, kustomize overlays, skaffold profiles * clean up * improve teardown process * substitute projectId and region references in ACM config * upgrade cloudbuild.yaml to skaffold v2 * fix access for private clusters * remove commented master_authorized_networks * substitute projectId and region references in ACM config * remove ZONE variable * move targets, teams to tf.locals * remove shell scripts, update README.md to contain manual steps * add instructions for terraform state bucket * fix terraform, update README.md, reintroduce zone for CloudSQL master instance * substitute projectId and region references in ACM config * fix skaffold v2 deploy configuration for nested skaffold.yamls * updated for $PROJECT_ID and $REGION * fix deployment to private anthos cluster * fix deployment to anthos cluster * add endpoint configuration for production environment * fix deployment to anthos cluster through connect gateway * add first cloudbuild definition for PR CI * fix style issues * add logging configuration * fix test execution * add prep step for maven builds * fix addlicense, fix skaffold build * change go install instructions * switch to GOROOT from GOPATH * hopefully fix go install * maybe fix go install * fix go install * try to fix go install * bump go version * fix addlicense * skip tests in build phase * ci fixes * ci fixes * fix ci * add licenses * remove unnecessary env var from ci * add go/ to gitignore for ci * fix contacts skaffold.yaml * add pr cache bucket * install pylint * fix ci * install python-venv * switch to python3-venv * fix arg * possibly fix test step * fix python tests * fix python tests * auto yes apt-get install * separate python environments for all tests * fix style issues * move dependencies from skaffold.yaml into ci-pr * improve naming of python venv * switch test definition to python3 * add balancereader tests * add /workspace to requirements.txt * install pytest * fix python tests * add tests for ledger services * add frontend tests * fix frontend pylint test * removed pytest for frontend * fix pylint for frontend * add tests to build pipeline * generalize init-db jobs into kustomize component * fix tests * fix cloudbuild tests * fix filename * fix env var reference * fix frontend tests in ci * fix test frontend ci * try another update for fixing frontend ci * fix issues, re-enable statuschecks * fix skaffold version * fix skaffold version * patch skaffold version * update cloud deploy image version * cloudbuild yaml skaffold version for cloud deploy * deploy skaffold_preview * rollback tolerateFailuresUntilDeadline * roll back skaffold apiVersion * remove irrelevant files from docker images * upgrade to v4beta1 * upgrade to v4beta1 * dev setup, pr namespace, e2e tests * fix flaky test * fix ci-pr cloudbuild.yaml * add license header * pass MAVEN_USER_HOME to skaffold * update for ci pr * fix ci ledger * fix iam bindings for cloudbuild pr sa * fix sql db initialization * improve ci-pr * fix artifacts indentation * fix tags indentation * fix substitution * fix substitution * fix tags * set skaffold profile correctly * fix verify * fix verify job 🤞 * potential fix for uploading screenshots * add verify to cloud deploy * small fix to trigger deploy * fix verification in clouddeploy * fix e2e tests * fix license header * fix e2e-test config for staging * fix e2e image * fix e2e container image * fix e2e tests * fix ci pr * fix pipeline * fix bucket reference for ci-pr * improve log message for cypress screenshots * fix E2E tests * exit with status code 1 if cypress fails instead of relying on screenshot folder * improve readability in google cloud build * fix e2e tests * fix file upload * remove error for testing artifact upload of e2e test failures * fix e2e tests for production * fix ci-pr * potentially fix e2e tests for accounts, ledger * remove verify from skaffold.yaml for accounts, ledger * fix e2e tests in skaffold * request faster machine for ci pr builds * fix ci-pr workload ksa * upgrade and fix cypress tests * fix license header * fix ci-pr * fix e2e tests * possible fix for ci-pr * fix skaffold deploy * add ci-pr to terraform * fix ci pr * remove cloud build artifacts from ci-pr * substitute projectId and region references in ACM config * fix README * remove projectid and region * remove bielski domain * update README with region information for cloud build * fix region for ci-pr pipeline * reduce ui test flakiness * "substitute $PROJECT_ID and $REGION references in ACM config" * update terraform.tfvars for bank-of-anthos-ci * update README with setup steps * fix region tags * add removed region tag * ci fixes for compatibility with new source structure * add removed region tags in skaffold.yaml * removed helper scripts for reproduction tests * fix terraform provider config * substitute boa-aablsk-delivery-refactor and europe-west1 references in ACM config * refactor pipelines to microservice scope * fix skaffold test execution * fix pylint issue * fix ledger ci * maybe fix PR ci * possible fix for jib maven build * potential fix for jib builds * bump skaffold version for java 17 * fix release names, ledger pipeline * fix pipelines * fix ledger pipeline * fix ledger pipeline * update account ci * remove unnecessary workaround * potentially fix ci-pr * fix release names * potentially fix ci pr * fix ci pr * re-added java build workaround * ci pr * CI PR * possibly fix ci pr * ci pr * possible fix for ci-pr * fix substitutions * potential fix for ci pr * possible fix for ledger builds * improve cloudbuild.yamls * fix deploy dependencies in ci-pr * add e2e tests * fix verify * improve README.md * update README.md * fix domain for prod * update project specific values for bank-of-anthos-ci deployment * set config for new repro * fix e2e tests * fix sql script * fix sql script * fix e2e tests for production * change timeouts and retries * fix load generator * fix screenshot upload to bucket * reduce number of CD targets, make VERIFY production only * remove namespace from loadgenerator base to avoid conflict in ci-pr and skaffold dev * fix delivery pipelines * revert any config values to bank-of-anthos-ci env * bump java to 17 * update documentation --------- Co-authored-by: Shabir Mohamed Abdul Samadh <7249208+Shabirmean@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
Initially the fork that this PR is based on was created to showcase Google Cloud's CICD story encompassing Cloud Build / Cloud Deploy / kustomize / skaffold. Going further we implemented the current bank-of-anthos CI with Cloud Build, integrated pre- and post-deployment tests with skaffold for an attractive devX story from inner to outer loop.
Change Summary
I'm still working on the change summary and link to a document including rationale for decisions.
Additional Notes
Will add these while working on the change summary and rationales.
Testing Procedure
Follow instructions in iac/tf-multienv-cicd-anthos-autopilot/README.md
Related PRs or Issues
Previous PR including terraform infrastructure