Add daily job to cleanup stale resources #1700
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| --- | ||
| version: "1.0" | ||
|
|
||
| accounts: | ||
| - name: "${ACCOUNT_PROJECT}" | ||
| driver: "aws" | ||
| options: | ||
| key: '${ACCOUNT_KEY}' | ||
| secret: '${ACCOUNT_SECRET}' | ||
|
|
||
| scanners: | ||
| - account_name: "${ACCOUNT_PROJECT}" | ||
| resources: | ||
| - type: 'node' | ||
| regions: | ||
| - us-east-1 | ||
| filters: | ||
| - type: "<" | ||
| pointer: "/created_at" | ||
| param: "${CREATION_DATE}" | ||
| converters: | ||
| param: "date" | ||
| - type: "regex" | ||
| pointer: "/extra/tags/repo" | ||
| param: "^(elastic-package|integrations)" | ||
| - type: "=" | ||
| pointer: "/extra/tags/environment" | ||
| param: "ci" | ||
| - type: "regex" | ||
| pointer: "/name" | ||
| param: "^elastic-package-(.*)" | ||
| - type: "!=" | ||
| pointer: "/state" | ||
| param: "unknown" | ||
| - type: "!=" | ||
| pointer: "/state" | ||
| param: "terminated" | ||
| - type: 'object_storage_bucket' | ||
| regions: | ||
| - us-east-1 | ||
| filters: | ||
| - type: "<" | ||
| pointer: "/created_at" | ||
| param: "${CREATION_DATE}" | ||
| converters: | ||
| param: "date" | ||
| value: "date" | ||
| - type: "regex" | ||
| pointer: "/extra/tags/repo" | ||
| param: "^(elastic-package|integrations)" | ||
| - type: "=" | ||
| pointer: "/extra/tags/environment" | ||
| param: "ci" | ||
| - type: "regex" | ||
| pointer: "/name" | ||
| param: "^elastic-package-(.*)" | ||
| - type: 'queue' | ||
| regions: | ||
| - us-east-1 | ||
| filters: | ||
| - type: "<" | ||
| pointer: "/extra/tags/created_at" | ||
| param: "${CREATION_DATE}" | ||
| converters: | ||
| param: "date" | ||
| value: "date_epoch_ms" | ||
| - type: "regex" | ||
| pointer: "/extra/tags/repo" | ||
| param: "^(elastic-package|integrations)" | ||
| - type: "=" | ||
| pointer: "/extra/tags/environment" | ||
| param: "ci" | ||
| - type: "regex" | ||
| pointer: "/id" | ||
| param: "^https://(.*)/elastic-package-(.*)" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| --- | ||
| version: "1.0" | ||
|
|
||
| accounts: | ||
| - name: "${ACCOUNT_PROJECT}" | ||
| driver: "gce" | ||
| options: | ||
| key: "${ACCOUNT_KEY}" | ||
| secret: "${ACCOUNT_SECRET}" | ||
| project: "${ACCOUNT_PROJECT}" | ||
|
|
||
| scanners: | ||
| - account_name: "${ACCOUNT_PROJECT}" | ||
| resources: | ||
| - type: "node" | ||
| regions: | ||
| - "us-east1" | ||
| filters: | ||
| - type: "<" | ||
| pointer: "/extra/creationTimestamp" | ||
| param: "${CREATION_DATE}" | ||
| converters: | ||
| param: "date" | ||
| value: "date" | ||
| - type: "=" | ||
| pointer: "/extra/labels/repo" | ||
| param: "elastic-package" | ||
| - type: "=" | ||
| pointer: "/extra/labels/environment" | ||
| param: "ci" | ||
| - type: "regex" | ||
| pointer: "/name" | ||
| param: "^elastic-package-(.*)" | ||
| - type: "!=" | ||
| pointer: "/state" | ||
| param: "unknown" | ||
| - type: "!=" | ||
| pointer: "/state" | ||
| param: "terminated" | ||
jsoriano marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| # yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json | ||
|
|
||
| # Removes stale Cloud resources (AWS and GCP) having matching labels, name prefixes and older than 24 hours | ||
| name: elastic-package-cloud-cleanup | ||
|
|
||
| env: | ||
| DOCKER_REGISTRY: docker.elastic.co | ||
| NOTIFY_TO: "ecosystem-team@elastic.co" | ||
|
|
||
| steps: | ||
| - label: "Cloud Cleanup" | ||
| key: "cloud-cleanup" | ||
| command: ".buildkite/scripts/cloud-cleanup.sh" | ||
| env: | ||
| RESOURCE_RETENTION_PERIOD: "24 hours" | ||
| DRY_RUN: "true" | ||
| agents: | ||
| provider: "gcp" | ||
|
|
||
| notify: | ||
| - email: "$NOTIFY_TO" | ||
| if: "build.state == 'failed' && build.env('BUILDKITE_PULL_REQUEST') == 'false'" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,126 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| source .buildkite/scripts/install_deps.sh | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| AWS_RESOURCES_FILE="aws.resources.txt" | ||
| GCP_RESOURCES_FILE="gcp.resources.txt" | ||
|
|
||
| RESOURCE_RETENTION_PERIOD="${RESOURCE_RETENTION_PERIOD:-"24 hours"}" | ||
| export DELETE_RESOURCES_BEFORE_DATE=$(date -Is -d "${RESOURCE_RETENTION_PERIOD} ago") | ||
|
|
||
| CLOUD_REAPER_IMAGE="${DOCKER_REGISTRY}/observability-ci/cloud-reaper:0.3.0" | ||
|
|
||
| resources_to_delete=0 | ||
|
|
||
| COMMAND="validate" | ||
| if [[ "${DRY_RUN}" != "true" ]]; then | ||
| COMMAND="plan" # TODO: to be changed to "destroy --confirm" | ||
| else | ||
| COMMAND="plan" | ||
| fi | ||
|
|
||
| any_resources_to_delete() { | ||
| local file=$1 | ||
| local number=0 | ||
| # First three lines are like: | ||
| # ⇒ Loading configuration... | ||
| # ✓ Succeeded to load configuration | ||
| # Scanning resources... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 | ||
| number=$(tail -n +4 "${file}" | wc -l) | ||
| if [ "${number}" -eq 0 ]; then | ||
| return 1 | ||
| fi | ||
| return 0 | ||
| } | ||
|
|
||
| cloud_reaper_aws() { | ||
| echo "Validating configuration" | ||
| docker run --rm -v $(pwd)/.buildkite/configs/cleanup.aws.yml:/etc/cloud-reaper/config.yml \ | ||
| -e ACCOUNT_SECRET="${ELASTIC_PACKAGE_AWS_SECRET_KEY}" \ | ||
| -e ACCOUNT_KEY="${ELASTIC_PACKAGE_AWS_ACCESS_KEY}" \ | ||
| -e ACCOUNT_PROJECT="${ELASTIC_PACKAGE_AWS_USER_SECRET}" \ | ||
| -e CREATION_DATE="${DELETE_RESOURCES_BEFORE_DATE}" \ | ||
| "${CLOUD_REAPER_IMAGE}" \ | ||
| cloud-reaper \ | ||
| --config /etc/cloud-reaper/config.yml \ | ||
| validate | ||
|
|
||
| echo "Scanning resources" | ||
| docker run --rm -v $(pwd)/.buildkite/configs/cleanup.aws.yml:/etc/cloud-reaper/config.yml \ | ||
| -e ACCOUNT_SECRET="${ELASTIC_PACKAGE_AWS_SECRET_KEY}" \ | ||
| -e ACCOUNT_KEY="${ELASTIC_PACKAGE_AWS_ACCESS_KEY}" \ | ||
| -e ACCOUNT_PROJECT="${ELASTIC_PACKAGE_AWS_USER_SECRET}" \ | ||
| -e CREATION_DATE="${DELETE_RESOURCES_BEFORE_DATE}" \ | ||
| "${CLOUD_REAPER_IMAGE}" \ | ||
| cloud-reaper \ | ||
| --config /etc/cloud-reaper/config.yml \ | ||
| ${COMMAND} | tee "${AWS_RESOURCES_FILE}" | ||
| } | ||
|
|
||
| cloud_reaper_gcp() { | ||
| echo "Validating configuration" | ||
| docker run --rm -v $(pwd)/.buildkite/configs/cleanup.gcp.yml:/etc/cloud-reaper/config.yml \ | ||
| -e ACCOUNT_SECRET="${ELASTIC_PACKAGE_GCP_KEY_SECRET}" \ | ||
| -e ACCOUNT_KEY="${ELASTIC_PACKAGE_GCP_EMAIL_SECRET}" \ | ||
| -e ACCOUNT_PROJECT="${ELASTIC_PACKAGE_GCP_PROJECT_SECRET}" \ | ||
| -e CREATION_DATE="${DELETE_RESOURCES_BEFORE_DATE}" \ | ||
| "${CLOUD_REAPER_IMAGE}" \ | ||
| cloud-reaper \ | ||
| --config /etc/cloud-reaper/config.yml \ | ||
| validate | ||
|
|
||
| echo "Scanning resources" | ||
| docker run --rm -v $(pwd)/.buildkite/configs/cleanup.gcp.yml:/etc/cloud-reaper/config.yml \ | ||
| -e ACCOUNT_SECRET="${ELASTIC_PACKAGE_GCP_KEY_SECRET}" \ | ||
| -e ACCOUNT_KEY="${ELASTIC_PACKAGE_GCP_EMAIL_SECRET}" \ | ||
| -e ACCOUNT_PROJECT="${ELASTIC_PACKAGE_GCP_PROJECT_SECRET}" \ | ||
| -e CREATION_DATE="${DELETE_RESOURCES_BEFORE_DATE}" \ | ||
| "${CLOUD_REAPER_IMAGE}" \ | ||
| cloud-reaper \ | ||
| --config /etc/cloud-reaper/config.yml \ | ||
| ${COMMAND} | tee "${GCP_RESOURCES_FILE}" | ||
| } | ||
|
|
||
| echo "--- Cleaning up GCP resources older than ${DELETE_RESOURCES_BEFORE_DATE}..." | ||
| cloud_reaper_gcp | ||
|
|
||
| if any_resources_to_delete "${GCP_RESOURCES_FILE}"; then | ||
| echo "Pending GCP resources" | ||
| resources_to_delete=1 | ||
| fi | ||
|
|
||
| echo "--- Cleaning up AWS resources older than ${DELETE_RESOURCES_BEFORE_DATE}..." | ||
| cloud_reaper_aws | ||
|
|
||
| if any_resources_to_delete "${AWS_RESOURCES_FILE}" ; then | ||
| echo "Pending AWS resources" | ||
| resources_to_delete=1 | ||
| fi | ||
|
|
||
| if [ "${resources_to_delete}" -eq 1 ]; then | ||
| message="There are resources to be deleted" | ||
| echo "${message}" | ||
| if running_on_buildkite ; then | ||
| buildkite-agent annotate \ | ||
| "${message}" \ | ||
| --context "ctx-cloud-reaper-error" \ | ||
| --style "error" | ||
| fi | ||
| exit 1 | ||
| fi | ||
|
|
||
| # TODO: List and delete the required resources using aws cli | ||
| echo "--- Cleaning up other AWS resources older than ${DELETE_RESOURCES_BEFORE_DATE}" | ||
| echo "--- Installing awscli" | ||
| with_aws_cli | ||
|
There was a problem hiding this comment. What is the deadline? So we can add support for other resources in There was a problem hiding this comment. There is no hard deadline for this. We were planning to merge this PR to start checking (with We were thinking to add in following PRs, at least listing and filtering by tags for the other resources using Should we wait for the There was a problem hiding this comment. If it can wait for some time, I should be able to add support for those resources. There was a problem hiding this comment.
I think probably we could wait for some time, WDYT @jsoriano @kpollich ? I've just created a PR to list if there is any AWS Redshift cluster stale due to CI builds. That would allow us to receive email notifications if there is any left cluster until that support is added into cloud-reaper: Once this support is added into In any case, this PR could be merged to start checking there is no resource like instances or queues. There was a problem hiding this comment. |
||
|
|
||
| export AWS_ACCESS_KEY_ID="${ELASTIC_PACKAGE_AWS_ACCESS_KEY}" | ||
| export AWS_SECRET_ACCESS_KEY="${ELASTIC_PACKAGE_AWS_ACCESS_KEY}" | ||
| export AWS_DEFAULT_REGION=us-east-1 | ||
|
|
||
| echo "--- TODO: Cleaning up Redshift clusters" | ||
| echo "--- TODO: Cleaning up IAM roles" | ||
| echo "--- TODO: Cleaning up IAM policies" | ||
| echo "--- TODO: Cleaning up Schedulers" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.