Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for persistent data volume #2

Merged
merged 5 commits into from
Feb 14, 2024
Merged

Add support for persistent data volume #2

merged 5 commits into from
Feb 14, 2024

Conversation

hellais
Copy link
Member

@hellais hellais commented Feb 13, 2024 •
edited

  • add basic readme file
  • also fix deploy checks when a branch lands on master, since it fails because the comment script doesn't have in it the reference to the open PR number

@hellais
Disable deploy checks when a branch lands on master
8d65853 
It fails because the comment script doesn't have in it the reference to
the open PR
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 13, 2024 •
edited

Terraform Run Output 🤖

Format and Style 🖌success

Initialization ⚙️success

Validation 🤖success

Validation Output 

$ terraform validate
Success! The configuration is valid.

Plan 📖success

  • Plan: 0 to add, 1 to change, 0 to destroy.
Show Plan 

$ terraform plan
Acquiring state lock. This may take a few moments...
module.terraform_state_backend.data.aws_iam_policy_document.bucket_policy[0]: Reading...
data.aws_availability_zones.available: Reading...
aws_cloudwatch_log_group.app: Refreshing state... [id=tf-ecs-group/app-dataapi]
data.aws_ami.debian_ami: Reading...
module.terraform_state_backend.data.aws_region.current: Reading...
aws_acm_certificate.dataapi: Refreshing state... [id=arn:aws:acm:eu-central-1:082866812839:certificate/27b7c5e3-ef70-41e8-a6cf-ac0d58a99613]
module.terraform_state_backend.aws_dynamodb_table.with_server_side_encryption[0]: Refreshing state... [id=ooni-production-terraform-state-lock]
aws_iam_role.app_instance: Refreshing state... [id=tf-ecs-ooni-instance-role]
aws_ecs_cluster.main: Refreshing state... [id=arn:aws:ecs:eu-central-1:082866812839:cluster/ooni-ecs-cluster]
module.terraform_state_backend.aws_s3_bucket.default[0]: Refreshing state... [id=ooni-production-terraform-state]
module.terraform_state_backend.data.aws_region.current: Read complete after 0s [id=eu-central-1]
aws_cloudwatch_log_group.ecs: Refreshing state... [id=tf-ecs-group/ecs-agent]
module.terraform_state_backend.data.aws_iam_policy_document.bucket_policy[0]: Read complete after 0s [id=2050088263]
data.aws_ssm_parameter.ecs_optimized_ami: Reading...
aws_vpc.main: Refreshing state... [id=vpc-08b101405472a0b46]
data.aws_ssm_parameter.ecs_optimized_ami: Read complete after 0s [id=/aws/service/ecs/optimized-ami/amazon-linux-2/recommended]
data.aws_availability_zones.available: Read complete after 0s [id=eu-central-1]
aws_iam_role.ecs_service: Refreshing state... [id=ooni_ecs_role]
module.terraform_state_backend.data.aws_iam_policy_document.aggregated_policy[0]: Reading...
module.terraform_state_backend.data.aws_iam_policy_document.aggregated_policy[0]: Read complete after 0s [id=2050088263]
aws_iam_instance_profile.app: Refreshing state... [id=tf-ecs-instprofile]
data.aws_ami.debian_ami: Read complete after 0s [id=ami-0e626c31414223120]
aws_iam_role_policy.instance: Refreshing state... [id=tf-ecs-ooni-instance-role:TfEcsOONIInstanceRole]
aws_ecs_task_definition.dataapi: Refreshing state... [id=ooni-dataapi-production-td]
aws_iam_role_policy.ecs_service: Refreshing state... [id=ooni_ecs_role:ooni_ecs_policy]
aws_route53_record.dataapi_cert_validation["dataapi.prod.ooni.io"]: Refreshing state... [id=Z02418652BOD91LFA5S9X__74a0d0e91f58697b6838bed5a2fb4939.dataapi.prod.ooni.io._CNAME]
aws_acm_certificate_validation.dataapi: Refreshing state... [id=2024-02-05 11:11:01.598 +0000 UTC]
aws_internet_gateway.gw: Refreshing state... [id=igw-05319878d828e0a53]
aws_subnet.main[0]: Refreshing state... [id=subnet-06d47d18b015109d4]
aws_subnet.main[1]: Refreshing state... [id=subnet-02241f60bd951358d]
aws_alb_target_group.dataapi: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:targetgroup/ooni-ecs-dataapi/c16001d24f858f20]
aws_security_group.clickhouse_sg: Refreshing state... [id=sg-0c0ffdfe535bededa]
aws_security_group.lb_sg: Refreshing state... [id=sg-090ff83a71e7ad235]
aws_route_table.r: Refreshing state... [id=rtb-06814accd93aca9ce]
aws_instance.clickhouse_server_prod_tier1: Refreshing state... [id=i-0aaa354af74a5e988]
module.terraform_state_backend.aws_s3_bucket_server_side_encryption_configuration.default[0]: Refreshing state... [id=ooni-production-terraform-state]
module.terraform_state_backend.aws_s3_bucket_public_access_block.default[0]: Refreshing state... [id=ooni-production-terraform-state]
module.terraform_state_backend.aws_s3_bucket_versioning.default[0]: Refreshing state... [id=ooni-production-terraform-state]
aws_route_table_association.a[0]: Refreshing state... [id=rtbassoc-049f8c5e84cb32e55]
aws_route_table_association.a[1]: Refreshing state... [id=rtbassoc-034c4831c1754e284]
aws_alb.main: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:loadbalancer/app/ooni-alb-ecs/4cf8bd00b53f8961]
aws_security_group.instance_sg: Refreshing state... [id=sg-0f39450a61732d69b]
module.terraform_state_backend.aws_s3_bucket_policy.default[0]: Refreshing state... [id=ooni-production-terraform-state]
aws_launch_template.app: Refreshing state... [id=lt-01b923563af0432b5]
module.terraform_state_backend.time_sleep.wait_for_aws_s3_bucket_settings[0]: Refreshing state... [id=2024-02-02T21:32:23Z]
module.terraform_state_backend.aws_s3_bucket_ownership_controls.default[0]: Refreshing state... [id=ooni-production-terraform-state]
aws_autoscaling_group.app: Refreshing state... [id=ooni-tier1-production-backend-asg20240119184843559900000002]
aws_route53_record.alb_dns: Refreshing state... [id=Z02418652BOD91LFA5S9X_dataapi.prod.ooni.io_A]
aws_alb_listener.front_end: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:listener/app/ooni-alb-ecs/4cf8bd00b53f8961/1cd9240a34a13129]
aws_alb_listener.front_end_https: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:082866812839:listener/app/ooni-alb-ecs/4cf8bd00b53f8961/b7ec77ceadac94c8]
aws_ecs_service.dataapi: Refreshing state... [id=arn:aws:ecs:eu-central-1:082866812839:service/ooni-ecs-cluster/ooni-ecs-dataapi-production]
data.aws_ebs_volume.clickhouse_data_volume: Reading...
aws_eip.clickhouse_ip: Refreshing state... [id=eipalloc-0f0a20bba4a1d858c]
aws_route53_record.clickhouse_dns: Refreshing state... [id=Z035992527R8VEIX2UVO0_clickhouse.tier1.prod.ooni.nu_A]
data.aws_ebs_volume.clickhouse_data_volume: Read complete after 0s [id=vol-0753909d310ed6d8d]
aws_volume_attachment.clickhouse_data_volume_attachment: Refreshing state... [id=vai-2291175638]
local_file.ansible_inventory: Refreshing state... [id=e7b3f71fb68cc29f026b251ba92a9bfe9a6e4fd6]
null_resource.ansible_update_known_hosts: Refreshing state... [id=4418919173081010018]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_ecs_service.dataapi will be updated in-place
  ~ resource "aws_ecs_service" "dataapi" {
        id                                 = "arn:aws:ecs:eu-central-1:082866812839:service/ooni-ecs-cluster/ooni-ecs-dataapi-production"
        name                               = "ooni-ecs-dataapi-production"
        tags                               = {
            "Name"       = "ooni-tier1-production"
            "Repository" = "https://github.com/ooni/devops"
        }
      ~ triggers                           = {
          ~ "redeployment" = "2024-02-14T13:54:23Z" -> "2024-02-14T13:58:48Z"
        }
        # (15 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.
Pusher @hellais
Action pull_request
Working Directory ./tf/environments/production
Workflow .github/workflows/check_deploy.yml
Last updated Wed, 14 Feb 2024 13:58:54 GMT

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 13, 2024 •
edited

Ansible Run Output 🤖

Ansible Playbook Recap 🔍


clickhouse.tier1.prod.ooni.nu : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

Ansible playbook output 📖success

Show Execution 

$ ansible-playbook playbook.yml --check --diff -i inventory.ini

PLAY [ClickHouse servers] ******************************************************

TASK [Gathering Facts] *********************************************************
fatal: [clickhouse.tier1.prod.ooni.nu]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:wuGEgIlSHvtKTYutVHFude7FVWDt4KkdropiF+iHCew.
Please contact your system administrator.
Add correct host key in known_hosts to get rid of this message.
Offending ED25519 key in known_hosts:5
  remove with:
  ssh-keygen -f "known_hosts" -R "clickhouse.tier1.prod.ooni.nu"
Host key for clickhouse.tier1.prod.ooni.nu has changed and you have requested strict checking.
Host key verification failed.", "unreachable": true}

PLAY RECAP *********************************************************************
clickhouse.tier1.prod.ooni.nu : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
Pusher @hellais
Action pull_request
Working Directory ./tf/environments/production
Workflow .github/workflows/check_deploy.yml
Last updated Wed, 14 Feb 2024 14:00:04 GMT

@hellais
Copy link
Member Author

hellais commented Feb 13, 2024

Something is funky in the plan. Need to troubleshoot that.

@hellais
Copy link
Member Author

hellais commented Feb 13, 2024

Ok I have fixed it.

What happened, was that debian updated their base OS image and so that was triggering a redeploy of the container. It's a good idea to figure out a pattern for a non destructive upgrade of the base OS, so I did that.

The basic idea is that we want to manually or semi-manually create the data volume and then pass in the tag for it as a data attribute to terraform. That way we don't destroy and recreate the data volume thereby loosing data.

I have added a comment about that.

@hellais hellais changed the title Disable deploy checks when a branch lands on master Add support for persistent data volume Feb 13, 2024
bassosimone
bassosimone previously approved these changes Feb 13, 2024
View reviewed changes
Copy link
Contributor

@bassosimone bassosimone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐳

@hellais hellais merged commit d4e22fc into main Feb 14, 2024
2 checks passed
@hellais hellais deleted the fix/gh-action branch February 14, 2024 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bassosimone bassosimone Awaiting requested review from bassosimone

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@hellais @bassosimone