Skip to content

Route53 Record + vpc endpoint extensive #411

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open

Conversation

bcarranza
Copy link
Collaborator

@bcarranza bcarranza commented Mar 1, 2023

  • Create a new route53 record to handle a generic record in route 53
  • Create a new record with sftp transfer server

Where I am using this branch

https://github.com/exact-payments/gruntwork-infrastructure-live/pull/1523/files#diff-e4e4cb018f9616b9193a6577d685fa8da8acf6abfa2037d213c0ed1b6177b767R12

Test cases

  • Test cases about indivual module: record-vpc-enpoint-sftp-transfer

  • Sftp connection test

root@debian-pod-ssh:~# sftp bcarranza@sftp-dataplatform.epops.cc
Connected to sftp-dataplatform.epops.cc.
sftp> exit
root@debian-pod-ssh:~# sftp bcarranza@sftp-dataplatform.epops.cc
Connected to sftp-dataplatform.epops.cc.
sftp> ls
bcarranza_files  example.txt      example1.txt   
  • Input

inputs = {
  route53_zone_id = dependency.route53.outputs.public_hosted_zone_map[local.domain_name]
  record_name = "sftp-dataplatform.${local.domain_name}"
  vpc_id = dependency.vpc.outputs.vpc_id
  transfer_server_id = element(split(".", dependency.sftp.outputs.transfer_endpoint), 0)
}
  • How looks like record route 53
    Screenshot 2023-03-03 at 13 09 01

  • Appy

root@cc6aa29f5343:/workspaces/gruntwork-infrastructure-live/ops/us-west-2/ops/p2a/3000-data-platform/3025-sftp-route53# terragrunt apply
WARN[0002] No double-slash (//) found in source URL /cloudposse/terraform-aws-transfer-sftp.git. Relative paths in downloaded Terraform code may not work.  prefix=[/workspaces/gruntwork-infrastructure-live/ops/us-west-2/ops/p2a/3000-data-platform/3020-sftp] 
Initializing modules...
- record in ../record

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Reusing previous version of hashicorp/null from the dependency lock file
- Reusing previous version of hashicorp/template from the dependency lock file
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/null v3.2.1...
- Installed hashicorp/null v3.2.1 (signed by HashiCorp)
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)
- Installing hashicorp/local v2.3.0...
- Installed hashicorp/local v2.3.0 (signed by HashiCorp)
- Installing hashicorp/aws v4.56.0...
- Installed hashicorp/aws v4.56.0 (signed by HashiCorp)

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Acquiring state lock. This may take a few moments...
data.template_file.log_name: Reading...
data.template_file.log_name: Read complete after 0s [id=6145fa111cd834d1c11d8ed1f63de159901010232de8c4dcfba72373f28bdef4]
data.aws_region.current: Reading...
data.aws_caller_identity.current: Reading...
data.aws_region.current: Read complete after 0s [id=us-west-2]
data.aws_caller_identity.current: Read complete after 1s [id=476264532441]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.aws_vpc_endpoint.this will be read during apply
  # (config refers to values not yet known)
 <= data "aws_vpc_endpoint" "this" {
      + arn                   = (known after apply)
      + cidr_blocks           = (known after apply)
      + dns_entry             = (known after apply)
      + dns_options           = (known after apply)
      + id                    = (known after apply)
      + ip_address_type       = (known after apply)
      + network_interface_ids = (known after apply)
      + owner_id              = (known after apply)
      + policy                = (known after apply)
      + prefix_list_id        = (known after apply)
      + private_dns_enabled   = (known after apply)
      + requester_managed     = (known after apply)
      + route_table_ids       = (known after apply)
      + security_group_ids    = (known after apply)
      + service_name          = (known after apply)
      + state                 = (known after apply)
      + subnet_ids            = (known after apply)
      + tags                  = (known after apply)
      + vpc_endpoint_type     = (known after apply)
      + vpc_id                = "vpc-0254921a46bca1455"

      + timeouts {
          + read = (known after apply)
        }
    }

  # data.local_file.get-vpc-endpoint-id-value will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "local_file" "get-vpc-endpoint-id-value" {
      + content        = (known after apply)
      + content_base64 = (known after apply)
      + filename       = "./vpc-endpoint-id.txt"
      + id             = (known after apply)
    }

  # null_resource.output-vpc-endpoint-id will be created
  + resource "null_resource" "output-vpc-endpoint-id" {
      + id = (known after apply)
    }

  # module.record.aws_route53_record.this will be created
  + resource "aws_route53_record" "this" {
      + allow_overwrite = (known after apply)
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "sftp-dataplatform.epops.cc"
      + type            = "A"
      + zone_id         = "Z10308181L9DK26UL0JK3"

      + alias {
          + evaluate_target_health = false
          + name                   = (known after apply)
          + zone_id                = (known after apply)
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + fqdn                        = (known after apply)
  + name                        = "sftp-dataplatform.epops.cc"
  + vpc_endpoint_dns_name       = (known after apply)
  + vpc_endpoint_hosted_zone_id = (known after apply)
  + vpc_endpoint_id             = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

null_resource.output-vpc-endpoint-id: Creating...
null_resource.output-vpc-endpoint-id: Provisioning with 'local-exec'...
null_resource.output-vpc-endpoint-id (local-exec): Executing: ["/bin/sh" "-c" "aws transfer describe-server --server-id s-2c18dc3b67c748018 --query 'Server.EndpointDetails.VpcEndpointId' > ./vpc-endpoint-id.txt"]
null_resource.output-vpc-endpoint-id: Creation complete after 2s [id=4522301945537840821]
data.local_file.get-vpc-endpoint-id-value: Reading...
data.local_file.get-vpc-endpoint-id-value: Read complete after 0s [id=ceb1bf6b4b041b945df013ce44a40dce56c39970]
data.aws_vpc_endpoint.this: Reading...
data.aws_vpc_endpoint.this: Read complete after 1s [id=vpce-055ff2bc56c264089]
module.record.aws_route53_record.this: Creating...
module.record.aws_route53_record.this: Still creating... [10s elapsed]
module.record.aws_route53_record.this: Still creating... [20s elapsed]
module.record.aws_route53_record.this: Still creating... [30s elapsed]
module.record.aws_route53_record.this: Still creating... [40s elapsed]
module.record.aws_route53_record.this: Still creating... [50s elapsed]
module.record.aws_route53_record.this: Creation complete after 53s [id=Z10308181L9DK26UL0JK3_sftp-dataplatform.epops.cc_A]
Releasing state lock. This may take a few moments...

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

fqdn = "sftp-dataplatform.epops.cc"
name = "sftp-dataplatform.epops.cc"
vpc_endpoint_dns_name = "vpce-055ff2bc56c264089-8drouhom.vpce-svc-046efd4167c4038e3.us-west-2.vpce.amazonaws.com"
vpc_endpoint_hosted_zone_id = "Z1YSA3EXCYUU9Z"
vpc_endpoint_id = "vpce-055ff2bc56c264089"
  • Test cases about indivual module: record
inputs = {
  
  route53_zone_id = dependency.route53.outputs.public_hosted_zone_map[local.domain_name]
  record_name = "sftp-dataplatform.${local.domain_name}"
  vpc_endpoint_dns_name = "vpce-0c5dcb513666d031b-1m3hyfue.vpce-svc-0369525cec7e36a11.us-west-2.vpce.amazonaws.com"
  vpc_endpoint_zone_id = "Z1YSA3EXCYUU9Z"
}
  • Apply
Acquiring state lock. This may take a few moments...

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_route53_record.this will be created
  + resource "aws_route53_record" "this" {
      + allow_overwrite = (known after apply)
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "sftp-dataplatform.epops.cc"
      + type            = "A"
      + zone_id         = "Z10308181L9DK26UL0JK3"

      + alias {
          + evaluate_target_health = false
          + name                   = "vpce-0c5dcb513666d031b-1m3hyfue.vpce-svc-0369525cec7e36a11.us-west-2.vpce.amazonaws.com"
          + zone_id                = "Z1YSA3EXCYUU9Z"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + fqdn = (known after apply)
  + name = "sftp-dataplatform.epops.cc"

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_route53_record.this: Creating...
aws_route53_record.this: Still creating... [10s elapsed]
aws_route53_record.this: Still creating... [19s elapsed]
aws_route53_record.this: Still creating... [29s elapsed]
aws_route53_record.this: Creation complete after 33s [id=Z10308181L9DK26UL0JK3_sftp-dataplatform.epops.cc_A]
Releasing state lock. This may take a few moments...

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

fqdn = "sftp-dataplatform.epops.cc"
name = "sftp-dataplatform.epops.cc"
  • how looks like the new route 53 record
    Screenshot 2023-03-02 at 09 51 01


resource "null_resource" "output-vpc-endpoint-id" {
provisioner "local-exec" {
command = "aws transfer describe-server --server-id ${var.transfer_server_id} --query 'Server.EndpointDetails.VpcEndpointId' > ${data.template_file.log_name.rendered}"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems extremely to use a provisioner to get vpc id enpdoint, but it is working well.
aws transfer module in aws and cloud posse doesn't have an output to vpc endpoint id.
Even terraform state has the value, but is a primitive string, so its forbidden to get.
Screenshot 2023-03-03 at 13 13 06

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bcarranza bcarranza changed the title record route53 Route53 Record + vpc endpoint extensive Mar 3, 2023
depends_on = [
data.aws_vpc_endpoint.this
]
}
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am reusing the first terraform called only record !

@bcarranza
Copy link
Collaborator Author

bcarranza commented Mar 3, 2023

thread: https://3pillarglobal.slack.com/archives/C02N9MVF0/p1677871126411499
The use of local-exec requires running in an environment with AWS CLI and depends on a text file where the data to extract is saved, so it's not very user-friendly. However, it was a desperate measure due to the lack of the VPC endpoint ID in AWS Transfer. We'll document this and discuss it next week with Garland Kan. @sekka1

"It's also good to highlight the positive aspect of this. The PR includes a generic Route53 record module that adds to the functionality of managed Kube. The only issue is with the record-sftp-transfer-server-vpc-endpoint

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant