Terraform module for setting up infrastructure for Shunting Yard.
For more information please refer to the main Apiary project page.
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| allowed_s3_buckets | List of S3 Buckets to which Shunting Yard will have read-write access. | list | n/a | yes |
| aws_region | AWS region to use for resources. | string | n/a | yes |
| cpu | The number of CPU units to reserve for the Shunting Yard container. Valid values can be 256, 512, 1024, 2048 and 4096. Reference: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html | string | "1024" |
no |
| ct_common_config_yaml | Common Circus Train configuration to be passed to internal Circus Train instance. It can be used, for example to configure Graphite for Circus Train. Refer to Circus Train README for an exhaustive list of options supported by Circus Train. | string | n/a | yes |
| ct_log4j_xml | Log4j XML file to be passed to internal Circus Train instance to configure logging. | string | n/a | yes |
| docker_image | Full path of Shunting Yard Docker image. | string | n/a | yes |
| docker_registry_auth_secret_name | Docker Registry authentication SecretManager secret name. | string | "" |
no |
| docker_version | Shunting Yard Docker image version. | string | n/a | yes |
| instance_name | Shunting Yard instance name to identify resources in multi-instance deployments. | string | "" |
no |
| memory | The amount of memory (in MiB) used to allocate for the Shunting Yard container. Valid values: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html | string | "4096" |
no |
| metastore_events_sns_topic | SNS Topic for Hive Metastore events. | string | n/a | yes |
| orphaned_data_strategy | Orphaned data strategy to use for stale data during replication. Supported strategies: "NONE", "HOUSEKEEPING" (default). | string | "HOUSEKEEPING" |
no |
| selected_tables | Tables selected for Shunting Yard Replication. Supported Format: [ "database_1.table_1", "database_2.table_2" ] Wildcards are not supported, i.e. you need to specify each table explicitly. | list | <list> |
no |
| shuntingyard_sqs_queue_stale_messages_timeout | Shunting Yard SQS Queue Cloudwatch Alert timeout for messages older than this number of seconds. | string | "300" |
no |
| shuntingyard_sqs_queue_wait_timeout | Wait timeout for connecting to the Shunting Yard SQS queue (in seconds) | string | "15" |
no |
| shuntingyard_tags | A map of tags to apply to resources. | map | n/a | yes |
| source_metastore_uri | Source Metastore URI for Shunting Yard. | string | n/a | yes |
| subnets | ECS container subnets. | list | n/a | yes |
| target_metastore_uri | Target Metastore URI for Shunting Yard. | string | n/a | yes |
| vpc_id | VPC ID. | string | n/a | yes |
Example module invocation:
module "apiary-shuntingyard" {
source = "git::https://github.com/ExpediaGroup/apiary-replication.git"
aws_region = "us-west-2"
vpc_id = "vpc-1"
subnets = ["subnet-1", "subnet-2"]
instance_name = "shuntingyard-test"
docker_image = "your.docker.repo/apiary-shuntingyard"
docker_version = "latest"
ct_common_config_yaml = "${data.template_file.ct_common_config_yaml.rendered}"
source_metastore_uri = "thrift://ip-address:9083"
target_metastore_uri = "thrift://ip-address:9083"
metastore_events_sns_topic = "arn:aws:sns:us-west-2:1234567:metastore-events-sns-topic"
selected_tables = [ "database_1.table_1", "database_2.table_2" ]
allowed_s3_buckets = [ "bucket-1", "bucket-2" ]
shuntingyard_tags = {
Name = "Apiary Replication"
Team = "Operations"
}
}
Please note that the parameter ct_common_config_yaml takes in the contents of the ct-common-config.yml file and should be passed in the manner described above using templates.
If you would like to ask any questions about or discuss Apiary please join our mailing list at
https://groups.google.com/forum/#!forum/apiary-user
This project is available under the Apache 2.0 License.
Copyright 2019 Expedia Inc.