Terraform module to create an AWS Elasticsearch cluster with support for AWS Cognito for authentication
module "eks" {
instance_count = 4
instance_type = "t2.medium.elasticsearch"
dedicated_master_type = "t2.medium.elasticsearch"
encrypt_at_rest = false
es_zone_awareness = true
es_version = "6.7"
enable_cognito = true #if you want authentication, see below
ebs_volume_size = 35
subnet_ids = ["subnet-xxx"]
advanced_options = {
"rest.action.multi.allow_explicit_index" = "true"
}
}
Authentication is provided in 2 parts, you have kibana authentication, and elasticsearch authentication.
This is pretty straight forward, at the moment the access policy is as declared in the module
data "aws_iam_policy_document" "es_vpc_management_access_base" {
statement {
actions = [
"es:ESHttpGet",
]
resources = [
aws_elasticsearch_domain.es_vpc.arn,
"${aws_elasticsearch_domain.es_vpc.arn}/*",
]
principals {
type = "AWS"
identifiers = var.enable_cognito ? [aws_iam_role.authenticated[0].arn] : distinct(compact(var.management_iam_roles))
}
}
statement {
actions = [
"es:ESHttpGet",
"es:ESHttpDelete",
"es:ESHttpHead",
"es:ESHttpPost",
"es:ESHttpPut"
]
resources = [
aws_elasticsearch_domain.es_vpc.arn,
"${aws_elasticsearch_domain.es_vpc.arn}/*",
]
principals {
type = "AWS"
identifiers = var.enable_cognito ? [aws_iam_role.admin_authenticated[0].arn] : distinct(compact(var.management_iam_roles))
}
}
}
data "aws_iam_policy_document" "es_vpc_management_access_base_overlay" {
source_json = data.aws_iam_policy_document.es_vpc_management_access_base.json
statement {
actions = [
"es:ESHttpGet",
"es:ESHttpPost",
"es:ESHttpPut"
]
resources = [
aws_elasticsearch_domain.es_vpc.arn,
"${aws_elasticsearch_domain.es_vpc.arn}/*",
]
principals {
type = "AWS"
identifiers = distinct(compact(flatten([var.log_pusher_iam_roles, aws_iam_role.log_pusher[0].arn])))
}
}
}es_vpc_management_access_base_overlay is used to allow an additional arn to push logs to the elasticsearch server.
This provides authenticated users with cognito to use elasticsearch APIs with the respective HTTP methods in actions list.
In order to allow logs to be pushed into es, you will need to provide a list of iam roles using log_pusher_iam_roles which are iam users or roles to allow the POST and PUT methods to insert log entries
If cognito is enabled, you will need to run the post-apply.sh after terraforming.
Key to note, post-apply.sh is used with terragrunt, if you are using terraform for your infrastructure, modify the script as such
This is a limitation currently with terraform + es + cognito. See github.com/terraform-providers/terraform-provider-aws/issues/5557
In general we will have to manage the cognito's user client pools outside of terraform as elasticsearch will create one on its own, if we were to make any changes using terraform it will cause the resources to be re-created everytime.
In order for users to access kibana, AWS Cognito is the authentication provider. By default this module creates two groups(admin, developer) to group our users for access control on available HTTP Methods
| Group | Allowed Methods |
|---|---|
| Admin | "es:ESHttp*" |
| Developer | "es:ESHttpGet","es:ESHttpPost" |
Users are allowed to sign up themselves, however they will be denied from accessing kibana until they are added to atleast the Developer group for GET and POST permissions
This will have to be done by accessing the AWS cognito console by the administrator and for him/her to manage the users in each group
| Name | Version |
|---|---|
| terraform | >= 0.13 |
| Name | Version |
|---|---|
| aws | n/a |
No modules.
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| additional_cidr_allow_443 | CIDR to allow port 443 communication from | list(any) |
[] |
no |
| admin_create_user_only | Only admin can create user (disables the sign up button) | bool |
true |
no |
| advanced_options | Map of key-value string pairs to specify advanced configuration options. Note that the values for these configuration options must be strings (wrapped in quotes) or they may be wrong and cause a perpetual diff, causing Terraform to want to recreate your Elasticsearch domain on every apply. | map(string) |
{} |
no |
| create_access_keys | Boolean to enable creation of iam access keys for the iam users app and backup | bool |
true |
no |
| create_iam_service_linked_role | Whether to create IAM service linked role for AWS ElasticSearch service. Can be only one per AWS account. | bool |
true |
no |
| create_log_pusher_role | create a IAM role that has persmission to push logs using the _bulk API to this elasticsearch | bool |
false |
no |
| dedicated_master_threshold | The number of instances above which dedicated master nodes will be used. Default: 10 | number |
10 |
no |
| dedicated_master_type | ES instance type to be used for dedicated masters (default same as instance_type) | string |
"false" |
no |
| domain_name | Domain name for Elasticsearch cluster | string |
"es-domain" |
no |
| domain_prefix | String to be prefixed to search domain. Default: tf- | string |
"tf-" |
no |
| ebs_volume_size | Optionally use EBS volumes for data storage by specifying volume size in GB (default 0) | number |
0 |
no |
| ebs_volume_type | Storage type of EBS volumes, if used (default gp2) | string |
"gp2" |
no |
| enable_cognito | Whether to enable AWS Cognito to handle user access controls to kibana and elasticsearch | bool |
false |
no |
| encrypt_at_rest | Enable encrption at rest (only specific instance family types support it: m4, c4, r4, i2, i3 default: false) | bool |
false |
no |
| environment | n/a | string |
"ci" |
no |
| es_version | Version of Elasticsearch to deploy (default 5.1) | string |
"5.1" |
no |
| es_zone_awareness | Enable zone awareness for Elasticsearch cluster (default false) | bool |
false |
no |
| force_destroy | When destroying this user, destroy even if it has non-Terraform-managed IAM access keys, login profile or MFA devices. Without force_destroy a user with non-Terraform-managed access keys and login profile will fail to be destroyed. | bool |
false |
no |
| instance_count | Number of data nodes in the cluster (default 6) | number |
6 |
no |
| instance_type | ES instance type for data nodes in the cluster (default t2.small.elasticsearch) | string |
"t2.small.elasticsearch" |
no |
| kms_key_id | KMS key used for elasticsearch | string |
"" |
no |
| log_publishing_options | List of maps of options for publishing slow logs to CloudWatch Logs. | list(map(string)) |
[] |
no |
| log_pusher_additional_policy | Additional policy ARN for log pusher role | string |
"" |
no |
| log_pusher_iam_roles | List of IAM users or role ARNs from which to permit PUT/POST requests meant to send logs to elasticsearch | list(string) |
[] |
no |
| management_iam_roles | List of IAM role ARNs from which to permit management traffic (default ['*']). Note that a client must match both the IP address and the IAM role patterns in order to be permitted access. | list(string) |
[ |
no |
| management_public_ip_addresses | List of IP addresses from which to permit management traffic (default []). Note that a client must match both the IP address and the IAM role patterns in order to be permitted access. | list(string) |
[] |
no |
| node_to_node_encryption_enabled | Whether to enable node-to-node encryption. | bool |
false |
no |
| num_availability_zones | Number of availability zones in which to deploy elasticsearch nodes | number |
2 |
no |
| path | Desired path for the IAM user | string |
"/" |
no |
| permissions_boundary | If provided, all IAM roles will be created with this permissions boundary attached. | string |
"" |
no |
| pgp_key | Either a base-64 encoded PGP public key, or a keybase username in the form keybase:username. Used to encrypt password and access key. | string |
"" |
no |
| project_name | n/a | string |
"es" |
no |
| security_group_ids | List of security groups to apply to the elasticsearch cluster | list(any) |
[] |
no |
| snapshot_start_hour | Hour at which automated snapshots are taken, in UTC (default 0) | number |
0 |
no |
| subnet_ids | List of subnets which elasticsearch nodes will be hosted in | list(any) |
n/a | yes |
| tags | tags to apply to all resources | map(string) |
{} |
no |
| use_prefix | Flag indicating whether or not to use the domain_prefix. Default: true | bool |
true |
no |
| vpc_id | VPC ID to import vpc data for use with AWS Elasticsearch | string |
n/a | yes |
| vpc_name | Name of VPC | string |
"" |
no |
| warm_count | The number of warm nodes in the cluster. Valid values are between 2 and 150. warm_count can be only and must be set when warm_enabled is set to true | number |
2 |
no |
| warm_enabled | Indicates whether to enable warm storage | bool |
false |
no |
| warm_type | The instance type for the Elasticsearch cluster's warm nodes. Valid values are ultrawarm1.medium.elasticsearch, ultrawarm1.large.elasticsearch | string |
"ultrawarm1.medium.elasticsearch" |
no |
| worker_node_role | If you will like eks nodes to assume this role, input the worker node role ARN to allow it to assume the log pusher role | string |
"" |
no |
| Name | Description |
|---|---|
| IDENTITY_POOL_NAME | variable to be used in post-apply.sh |
| USER_POOL_ID | variable to be used in post-apply.sh |
| USER_POOL_NAME | variable to be used in post-apply.sh |
| app_iam_access_key_id | The access key ID for log pusher |
| app_iam_access_key_secret | The access key secret for log pusher |
| app_iam_user_arn | The ARN assigned by AWS for log pusher user |
| app_iam_user_name | ES log pusher user's name |
| arn | Amazon Resource Name (ARN) of the domain |
| domain_id | Unique identifier for the domain |
| domain_name | The name of the Elasticsearch domain |
| endpoint | Domain-specific endpoint used to submit index, search, and data upload requests |
| kibana_endpoint | Domain-specific endpoint for kibana without https scheme |
| log_pusher_arn | ARN of iam role that is allowed to send logs to elasticsearch |