Skip to content

GovTechSG/terraform-aws-elasticsearch-cognito

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

45 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

AWS Elasticsearch

Terraform module to create an AWS Elasticsearch cluster with support for AWS Cognito for authentication

Usage

module "eks" {
  instance_count            = 4
  instance_type             = "t2.medium.elasticsearch"
  dedicated_master_type     = "t2.medium.elasticsearch"
  encrypt_at_rest           = false
  es_zone_awareness         = true
  es_version                = "6.7"
  enable_cognito            = true #if you want authentication, see below
  ebs_volume_size           = 35
  subnet_ids                = ["subnet-xxx"]
  advanced_options = {
    "rest.action.multi.allow_explicit_index" = "true"
  }
}

Authentication

Authentication is provided in 2 parts, you have kibana authentication, and elasticsearch authentication.

Elasticsearch

This is pretty straight forward, at the moment the access policy is as declared in the module

data "aws_iam_policy_document" "es_vpc_management_access_base" {
  statement {
    actions = [
      "es:ESHttpGet",
    ]

    resources = [
      aws_elasticsearch_domain.es_vpc.arn,
      "${aws_elasticsearch_domain.es_vpc.arn}/*",
    ]

    principals {
      type        = "AWS"
      identifiers = var.enable_cognito ? [aws_iam_role.authenticated[0].arn] : distinct(compact(var.management_iam_roles))
    }
  }

  statement {
    actions = [
      "es:ESHttpGet",
      "es:ESHttpDelete",
      "es:ESHttpHead",
      "es:ESHttpPost",
      "es:ESHttpPut"
    ]

    resources = [
      aws_elasticsearch_domain.es_vpc.arn,
      "${aws_elasticsearch_domain.es_vpc.arn}/*",
    ]

    principals {
      type        = "AWS"
      identifiers = var.enable_cognito ? [aws_iam_role.admin_authenticated[0].arn] : distinct(compact(var.management_iam_roles))
    }
  }
}

data "aws_iam_policy_document" "es_vpc_management_access_base_overlay" {
  source_json = data.aws_iam_policy_document.es_vpc_management_access_base.json
  statement {
    actions = [
      "es:ESHttpGet",
      "es:ESHttpPost",
      "es:ESHttpPut"
    ]

    resources = [
      aws_elasticsearch_domain.es_vpc.arn,
      "${aws_elasticsearch_domain.es_vpc.arn}/*",
    ]

    principals {
      type        = "AWS"
      identifiers = distinct(compact(flatten([var.log_pusher_iam_roles, aws_iam_role.log_pusher[0].arn])))
    }
  }
}

es_vpc_management_access_base_overlay is used to allow an additional arn to push logs to the elasticsearch server.

This provides authenticated users with cognito to use elasticsearch APIs with the respective HTTP methods in actions list.

In order to allow logs to be pushed into es, you will need to provide a list of iam roles using log_pusher_iam_roles which are iam users or roles to allow the POST and PUT methods to insert log entries

enable_cognito

If cognito is enabled, you will need to run the post-apply.sh after terraforming.

Key to note, post-apply.sh is used with terragrunt, if you are using terraform for your infrastructure, modify the script as such

This is a limitation currently with terraform + es + cognito. See github.com/terraform-providers/terraform-provider-aws/issues/5557

In general we will have to manage the cognito's user client pools outside of terraform as elasticsearch will create one on its own, if we were to make any changes using terraform it will cause the resources to be re-created everytime.

Cognito

In order for users to access kibana, AWS Cognito is the authentication provider. By default this module creates two groups(admin, developer) to group our users for access control on available HTTP Methods

Group Allowed Methods
Admin "es:ESHttp*"
Developer "es:ESHttpGet","es:ESHttpPost"

Users are allowed to sign up themselves, however they will be denied from accessing kibana until they are added to atleast the Developer group for GET and POST permissions

This will have to be done by accessing the AWS cognito console by the administrator and for him/her to manage the users in each group

Requirements

Name Version
terraform >= 0.13

Providers

Name Version
aws n/a

Modules

No modules.

Resources

Name Type
aws_cognito_identity_pool.kibana resource
aws_cognito_identity_pool_roles_attachment.main resource
aws_cognito_user_group.admin resource
aws_cognito_user_group.developer resource
aws_cognito_user_pool.kibana resource
aws_cognito_user_pool_domain.kibana resource
aws_elasticsearch_domain.es_vpc resource
aws_elasticsearch_domain_policy.es_vpc_management_access resource
aws_iam_access_key.log-pusher resource
aws_iam_group.log-pusher resource
aws_iam_group_membership.log-pusher resource
aws_iam_group_policy.log-pusher-group-policy resource
aws_iam_policy.authenticated resource
aws_iam_role.admin_authenticated resource
aws_iam_role.authenticated resource
aws_iam_role.developer_authenticated resource
aws_iam_role.es_assume_role resource
aws_iam_role.log_pusher resource
aws_iam_role.unauthenticated resource
aws_iam_role_policy.admin_authenticated resource
aws_iam_role_policy.developer_authenticated resource
aws_iam_role_policy.unauthenticated resource
aws_iam_role_policy_attachment.admin_attachment resource
aws_iam_role_policy_attachment.authenticated_attachment resource
aws_iam_role_policy_attachment.developer_attachment resource
aws_iam_role_policy_attachment.es_cognito_access resource
aws_iam_role_policy_attachment.log_pusher_cloudwatch_attach resource
aws_iam_service_linked_role.es resource
aws_iam_user.log-pusher resource
aws_kms_key.kms resource
aws_security_group.allow_443 resource
aws_iam_policy.amazon_es_cognito_access data source
aws_iam_policy_document.ec2_base_assume_role data source
aws_iam_policy_document.ec2_overlay_assume_role data source
aws_iam_policy_document.es_vpc_management_access_base data source
aws_iam_policy_document.es_vpc_management_access_base_overlay data source
aws_vpc.vpc data source

Inputs

Name Description Type Default Required
additional_cidr_allow_443 CIDR to allow port 443 communication from list(any) [] no
admin_create_user_only Only admin can create user (disables the sign up button) bool true no
advanced_options Map of key-value string pairs to specify advanced configuration options. Note that the values for these configuration options must be strings (wrapped in quotes) or they may be wrong and cause a perpetual diff, causing Terraform to want to recreate your Elasticsearch domain on every apply. map(string) {} no
create_access_keys Boolean to enable creation of iam access keys for the iam users app and backup bool true no
create_iam_service_linked_role Whether to create IAM service linked role for AWS ElasticSearch service. Can be only one per AWS account. bool true no
create_log_pusher_role create a IAM role that has persmission to push logs using the _bulk API to this elasticsearch bool false no
dedicated_master_threshold The number of instances above which dedicated master nodes will be used. Default: 10 number 10 no
dedicated_master_type ES instance type to be used for dedicated masters (default same as instance_type) string "false" no
domain_name Domain name for Elasticsearch cluster string "es-domain" no
domain_prefix String to be prefixed to search domain. Default: tf- string "tf-" no
ebs_volume_size Optionally use EBS volumes for data storage by specifying volume size in GB (default 0) number 0 no
ebs_volume_type Storage type of EBS volumes, if used (default gp2) string "gp2" no
enable_cognito Whether to enable AWS Cognito to handle user access controls to kibana and elasticsearch bool false no
encrypt_at_rest Enable encrption at rest (only specific instance family types support it: m4, c4, r4, i2, i3 default: false) bool false no
environment n/a string "ci" no
es_version Version of Elasticsearch to deploy (default 5.1) string "5.1" no
es_zone_awareness Enable zone awareness for Elasticsearch cluster (default false) bool false no
force_destroy When destroying this user, destroy even if it has non-Terraform-managed IAM access keys, login profile or MFA devices. Without force_destroy a user with non-Terraform-managed access keys and login profile will fail to be destroyed. bool false no
instance_count Number of data nodes in the cluster (default 6) number 6 no
instance_type ES instance type for data nodes in the cluster (default t2.small.elasticsearch) string "t2.small.elasticsearch" no
kms_key_id KMS key used for elasticsearch string "" no
log_publishing_options List of maps of options for publishing slow logs to CloudWatch Logs. list(map(string)) [] no
log_pusher_additional_policy Additional policy ARN for log pusher role string "" no
log_pusher_iam_roles List of IAM users or role ARNs from which to permit PUT/POST requests meant to send logs to elasticsearch list(string) [] no
management_iam_roles List of IAM role ARNs from which to permit management traffic (default ['*']). Note that a client must match both the IP address and the IAM role patterns in order to be permitted access. list(string)
[
"*"
]
no
management_public_ip_addresses List of IP addresses from which to permit management traffic (default []). Note that a client must match both the IP address and the IAM role patterns in order to be permitted access. list(string) [] no
node_to_node_encryption_enabled Whether to enable node-to-node encryption. bool false no
num_availability_zones Number of availability zones in which to deploy elasticsearch nodes number 2 no
path Desired path for the IAM user string "/" no
permissions_boundary If provided, all IAM roles will be created with this permissions boundary attached. string "" no
pgp_key Either a base-64 encoded PGP public key, or a keybase username in the form keybase:username. Used to encrypt password and access key. string "" no
project_name n/a string "es" no
security_group_ids List of security groups to apply to the elasticsearch cluster list(any) [] no
snapshot_start_hour Hour at which automated snapshots are taken, in UTC (default 0) number 0 no
subnet_ids List of subnets which elasticsearch nodes will be hosted in list(any) n/a yes
tags tags to apply to all resources map(string) {} no
use_prefix Flag indicating whether or not to use the domain_prefix. Default: true bool true no
vpc_id VPC ID to import vpc data for use with AWS Elasticsearch string n/a yes
vpc_name Name of VPC string "" no
warm_count The number of warm nodes in the cluster. Valid values are between 2 and 150. warm_count can be only and must be set when warm_enabled is set to true number 2 no
warm_enabled Indicates whether to enable warm storage bool false no
warm_type The instance type for the Elasticsearch cluster's warm nodes. Valid values are ultrawarm1.medium.elasticsearch, ultrawarm1.large.elasticsearch string "ultrawarm1.medium.elasticsearch" no
worker_node_role If you will like eks nodes to assume this role, input the worker node role ARN to allow it to assume the log pusher role string "" no

Outputs

Name Description
IDENTITY_POOL_NAME variable to be used in post-apply.sh
USER_POOL_ID variable to be used in post-apply.sh
USER_POOL_NAME variable to be used in post-apply.sh
app_iam_access_key_id The access key ID for log pusher
app_iam_access_key_secret The access key secret for log pusher
app_iam_user_arn The ARN assigned by AWS for log pusher user
app_iam_user_name ES log pusher user's name
arn Amazon Resource Name (ARN) of the domain
domain_id Unique identifier for the domain
domain_name The name of the Elasticsearch domain
endpoint Domain-specific endpoint used to submit index, search, and data upload requests
kibana_endpoint Domain-specific endpoint for kibana without https scheme
log_pusher_arn ARN of iam role that is allowed to send logs to elasticsearch