Lightlytics terraform module for AWS

A terraform module connecting AWS account to Lightlytics built by Kaltura.

Module Input Variables

• environment = variable environment
• account_id = variable name
• aws_region = variable aws_region
• vpc_id = variable vpc_id
• endpoint_subnet_ids = variable subnets ids for VPC Endpoint
• lightlytics_api_url = "https://< ORGANIZATION NAME >-${var.aws_region}-pvl.lightlytics.com"
• lightlytics_external_api_url = "https://< ORGANIZATION NAME >.lightlytics.com"
• lightlytics_endpoint_service_name = GET FROM LIGHTLYTICS AFTER THEY CREATE IT PER REGION\ACCOUNT
• lambda_init_s3_source_code_bucket = "<Lightlytics_S3_Bucket>-${var.aws_region}"
• lambda_init_s3_source_code_key = S3_Key
• lambda_layer_source_code_bucket = "<Lightlytics_S3_Bucket>-${var.aws_region}"
• lambda_layer_source_code_key = S3_Key
• lambda_flow_logs_s3_source_code_bucket = "<Lightlytics_S3_Bucket>-${var.aws_region}"
• lambda_flow_logs_s3_source_code_key = S3_Key
• lambda_cloud_watch_s3_source_code_bucket = "<Lightlytics_S3_Bucket>-${var.aws_region}"
• lambda_cloud_watch_s3_source_code_key = S3_Key
• lightlytics_account = variable lightlytics_account
• LightlyticsInternalAccountId = variable LightlyticsInternalAccountId
• lightlytics_account_externalID = variable lightlytics_account_externalID
• lightlytics_auth_token = variable lightlytics_auth_token
• collection_token = variable collection_token

Usage

module "lightlytics" {
  source = "github.com/kaltura/ovp-tf-module-lightlytics/<VERSION>"
  environment                                  =
  account_id                                   =
  aws_region                                   =   
  vpc_id                                       = 
  endpoint_subnet_ids                          = 
  lightlytics_api_url                          = 
  lightlytics_external_api_url                 = 
  lightlytics_endpoint_service_name            =  
  lambda_init_s3_source_code_bucket            = "<LightLytics_S3_Bucket_Name>-${var.aws_region}"
  lambda_init_s3_source_code_key               = "<Lambda_S3_Key_Name>"
  lambda_layer_source_code_bucket              = "<LightLytics_S3_Bucket_Name>-${var.aws_region}"
  lambda_layer_source_code_key                 = "<Lambda_S3_Key_Name>"
  lambda_flow_logs_s3_source_code_bucket       = "<LightLytics_S3_Bucket_Name>-${var.aws_region}"
  lambda_flow_logs_s3_source_code_key          = "<Lambda_S3_Key_Name>"
  lambda_cloud_watch_s3_source_code_bucket     = "<LightLytics_S3_Bucket_Name>-${var.aws_region}"
  lambda_cloud_watch_s3_source_code_key        = "<Lambda_S3_Key_Name>"
  lightlytics_account                          =
  LightlyticsInternalAccountId                 =
  lightlytics_account_externalID               =   
  lightlytics_auth_token                       = 
  collection_token                             = 
  tags {
    "Environment" = "${var.environment}"
  }
}

Adding AWS account

• MUST BE SIGNED IN BROWSER TO THE ACCOUNT YOU ARE ABOUT TO ADD

• Under the relevant Workspace --> Settings "mechanical wheel" --> "Integrations" --> Click the plus sign "+" to add and account --> Input the Account ID + display name and click "Add Account" --> click the "Launch Stack" which will open it in the AWS Account and navigate to the URL which points to the CloudFormation.yaml --> click "Continue" --> click "Close and Cancel"

• North Virginia has to be added as a default region in Lightlytics

• Get the values and update them in the Secret Manager - "lightlytics-secrets":

  • LightlyticsInternalAccountId
  • AccountAuthToken
  • LightlyticsCollectionToken
  • ExternalId

• The following vars are taken from the main tf.state:

  • variable "environment" {}
  • variable "account_id" {}
  • variable "aws_region" {}
  • variable "vpc_id" {}
  • variable "endpoint_subnet_ids" {}

• VAR - might change\need update:

  • lightlytics_account
  • Lambda source code and key:
    • s3_bucket = "prod-lightlytics-artifacts-us-east-1"
    • s3_key - depending on the lambda
  • lightlytics_api_url
  • lightlytics_external_api_url
  • lightlytics_endpoint_service_name

• Lambda

  • Init - Scans initially (and nightly) the entire AWS account and sends data to Lightlytics
  • CloudWatch - Creates a CloudWatch rule to monitor events and sends data to Lightlytics In real time
  • FLowLogs - Monitors S3 bucket and sends the flow logs to Lightlytics
    • collect_flow_logs_enabled - true\false - select your requirements

• Curl command that enables the Account

    curl -X POST '${var.lightlytics_external_api_url}/graphql' \
      -H 'Content-Type: application/json' \
      -H 'Authorization: Bearer ${var.lightlytics_auth_token}' \
      -d '{"query":"mutation AccountAcknowledge($input: AccountAckInput){\r\n accountAcknowledge(account: $input)\r\n }","variables": {"input": {"lightlytics_internal_account_id":"${var.LightlyticsInternalAccountId}","role_arn":"arn:aws:iam::${var.account_id}:role/${var.environment}-lightlytics-role","account_type":"AWS","account_aliases":"","aws_account_id":"${var.account_id}","stack_region":"${var.aws_region}","stack_id":"","init_stack_version":1}}}'

Feature notes

v1.0

• Lambda
• IAM Policy & Roles
• Var
• VPC Flow Logs & S3 Bucket
• Cloud Watch Rule
• curl

v1.1

• New Cloud Watch Rules
• Flow Logs S3 Bucket LifeCycle Rule

v1.2

• Extracted S3 Lambda source + Key
• Lambda Batch updates from 1000 to 4000

v1.3

• Changed Lambda to send the FlowLogs + CloudWatch logs via AWS VPC Endpoint

How you can help (guidelines for contributors)

Thank you for helping Kaltura grow! If you'd like to contribute please follow these steps:

• Use the repository issues tracker to report bugs or feature requests
• Read Contributing Code to the Kaltura Platform
• Sign the Kaltura Contributor License Agreement

Where to get help

• Join the Kaltura Community Forums to ask questions or start discussions
• Read the Code of conduct and be patient and respectful

Get in touch

You can learn more about Kaltura and start a free trial at: http://corp.kaltura.com
Contact us via Twitter @Kaltura or email: community@kaltura.com
We'd love to hear from you!

License and Copyright Information

All code in this project is released under the AGPLv3 license unless a different license for a particular library is specified in the applicable library path.

Authors

Copyright © Kaltura Inc. All rights reserved.

@denlitvakka Authors and contributors: See GitHub contributors list.

Providers

Name	Version
aws	3.70.0
null	3.1.0
time	n/a

Resources

Name	Type
aws_cloudwatch_event_rule.lightlytics-CloudWatch-rule	resource
aws_cloudwatch_event_target.lightlytics-lambda-cloud-watch-target	resource
aws_flow_log.lightlytics-flow-logs	resource
aws_iam_policy.lightlytics-CloudWatch-policy	resource
aws_iam_policy.lightlytics-FlowLogs-lambda-policy	resource
aws_iam_policy.lightlytics-init-policy	resource
aws_iam_policy.lightlytics-policy	resource
aws_iam_role.lightlytics-CloudWatch-role	resource
aws_iam_role.lightlytics-FlowLogs-lambda-role	resource
aws_iam_role.lightlytics-init-role	resource
aws_iam_role.lightlytics-role	resource
aws_iam_role_policy_attachment.lightlytics-role-attach-cloud-watch	resource
aws_iam_role_policy_attachment.lightlytics-role-attach-flow-logs	resource
aws_iam_role_policy_attachment.lightlytics-role-attach-global	resource
aws_iam_role_policy_attachment.lightlytics-role-attach-init	resource
aws_lambda_function.lightlytics-CloudWatch-lambda	resource
aws_lambda_function.lightlytics-FlowLogs-lambda	resource
aws_lambda_function.lightlytics-init-lambda	resource
aws_lambda_function_event_invoke_config.lightlytics-options-cloud-watch	resource
aws_lambda_function_event_invoke_config.lightlytics-options-flow-logs	resource
aws_lambda_function_event_invoke_config.lightlytics-options-init	resource
aws_lambda_layer_version.lightlytics-lambda-layer	resource
aws_lambda_permission.lightlytics-cloud-watch-allow-lambda	resource
aws_lambda_permission.lightlytics-flow-logs-allow-lambda-s3	resource
aws_s3_bucket.lightlytics-flow-logs-bucket	resource
aws_s3_bucket_notification.lightlytics-lambda-s3-trigger	resource
aws_security_group.allow_443_outbound	resource
aws_vpc_endpoint.lambda_send_url	resource
null_resource.lightlytics-enable-account	resource
time_sleep.wait_15_seconds	resource

Inputs

Name	Description	Type	Default	Required
LightlyticsInternalAccountId	n/a	any	n/a	yes
account_id	n/a	any	n/a	yes
aws_region	n/a	any	n/a	yes
collect_flow_logs_enabled	n/a	bool	true	no
collection_token	n/a	any	n/a	yes
domain_name	n/a	string	"lightlytics.com"	no
endpoint_subnet_ids	n/a	any	n/a	yes
environment	n/a	any	n/a	yes
flow_logs_bucket_lifecycle_rule	n/a	list(object({ id = string prefix = string enabled = bool days = number }))	[ { "days": 1, "enabled": true, "id": "purge", "prefix": "AWSLogs/" } ]	no
lambda_cloud_watch_s3_source_code_bucket	n/a	any	n/a	yes
lambda_cloud_watch_s3_source_code_key	n/a	any	n/a	yes
lambda_flow_logs_batch_size	n/a	number	4000	no
lambda_flow_logs_cloud_watch_env	n/a	string	"production"	no
lambda_flow_logs_cloud_watch_max_event_age	n/a	number	21600	no
lambda_flow_logs_cloud_watch_max_retry	n/a	number	2	no
lambda_flow_logs_cloud_watch_memory_size	n/a	number	128	no
lambda_flow_logs_cloud_watch_node_env	n/a	string	"production"	no
lambda_flow_logs_cloud_watch_timeout	n/a	number	120	no
lambda_flow_logs_env	n/a	string	"prod"	no
lambda_flow_logs_max_event_age	n/a	number	21600	no
lambda_flow_logs_max_retry	n/a	number	2	no
lambda_flow_logs_memory_size	n/a	number	128	no
lambda_flow_logs_node_env	n/a	string	"prod"	no
lambda_flow_logs_s3_source_code_bucket	n/a	any	n/a	yes
lambda_flow_logs_s3_source_code_key	n/a	any	n/a	yes
lambda_flow_logs_timeout	n/a	number	120	no
lambda_init_env	n/a	string	"prod"	no
lambda_init_max_event_age	n/a	number	21600	no
lambda_init_max_retry	n/a	number	2	no
lambda_init_memory_size	n/a	number	128	no
lambda_init_node_env	n/a	string	"prod"	no
lambda_init_s3_source_code_bucket	n/a	any	n/a	yes
lambda_init_s3_source_code_key	n/a	any	n/a	yes
lambda_init_timeout	n/a	number	900	no
lambda_layer_source_code_bucket	n/a	any	n/a	yes
lambda_layer_source_code_key	n/a	any	n/a	yes
lightlytics_account	n/a	any	n/a	yes
lightlytics_account_externalID	n/a	any	n/a	yes
lightlytics_api_url	n/a	any	n/a	yes
lightlytics_auth_token	n/a	any	n/a	yes
lightlytics_endpoint_service_name	n/a	any	n/a	yes
vpc_id	n/a	any	n/a	yes