Skip to content

Latest commit

 

History

History
101 lines (69 loc) · 5.37 KB

serverless.md

File metadata and controls

101 lines (69 loc) · 5.37 KB
Raw

Deployment to OpenSearch Serverless (Experimental)

View this page in Japanese (日本語) | Back to README

This page explains how to deploy SIEM on OpenSearch to Amazon OpenSearch Serverless and notes.

Introduction

Please note that there are differences from Managed Cluster because the service features and internal versions are different when deploying SIEM on OpenSearch to Amazon OpenSearch Serverless.

Difference OpenSearch managed cluster OpenSearch Serverless
Index and Shard management Managed by the user Managed by the service and no user management required. Automatic scaling
Maximum number of indices and shards 1000 shards per instance [Time series Collection]
Up to 120 indices
[Search Collection]
Up to 20 indices
* Please refer to the following quota page
Security Analytics Available since OpenSearch 2.5 Not implemented
Index name and rotation The index name is given the selected date and automatically rotated The index name is fixed and manually numbered (e.g. log-aws-xxxx-001)
Deduplication Duplicate logs are excluded and not loaded into OpenSearch [Time series collection]
Not deduplicated. It is deduplicated only when processed by the same es-loader Lambda instance
[Search collection]
Deduplicated
Sorting, aggregations It can be changed by configuration. The default configuration for SIEM is 200 doc_values is up to 100 fields. Please be careful when importing logs with many fields
Delete Logs Delete by index,
Delete by search expression (_delete_by_query)		 Delete by index
Supported OpenSearch API operations Almost all APIs At least reindex and snapshot are not supported

Please refer to the official documentation for the differences as a service.

Preparation

Access from within VPC

If you want to load logs into OpenSearch Serverless from within VPC, create an Amazon OpenSearch Serverless (AOSS) VPC Endpoint in your VPC. If you have public access, skip this step.

  1. Create a VPC
  2. Create an AOSS Endpoint
  3. Create a Security Group that allows inbound tcp/443 and associate it with the AOSS Endpoint.

OpenSearch Serverless Collection

AWS CDK or AWS CloudFormation templates create a new OpenSearch Serverless collection with the following parameters:

  1. Collection name: The name specified in the DomainOrCollectionName parameter
  2. Collection type: Time series
  3. Network access type: Public
  4. Encryption: AWS owned key

If you need a collection with different parameters, please create it yourself in advance.

Please choose one of the following encryption keys:

Only [AWS owned key] is supported for encryption in the SIEM solution.

Deploying SIEM

  1. Run the AWS CDK or CloudFormation template.
  2. Parameters
    • Select [opensearch_serverless] for DeploymentTarget
    • Enter [any collection name] for DomainOrCollectionName. If you want to use an existing collection, enter [existing collection name].
    • If you want to access from within VPC, enter [AOSS Endpoint ID] for VpcEndpointId
    • Other parameters are common to Managed Cluster installation.
  3. Configure data access policy for OpenSearch Dashboards.

Configuring data access policy

CDK/CloudFormation only sets the policy required to load logs. Please manually set the policy to login to OpenSearch Dashboards.

Example:

  1. Select [Data access policies] from the left menu
  2. Select [Create access policy]
    1. Enter [any policy name] in Access policy name. Example: dashboards-access
    2. Enter the IAM that allows access in [Add principals]
    3. Select [Grant] in Resources and Permissions
      1. Select [Select all] in "Alias and templates permissions"
      2. Enter the CloudFormation-specified [Collection Name] in Collections field
      3. Select [Select all] in "Index permissions"
      4. Enter the CloudFormation-specified [Collection Name] in Collection field
      5. Enter * for Index Name.

Index rotation

By default, 001 is added to the index name. No automatic rotation. e.g. log-aws-cloudtrail-001 Specify the suffix manually when rotating.

Setting Example

# user.ini
[cloudtrail]
index_suffix = 002

Index Name: log-aws-cloudtrail-002

Enabling single sign-on

For single sign-on with AWS IAM Identity Center, see AWS Control Tower Integration - SAML federation

Known Issue

  • When loading logs, internal errors such as "Internal error occurred while processing request" may occur. Automatic retry processing is performed, but if it fails continuously, the log will be moved to DLQ. Please execute Dead-letter queue redrive.