All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- Support for OCSF #293
- Support for Parquet file format #275
- Enabled SQS/SNS/EventBridge to invoke es-loader (Lambda function) and load logs from Non-SIEM-managed S3 Bucket into OpenSearch Service #232,#291
- Enhance support for China region and US gov-cloud #281
- Enhance GuardDuty for malware protection #278
- Enrich logs with user-agent #273
- Support CSV delimiter #271
- Support OpenSearch Service v1.3 #270
- Enrich logs with IoC (experimental) #269
- Expand log directory to include all logs from AWSLogs/UserLogs #291
- Updated CloudTrail dashboard #289
- Updated CloudWatch Dashboard to show AWS resources metrics $287
- Updated CloudFront dashboard $79
- Updated OpenSearch Indexing Metrics dashboard $272
- Default S3 bucket encryption #287
- Change Lambda platform CPU architecture from x86_64 to arm64 #164
- Fixed unnecessary loop for split logs #285
- Fixed failure to load escaped string in JSON file #284
- Fixed parse issue: (CloudTrail) responseElements.imageId, additionalEventData.requestParameters, tags.AmazonFSx/tags.AmazonFSx.FileSystemId #265,#282,#283
- Fixed parse issue: (AWS Config)
- Fixed failure to deploy in ap-northeast-3 by CDK #280
- Fixed convert_iso8601_to_datetime except logic #268
- Fixed issue with deploying with CDK #267
- Fixed issue with extracting 12 digits distribution_id of CloudFront #61
- Access Point ARN field to s3 access log #260
- CEF File format #28,#259
- Supported and normalized log: AWS Client VPN, AWS CloudHSM #197,#253,#257
- Dashboard: AWS Client VPN, AWS CloudHSM #197,#253,#257
- Support for multiple CIDR blocks in vpc #252
- Enhanced error handling of deplyment-aes when opensearch domain exists #262
- Fixed issue with updating cdk.json #261
- UnicodeEncodeError when non utf8 code includs in log #255
- Fixed regex issue: S3 accesslog #254
- Supported OpenSearch Service version: 1.1, 1.2 #210,#237
- Supported and normalized log: AWS Trusted Advisor, Amazon Inspector #8,#207,#236
- Ability to stop log ingestion in an emergency when disk is full #44,#234
- CloudWatch dashboards to monitor SIEM status #233
- Dashboard in OpenSearch to monitor metrics of indices and shards #227
- New parsing logic of timestamp #226
- ECS Normalization of Security Hub #239
- ECS Normalization of GuardDuty #238
- Initial version of Amazon OpenSearch Service to v1.2 from v1.0 #237
- Error handling in es-loader. Until v2.6.1, when an error occurred in es-loader, processing was stopped and all error logs were sent to SQS for reprocessing. From this version, parsing errors will not be reprocessed and will not be sent to SQS and DLQ either #231
- Reduce default number of shards: log-aws-inspector, log-aws-securityhub, log-aws-workspaces, log-aws-trustedadvisor, .opendistro-alerting-alerts, .opendistro-alerting-config, .opendistro-ism-config, .opendistro-job-scheduler-lock, #211,#228
- Increase the refresh interval #211
- Legacy index templates is removed. Please use new index templates if you configure index settings. See the doc #152
- KMS VPC endpoint. If you use client-side encryption, add it manually #235
- Fixed parse issue: (CloudTrail) responseElements.session, responseElements.database, requestParameters.imageId, requestParameters.description, responseElements.data, requestParameters.metrics, requestParameters.CreateLaunchTemplateRequest.LaunchTemplateData.TagSpecification.Tag.Value, requestParameters.parameters, requestParameters.parameters, esponseElements.status,requestParameters.settings, requestParameters.ReplicationConfiguration.Rule.Filter, responseElements.networkInterface.tagSet.items.value, requestParameters.tagSpecificationSet.items.tags.value, responseElements.tableMetadata.parameters.projection.part_date.interval, requestParameters.accountIds #163,#187,#188,#189,#190,#191,#192,#193,#240,#242,#243,#245,#246,#247,#249
- Invalid datetime format of workspaces inventory #229
- Fixed parse error of CloudTrail Insights, 0 bytes ALB access log #219,#241
- Added connection status to Workspaces log #201
- Added 2 dashboards (S3 access log / VPC Flow Logs with v5 custom field) #198,#199
- Readme, docs, deployment script of Chinese Version #186
- CloudFormation template to exporter SecurityHub and Config to S3 bucket #184
- Another logging options for AWS WAF #185
- Enhanced parsing logic, updated Dashboard for GuardDuty and Security Hub #179,#182,#183,#203
- Migrated new index template form legacy template of GuardDuty/Security Hub #203
- Enhanced workspace dashboard #202
- Reduced query rate of AWS API for DescribeWorkspaces #200
- Legacy template is deprecated and will be obsolete in v2.7.0. Please update to index templates and component templates if you use legacy template. If you load only AWS resource log and don't configure OpenSearch settings such as field type, you can ignore. For more details, see Index templates #152
- Defined Inspector v2 field in Security Hub #203
- Contribution: Okta log support. Thanks to @yopiyama #168
- Supported and normalized log: Config, Config Rules, OpenSearch audit log, ElastiCache Redis SlOWLOG #3,#119,#144
- Enable to index log to a new index when the index is read-only #161
- Enable to rename original log field name before ETL process #172
- Enable to add field prefix to original log #173
- Enable to add list field type to related.* field in aws.ini/user.ini #176
- Change client library from Python Elasticsearch Client to OpenSearch-py #171
- Refactored parse logic of CloudWatch Logs/FireLens #171
- Enable to partially load parse error logs when logs are forwarded by FireLens, which container logs have stdout and stderr #171
- Split file format ETL logic to modules #171
- Strictly follow PEP8 and ShellCheck #171
- Updated IAM Policy/Role name and log group name for OpenSearch Service #171
- Legacy template is deprecated and will be obsolete in v2.7.0. Please update to index templates and component templates if you use legacy template. If you load only AWS resource log and don't configure OpenSearch settings such as field type, you can ignore. For more details, see Index templates #152
- Fixed issue of parsing nano seconds #160
- Fixed regex pattern of ALB #162
- Fixed parse error of requestParameters.configuration of CloudTrail #166
- Fixed parse issue: requestParameters.command, requestParameters.schedule,requestParameters.scope,responseElements.status #167,#171,#177
- Fixed issue of deploy script and auto_setup_on_cloudshell.sh #177
- Extracted WAF header #128
- Error Log number when es-loader failed to load log into OpenSearch Service #158
- Enhanced related.* and /var/log/secure of Linux log #159
- Renamed Solution name from "SIEM on Amazon Elasticsearch Service" to "SIEM on Amazon OpenSearch Service" #157
- Fixed parse issue: responseElements.tableMetadataList.parameters.projection.date.interval, requestParameters.configuration. #153,#156
- Added Amazon WorkSpaces dashboard #150
- Updated AWS Security Hub dashboard #2
- Refactored ETL logic of FireLens #149
- partially migrated from legacy index templates to new composable ones. rollover policy, workspaces and windows events index #152
- Fixed logic of truncating field value which is bigger than 32,766 byte #138
- Support Amazon ES 7.10 #103
- Supported and normalized log: Windows on EC2(XML Event Log), Amazon FSx for Windows File Server(Audit log), AWS Directory Service, Amazon WorkSpaces #118,#120,#140,#145
- Support Elastic Common Schema v1.10.0. related.* are useful. #146
- Added CloudTrail Insight to supported service list in document #130
- Auto install script on CloudShell #65
- Automatically parsing micro epoch time #137
- Truncated log value more than 32,766 byte #138,
- Enhanced logic of parsing MySQL/MariaDB ,multiple log patterns, using CloudWatch Log's timestamp instead of original log's timestamp, Aurora MySQL v5.6 #134,#143
- Unnecessary vpc endpoint for SNS service #132
- Fixed parse issue: responseElements.responseParameters.method.response.header.Access-Control-Allow-Methods, requestParameters.tableInput.parameters.projection.date.interval, requestParameters.FilterValues, requestParameters.CreateVpcEndpointRequest.TagSpecification.Tag.Value #124,#125,#136,#141
- Set default index patterns as log-* #133
- Enhanced for accurate parsing CloudWatch log/EventBridge #127,#129,#142
- Extracting event.outcome for Linux OS #126
- PR security vulnerability of urllib3 from 1.26.4 to 1.26.5 #121,#122
- Truncate and load longer field than 32766 byte because of Lucene limitation #102,#117
- Fixed parse issue: additionalEventData.bytesTransferredOut,requestParameters.CreateFleetRequest.TagSpecification.Tag.Value,requestParameters.overrides.containerOverrides.environment.value,requestParameters.CreateLaunchTemplateVersionRequest.LaunchTemplateData.TagSpecification.Tag.Value,responseElements.CreateLaunchTemplateVersionResponse.launchTemplateVersion.launchTemplateData.tagSpecificationSet.item.tagSet.item.value,requestParameters.result,requestParameters.tags,requestParameters.containerOverrides.environment.value,requestParameters.CreateSnapshotsRequest.TagSpecification.Tag.Value,requestParameters.status,requestParameters.searchExpression.subExpressions.subExpressions.filters.value,responseElements.description,responseElements.policy of CloudTrail #95,#98,#99,#100,#101,#106,#108,#109,#110,#111,#112,#113,#114
- Fixed regex pattern of ALB and CLB #115,#116
- FAQ: How to reset password of master user #93
- Chinese README #83
- Automatically importing dashboard and saved objects into Global tenants #82
- Change the default instance from t3.small to t3.medium #94
- Fixed parse issue: requestParameters.value, requestParameters.DescribeLaunchTemplateVersionsRequest.LaunchTemplateVersion.content, requestParameters.Tagging.TagSet.Tag.Value, requestParameters.content, requestParameters.groupDescription, requestParameters.logStreamNamePrefix, responseElements.availabilityZones of CloudTrail #84,#87,#88,#89,#90,#91,#92
- PR security vulnerability of urllib3 from 1.26.3 to 1.26.4 #85,#86
- Supported and normalized log: AWS Network Firewall, Amazon MSK(Blokder log), Amazon RDS(MariaDB/MySQL/PostgreSQL), vpc flow logs v5 format #5,#6,#17,#62,#67,#71
- Added multiline-text log parser #54
- Enabled system log of Amazon ES instance during initial deployment #64
- Enabled parsing china region and Gov Cloud for CloudTrail #72
- Refactored deployment script with crhelper #69
- Changed frequency of updating GeoDB from 0:20UTC to every 12 hours #70
- Changed s3_key of s3access log to regex. Users don't need to change anything #74
- Fixed parse issue: responseElements.credentials, requestParameters.CreateSnapshotsRequest.TagSpecification.Tag.Value, requestParameters.source, responseElements.multiAZ, requestParameters.partitionInputList, responseElements.errors.partitionValues of CloudTrail #33,#59,#66,#67,#75,#76
- Fixed issue with importing no data log #53
- Followed ECS rule for dns.question.name of vpc dns query log #60
- Fixed extracting distribution_id of CloudFront #61
- Fixed ambiguous error message like "[[], [], [], [], []]" may appear #68
- fixed ECS violation field, event.severity of Security Hub #77
- PR security vulnerability of urllib3 from 1.26.2 to 1.26.3 #78,#79
- Added IAM role for EC2 to import ETL existing Logs in S3 #41
- Supported timestamp with nested field #38
- Added custom metrics of es-loader for CloudWatch Metrics #13
- Enabled memory caching of Lambda function, es-loader. When hitting cache, parsing will be 500x faster. Other logics are also tuned up #39,#45
- Changing plain logging to structured logging of es-loader #13
- Allowed Amazon Athena to query S3 bucket made by SIEM #35
- Changed S3 file path of CloudFormation template. This affect only when you create own CloudFormation template #51
- Changed default Amazon ES's system index setting such as .opendistro-alerting-alert-history #48
- Limited number of es-loader's concurrency to avoid abnormal situation. Default is 10. You can change it to input CloudFormation parameter #43
- Helper functions in siem class such as merge function. Re-wrote and moved to siem.utils.py #39, #45
- Removed legacy parsing timestamp code. It used until v1.5.2
- Removed feature of importing logs from Kinesis Data Stream
- Fixed SQS timeout issue in VPC #52
- Fixed parse issue, ELB logs with space(#50, #47), CloudFront Logs with IPv6(#46), non-ascii and JSON logs(#49)
- Fixed NuSuchKey error when S3 key includes meta character #42
- Contribution: Deep Security Support. Thanks to @EijiSugiura #27
- Parse issue of S3 access logs with double quotes on the UA. #34
- Parse issue of ALB and CLB logs when IPv6 address is contained #31
- Issue with key policy created by CDK #32
- VPC config validation raises KeyError when using VPC peering instead IGW #29
- Supported and normalized log: Security Hub(Security Hub, GuardDuty, Macie, IAM Analyzer, Inspector), Linux SSH log via CWL, ECS via FireLens(Framework only) #7
- Dashboard: ELB, Security Hub(GuardDuty) #1,#18
- Split and parse logs in parallel when logs are huge.
- Functionality of filtering unnecessary logs #19
- Supported nano seconds of es-loader(just truncated) #23
- Amazon ES's initial version to v7.9.1 from v7.7.0 #24
- Increased es-loader's memory to 2048 MB from 512 MB #21
- Fix version in deployment procedure: Node(Active LTS), Python(3.8) #25,#26
- Dashboard: changed findings count logic of GuardDuty #20
- Enhanced extract logic of AWS account and region
- An error occurred when processing the file of size 0 byte with es-loader #15
- CLB logs parsing fails #16
- Regex error of S3 access log and CloudFront
- Typo of GuardDuty dashboard
- All files, initial version