Add record id to artifacts and record

[chunyong-lin]

to: @airbnb/streamalert-maintainers
related to: #1238
resolves:

Background

This PR is tiny and it is to add a new key streamalert_record_id to artifacts as well as original records (in streamalert_normalization field). This new keywill be helpful tracing back to the original record when searching in "artifacts" table.

For example, we will be able to use table join search

SELECT artifacts.*,
         events.detail
FROM 
    (SELECT streamalert_record_id AS record_id,
         type,
         value
    FROM "PREFIX_streamalert"."artifacts"
    WHERE dt='2020-04-30-01'
            AND value='Root') AS artifacts
LEFT JOIN 
    (SELECT CAST(json_extract(streamalert_normalization,
         '$.streamalert_record_id') AS varchar) AS record_id, detail
    FROM "PREFIX_streamalert"."cloudwatch_events"
    WHERE dt='2020-04-30-01') AS events
    ON artifacts.record_id = events.record_id LIMIT 10 

Changes

• Change key streamalert:normalization to streamalert_normalization to support athena better.
• Add a new key streamalert_record_id to artifacts, for example

{
  "function": "AWS region",
  "streamalert_record_id": "abcdef0123456789",
  "source_type": "cloudwatch",
  "type": "cloudwatch:events",
  "value": "us-west-2"
}

• The new key streamalert_record_id will be also insert to original record which will be returned to source Firehose delivery stream and saved in S3 for Historical Search. Transformed record will look like this

{
    "record": {
        "region": "us-east-1",
        "detail": {
            "awsRegion": "us-west-2"
        }
    },
    "streamalert_normalization": {
        "streamalert_record_id": "abcdef0123456789",
        "region": [
            {
                "values": ["region_name"],
                "function": "AWS region"
            },
            {
                "values": ["region_name"],
                "function": "AWS region"
            }
        ]
    }
}

Testing

• Update unit test cases
• Deploy to staging

[Ryxias]

YES YES YES YES YES

YES!