Retro Tag script does not recognize CloudTrail events for CreateTable on DynamoDb.

[ecout]

If auto-tag is enabled, the lambda tags the dynamodb tables correctly, but when running the Ruby script the following is returned:

Completed collecting resources in 34 seconds Found 2 total events to process, looking for events with existing resources... Completed event scan in 0 seconds +------------------------------------------------------------------------------------+ | Retro-Active Tagging for Existing Resources Summary | +---------------------------+------------------------------------------------+-------+ | Service | Event | Count | +--------------------------------------------------------------------------------------+ | DynamoDB Tables | CreateTable | 2 | +---------------------------+------------------------------------------------+-------+ Total CloudTrail Events: 0 Unique CloudTrail S3 Objects: 0 Starting 3 Lambda Function threads... Error: No CloudTrail S3 objects found to process

Obviously the CloudTrail .gz files are in the S3 bucket. Other events are processed successfully.

Keep in mind that the only difference between how the Auto-Tag and the Retro-Tag lambda functions are configured is the handler for each case and that the error is being thrown by the retro_tag.rb script on line 301 because all_cloudtrail_s3_keys.count !> 0.

While services.cloudtrail_s3 prints values for other resources it ALWAYS comes back empty for DynamoDB, regardless of whether the cache contains the path to the CloudTrail event on S3, the resource is available, and the Athena .csv contains the event.

Why is it that the aws_resource/dynamo_db_table.rb file has the following property empty? Where other classes in that module have the arn.
DynamoDB
def aws_response_resource_name '' end

RDS:
def aws_response_resource_name 'db_instance_arn' end

The following elements do NOT exist in the DynamoDB response, hence the arn must be built:

`def resource_name_exists?(**args)
(args[:response_elements]['tableDescription'] &&
args[:response_elements]['tableDescription']['tableArn'])
end

def resource_name(**args)
  args[:response_elements]['tableDescription']['tableArn'].sub(/.*table\/(.*)$/, '\1')
end`

Reference: https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/DynamoDB/Client.html#list_tables-instance_method

[rayjanoka]

Hey ya so there are two separate parts for each resource. The CloudTrail event processing and a check to verify the resource still exists with the ruby sdk.

First the CloudTrail event processing below says find the CreateTable event and then grab the data. The CloudTrail output is a pain to get, sometimes you have to just create the resource in AWS and go look in CloudTrail at the event yourself.

    def aws_event_name
      %w[CreateTable]
    end

    def resource_name_exists?(**args)
      (args[:response_elements]['tableDescription'] &&
          args[:response_elements]['tableDescription']['tableArn'])
    end

    def resource_name(**args)
      args[:response_elements]['tableDescription']['tableArn'].sub(/.*table\/(.*)$/, '\1')
    end

Sample CreateTable CloudTrail Event, here you can see the :response_elements -> tableDesciption -> tableArn

➜ cat event-dynamodb_table-cloudtrail_event3.json
{
    "version": "0",
    "id": "7ee9a2a2-e46e-0854-675f-f23c81ec059b",
    "detail-type": "AWS API Call via CloudTrail",
    "source": "aws.dynamodb",
    "account": "1234567890",
    "time": "2018-02-01T04:19:54Z",
    "region": "ap-southeast-2",
    "resources": [],
    "detail": {
        "eventVersion": "1.05",
        "userIdentity": {
            "invokedBy": "cloudformation.amazonaws.com"
        },
        "eventTime": "2018-02-01T04:19:54Z",
        "eventSource": "dynamodb.amazonaws.com",
        "eventName": "CreateTable",
        "awsRegion": "ap-southeast-2",
        "sourceIPAddress": "cloudformation.amazonaws.com",
        "userAgent": "cloudformation.amazonaws.com",
        "requestParameters": {
            "provisionedThroughput": {
                "writeCapacityUnits": 1,
                "readCapacityUnits": 1
            },
            "tableName": "autotag-test-AutoTagTestDynamoDBTable-10K8AMLBNY26V",
            "keySchema": [
                {
                    "attributeName": "NodeID",
                    "keyType": "HASH"
                }
            ],
            "attributeDefinitions": [
                {
                    "attributeType": "S",
                    "attributeName": "NodeID"
                }
            ]
        },
        "responseElements": {
            "tableDescription": {
                "attributeDefinitions": [
                    {
                        "attributeType": "S",
                        "attributeName": "NodeID"
                    }
                ],
                "tableName": "autotag-test-AutoTagTestDynamoDBTable-10K8AMLBNY26V",
                "provisionedThroughput": {
                    "writeCapacityUnits": 1,
                    "numberOfDecreasesToday": 0,
                    "readCapacityUnits": 1
                },
                "tableArn": "arn:aws:dynamodb:ap-southeast-2:1234567890:table/autotag-test-AutoTagTestDynamoDBTable-10K8AMLBNY26V",
                "creationDateTime": "Feb 1, 2018 4:19:54 AM",
                "tableId": "9f690050-78a4-44a1-9e91-eb69d4398a67",
                "itemCount": 0,
                "keySchema": [
                    {
                        "attributeName": "NodeID",
                        "keyType": "HASH"
                    }
                ],
                "tableStatus": "CREATING",
                "tableSizeBytes": 0
            }
        },
        "requestID": "M5IAFMVPIGLHF9G1U8TVG4MB6BVV4KQNSO5AEMVJF66Q9ASUAAJG",
        "eventID": "588f1ec2-e64c-419e-ab68-11551a49afc6",
        "eventType": "AwsApiCall",
        "apiVersion": "2012-08-10",
        "recipientAccountId": "1234567890"
    }
}⏎           

Then we check to make sure the resource exists by calling list_tables, look for the table_names array and response_resource_name() shouldn't be getting called because a string is returned here: https://github.com/GorillaStack/retro-tag/blob/master/aws_resource/default.rb#L63

def aws_client_method
      'list_tables'
    end

    def aws_client_method_args
      {}
    end

    def aws_response_collection
      'table_names'
    end

    def aws_response_resource_name
      ''
    end

[rayjanoka]

You can try running with --details and see if we get any help