Fixed aws_cloudwatch_log_group table for the error key column is not globally unique Closes #1975

[ParthaI]

Integration test logs

Logs

No env file present for the current environment:  staging 
 Falling back to .env config
No env file present for the current environment:  staging
customEnv TURBOT_TEST_EXPECTED_TIMEOUT undefined

SETUP: tests/aws_cloudwatch_log_group []

PRETEST: tests/aws_cloudwatch_log_group

TEST: tests/aws_cloudwatch_log_group
Running terraform
data.aws_region.alternate: Reading...
data.aws_region.alternate: Read complete after 0s [id=us-east-2]
data.aws_caller_identity.current: Reading...
data.aws_region.primary: Reading...
data.aws_partition.current: Reading...
data.aws_partition.current: Read complete after 0s [id=aws]
data.aws_region.primary: Read complete after 0s [id=us-east-1]
data.aws_caller_identity.current: Read complete after 0s [id=333333333333]
data.null_data_source.resource: Reading...
data.null_data_source.resource: Read complete after 0s [id=static]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_cloudwatch_log_group.named_test_resource will be created
  + resource "aws_cloudwatch_log_group" "named_test_resource" {
      + arn               = (known after apply)
      + id                = (known after apply)
      + name              = "turbottest1235"
      + name_prefix       = (known after apply)
      + retention_in_days = 0
      + skip_destroy      = false
      + tags              = {
          + "name" = "turbottest1235"
        }
      + tags_all          = {
          + "name" = "turbottest1235"
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + account_id    = "333333333333"
  + aws_partition = "aws"
  + region_name   = "us-east-1"
  + resource_aka  = (known after apply)
  + resource_name = "turbottest1235"
aws_cloudwatch_log_group.named_test_resource: Creating...
aws_cloudwatch_log_group.named_test_resource: Creation complete after 3s [id=turbottest1235]

Warning: Deprecated

  with data.null_data_source.resource,
  on variables.tf line 44, in data "null_data_source" "resource":
  44: data "null_data_source" "resource" {

The null_data_source was historically used to construct intermediate values
to re-use elsewhere in configuration, the same can now be achieved using
locals

(and one more similar warning elsewhere)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

account_id = "333333333333"
aws_partition = "aws"
region_name = "us-east-1"
resource_aka = "arn:aws:logs:us-east-1:333333333333:log-group:turbottest1235"
resource_name = "turbottest1235"

Running SQL query: test-get-query.sql
Warning: terminal options has been deprecated and will be removed in a future version of Steampipe.
These can now be set in a steampipe workspace.
(/Users/parthas/.steampipe/config/default.spc:34,20-42,2)
[
  {
    "name": "turbottest1235"
  }
]
✔ PASSED

Running SQL query: test-hydrate-query.sql
Warning: terminal options has been deprecated and will be removed in a future version of Steampipe.
These can now be set in a steampipe workspace.
(/Users/parthas/.steampipe/config/default.spc:34,20-42,2)
[
  {
    "tags": {
      "name": "turbottest1235"
    }
  }
]
✔ PASSED

Running SQL query: test-list-call-query.sql
Warning: terminal options has been deprecated and will be removed in a future version of Steampipe.
These can now be set in a steampipe workspace.
(/Users/parthas/.steampipe/config/default.spc:34,20-42,2)
[
  {
    "arn": "arn:aws:logs:us-east-1:333333333333:log-group:turbottest1235:*",
    "metric_filter_count": 0,
    "name": "turbottest1235",
    "retention_in_days": null,
    "stored_bytes": 0
  }
]
✔ PASSED

Running SQL query: test-turbot-query.sql
Warning: terminal options has been deprecated and will be removed in a future version of Steampipe.
These can now be set in a steampipe workspace.
(/Users/parthas/.steampipe/config/default.spc:34,20-42,2)
[
  {
    "account_id": "333333333333",
    "akas": [
      "arn:aws:logs:us-east-1:333333333333:log-group:turbottest1235:*"
    ],
    "partition": "aws",
    "region": "us-east-1",
    "tags": {
      "name": "turbottest1235"
    },
    "title": "turbottest1235"
  }
]
✔ PASSED

POSTTEST: tests/aws_cloudwatch_log_group

TEARDOWN: tests/aws_cloudwatch_log_group

SUMMARY:

1/1 passed.

Example query results

Results

> select * from aws_aab.aws_cloudwatch_log_group where name = 'test'
+------+------------------------------------------------------+---------------------------+------------+---------------------+-------------------+--------------+----------------------------------------------------------------------------------------------+------------->
| name | arn                                                  | creation_time             | kms_key_id | metric_filter_count | retention_in_days | stored_bytes | data_protection                                                                              | data_protect>
+------+------------------------------------------------------+---------------------------+------------+---------------------+-------------------+--------------+----------------------------------------------------------------------------------------------+------------->
| test | arn:aws:logs:us-east-1:333333333333:log-group:test:* | 2023-11-17T22:34:26+05:30 | <null>     | 0                   | <null>            | 0            | {"LastUpdatedTime":null,"LogGroupIdentifier":null,"PolicyDocument":null,"ResultMetadata":{}} | <null>      >
| test | arn:aws:logs:us-east-2:333333333333:log-group:test:* | 2023-10-09T21:31:24+05:30 | <null>     | 0                   | 3                 | 0            | {"LastUpdatedTime":null,"LogGroupIdentifier":null,"PolicyDocument":null,"ResultMetadata":{}} | <null>      >
+------+------------------------------------------------------+---------------------------+------------+---------------------+-------------------+--------------+----------------------------------------------------------------------------------------------+------------->

Time: 7.8s. Rows fetched: 3. Hydrate calls: 9.

> select * from aws_aab.aws_cloudwatch_log_group where name = 'test' and region = 'us-east-1'
+------+------------------------------------------------------+---------------------------+------------+---------------------+-------------------+--------------+----------------------------------------------------------------------------------------------+------------->
| name | arn                                                  | creation_time             | kms_key_id | metric_filter_count | retention_in_days | stored_bytes | data_protection                                                                              | data_protect>
+------+------------------------------------------------------+---------------------------+------------+---------------------+-------------------+--------------+----------------------------------------------------------------------------------------------+------------->
| test | arn:aws:logs:us-east-1:333333333333:log-group:test:* | 2023-11-17T22:34:26+05:30 | <null>     | 0                   | <null>            | 0            | {"LastUpdatedTime":null,"LogGroupIdentifier":null,"PolicyDocument":null,"ResultMetadata":{}} | <null>      >
+------+------------------------------------------------------+---------------------------+------------+---------------------+-------------------+--------------+----------------------------------------------------------------------------------------------+------------->

Time: 3.3s. Rows fetched: 1. Hydrate calls: 3.
> select * from aws_aab.aws_cloudwatch_log_group where name = 'test' and region = 'us-east-4'
+------+-----+---------------+------------+---------------------+-------------------+--------------+-----------------+------------------------+-------+------+------+-----------+--------+------------+------+
| name | arn | creation_time | kms_key_id | metric_filter_count | retention_in_days | stored_bytes | data_protection | data_protection_policy | title | tags | akas | partition | region | account_id | _ctx |
+------+-----+---------------+------------+---------------------+-------------------+--------------+-----------------+------------------------+-------+------+------+-----------+--------+------------+------+
+------+-----+---------------+------------+---------------------+-------------------+--------------+-----------------+------------------------+-------+------+------+-----------+--------+------------+------+

Time: 93ms.

[ParthaI]

Hi @bryanburke, I anticipate this fix will resolve the issue. I've tested the code changes with the following test cases, including creating two Log Groups with the same name in different regions within an account.

Case 1: By providing name value in where clause (Returned two results without any error)

> select * from aws_aab.aws_cloudwatch_log_group where name = 'test'
+------+------------------------------------------------------+---------------------------+------------+---------------------+-------------------+--------------+----------------------------------------------------------------------------------------------+------------->
| name | arn                                                  | creation_time             | kms_key_id | metric_filter_count | retention_in_days | stored_bytes | data_protection                                                                              | data_protect>
+------+------------------------------------------------------+---------------------------+------------+---------------------+-------------------+--------------+----------------------------------------------------------------------------------------------+------------->
| test | arn:aws:logs:us-east-1:333333333333:log-group:test:* | 2023-11-17T22:34:26+05:30 | <null>     | 0                   | <null>            | 0            | {"LastUpdatedTime":null,"LogGroupIdentifier":null,"PolicyDocument":null,"ResultMetadata":{}} | <null>      >
| test | arn:aws:logs:us-east-2:333333333333:log-group:test:* | 2023-10-09T21:31:24+05:30 | <null>     | 0                   | 3                 | 0            | {"LastUpdatedTime":null,"LogGroupIdentifier":null,"PolicyDocument":null,"ResultMetadata":{}} | <null>      >
+------+------------------------------------------------------+---------------------------+------------+---------------------+-------------------+--------------+----------------------------------------------------------------------------------------------+------------->

Time: 7.8s. Rows fetched: 3. Hydrate calls: 9.

Case 2: By providing name and a valid region value in the where clause (Returned one result for a provided region in the where clause)

> select * from aws_aab.aws_cloudwatch_log_group where name = 'test' and region = 'us-east-1'
+------+------------------------------------------------------+---------------------------+------------+---------------------+-------------------+--------------+----------------------------------------------------------------------------------------------+------------->
| name | arn                                                  | creation_time             | kms_key_id | metric_filter_count | retention_in_days | stored_bytes | data_protection                                                                              | data_protect>
+------+------------------------------------------------------+---------------------------+------------+---------------------+-------------------+--------------+----------------------------------------------------------------------------------------------+------------->
| test | arn:aws:logs:us-east-1:333333333333:log-group:test:* | 2023-11-17T22:34:26+05:30 | <null>     | 0                   | <null>            | 0            | {"LastUpdatedTime":null,"LogGroupIdentifier":null,"PolicyDocument":null,"ResultMetadata":{}} | <null>      >
+------+------------------------------------------------------+---------------------------+------------+---------------------+-------------------+--------------+----------------------------------------------------------------------------------------------+------------->

Time: 3.3s. Rows fetched: 1. Hydrate calls: 3.

Case 3: By providing name and an invalid region value in where clause (Returned empty row)

> select * from aws_aab.aws_cloudwatch_log_group where name = 'test' and region = 'us-east-4'
+------+-----+---------------+------------+---------------------+-------------------+--------------+-----------------+------------------------+-------+------+------+-----------+--------+------------+------+
| name | arn | creation_time | kms_key_id | metric_filter_count | retention_in_days | stored_bytes | data_protection | data_protection_policy | title | tags | akas | partition | region | account_id | _ctx |
+------+-----+---------------+------------+---------------------+-------------------+--------------+-----------------+------------------------+-------+------+------+-----------+--------+------------+------+
+------+-----+---------------+------------+---------------------+-------------------+--------------+-----------------+------------------------+-------+------+------+-----------+--------+------------+------+

Time: 93ms.

Could you please give it a try by checking out the PR branch ('issue-1975')? Please let us know if you encounter any other issues.

Steps for checkout

• git clone https://github.com/turbot/steampipe-plugin-aws.git
• cd steampipe-plugin-aws
• git checkout issue-1975
• make
• Run steampipe query

Thank you for your interest in Steampipe. We appreciate it!

[bryanburke]

@ParthaI I just tested the plugin built from this branch, and the issue is resolved. Thanks!

[misraved]

@ParthaI I believe we need to update the table docs to reflect the usage of both the name and the region column.

Please let me know if you have questions. Thanks!!

[misraved]

@ParthaI shouldn't we be adding an additional check for the region column as well?

If a user does not pass in the region along with the name, then we should return nil.

[ParthaI]

@misraved, in accordance with our design, we utilize the SupportedRegionMatrix.

Here's how it works:

• If a user queries without specifying the region but includes the name, we will perform a list API call. This approach ensures that there will be no errors such as 'key column is not globally unique'.
• Conversely, if the region value is specified along with the name, a Get API call will be executed, but only for the specified region.

Note: We can have multiple LogGroups with same name but in different region. Therefore, our approach ensures that users receive actual results rather than an empty row in such cases.