bug: document column on aws_iam_policy_versions table is always null

[aruneko]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When I fetch aws_iam_policy_versions table with AWS Source plugin v23.2.0, the document column always null.

Expected Behavior

The document column includes policy document json.

CloudQuery (redacted) config

kind: source
spec:
name: aws
path: cloudquery/aws
registry: cloudquery
version: "v23.2.0"
destinations: ["postgresql"]
tables:
- aws_iam_accounts
- aws_iam_credential_reports
- aws_iam_group_policies
- aws_iam_groups
- aws_iam_password_policies
- aws_iam_policies
- aws_iam_role_policies
- aws_iam_roles
- aws_iam_server_certificates
- aws_iam_user_access_keys
- aws_iam_user_attached_policies
- aws_iam_user_policies
- aws_iam_users
- aws_iam_virtual_mfa_devices
skip_tables:
- aws_iam_group_last_accessed_details
- aws_iam_policy_last_accessed_details
- aws_iam_role_last_accessed_details
- aws_iam_user_last_accessed_details
- aws_iam_group_attached_policies
- aws_iam_role_attached_policies
- aws_iam_signing_certificates
- aws_iam_ssh_public_keys
- aws_iam_user_groups
spec:
concurrency: 1000
accounts:
- id: "debug_account"
local_profile: "debug"
aws_debug: false

Steps To Reproduce

No response

CloudQuery (redacted) logs

2023-12-26T06:47:38Z INF table sync finished client=596529977512:us-east-1 errors=0 module=aws-src resources=1203 table=aws_iam_policies
2023-12-26T06:47:38Z INF table sync finished client=596529977512:us-east-1 errors=0 module=aws-src resources=3594 table=aws_iam_policy_versions

CloudQuery version

4.3.4

Additional Context

I know my commit causes the problem.

Since ListPolicyVersions API doesn't returns document column, if we fetch the column we have to call GetPolicyVersion API for each response of ListPolicyVersions API.

At the moment, I have 3 suggestions to fix.

• use column resolver to fetch document column in aws_iam_policy_versions table.
  • pros: It may not be a breaking change.
  • cons: Most of cases I don't want to fetch all of policy documents. It is enough to fetch a default version of policy document. In addition, it may cause ThrottlingException and draw the fetching out.
• make aws_iam_policy_version_documents sub table against to aws_iam_policy_versions table.
  • pros: Users can choose fetching or not fetching the table. If users fetching the table, the concurrency setting is effective for morality crawling.
  • cons: same as above. And it may be a breaking change.
• make aws_iam_policy_default_version_documents sub table against to aws_iam_policy table.
  • pros: Since it fetches only default policy version, the numbers of calling API are limited.
  • cons: We can't fetch all of version history of each policy documents.

Any other ideas are welcome for me!

Pull request (optional)

• I can submit a pull request

[bbernays]

@aruneko - This should be fixed in the latest version of the AWS Plugin v23.3.1
https://hub.cloudquery.io/plugins/source/cloudquery/aws/v23.3.1/docs