fix(aws)!: Ensure PKs are consistent for multiple tables#9677
Merged
kodiakhq[bot] merged 4 commits intocloudquery:mainfrom Apr 5, 2023
Merged
fix(aws)!: Ensure PKs are consistent for multiple tables#9677kodiakhq[bot] merged 4 commits intocloudquery:mainfrom
kodiakhq[bot] merged 4 commits intocloudquery:mainfrom
Conversation
This PR has the following changes to source plugin(s) tables:
|
disq
approved these changes
Apr 4, 2023
Member
disq
left a comment
disq
left a comment
There was a problem hiding this comment.
throw 🔥; (couldn't find a flamethrower emoji so defaulted to ECMAScript)
Member
|
Adding a comment here that for the |
erezrokah
approved these changes
Apr 5, 2023
erezrokah
changed the title
kodiakhq bot
pushed a commit
that referenced
this pull request
Apr 5, 2023
🤖 I have created a release *beep* *boop* --- ## [16.0.0](plugins-source-aws-v15.7.0...plugins-source-aws-v16.0.0) (2023-04-05) ### ⚠ BREAKING CHANGES * **aws:** Add `request_account_id` to `aws_organizations_accounts` primary key ([#9733](#9733)) * **aws:** Add `request_account_id` to `aws_organizations_roots` primary key ([#9732](#9732)) * **aws:** Add Columns to `aws_ec2_subnets` Primary Key to handle when subnet is shared ([#9731](#9731)) * **aws:** Primary Key changed for `aws_iam_signing_certificates` ([#9677](#9677)) * **aws:** Primary Key changed for `aws_iam_ssh_public_keys` ([#9677](#9677)) * **aws:** Primary Key changed for `aws_iam_user_groups` ([#9677](#9677)) * **aws:** Primary Key changed for `aws_iam_user_policies` ([#9677](#9677)) * **aws:** In `aws_iam_user_policies`, primary Key was changed from `(_cq_id)` to `(account_id, user_arn, policy_name)` so in `overwrite` policies are updated and not always appended * **aws:** Move Role and Group Attached Policies to Separate tables ([#9508](#9508)) * **aws:** Replace `arn` with `id` in `aws_eventbridge_event_bus_targets` primary key ([#9648](#9648)) * **aws:** Replace `arn` with `id` in `aws_eventbridge_event_bus_targets` primary key as `arn` refers to the ARN of the resource that is being invoked, not the ARN of the target resource. The `Id` is what uniquely identifies the target within a specific rule. * **aws:** Change Primary keys for `aws_dynamodb_global_tables` to include `region` ([#9651](#9651)) * **aws:** Change `aws_inspector2_findings` Primary Key to Include `request_account_id` and `request_region` ([#9650](#9650)) * **aws:** Add `cluster_arn` to `aws_ecs_cluster_services` primary key as cluster ID is absent in the "old format" ARNs. See [ecs-account-settings.html#ecs-resource-ids](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-account-settings.html#ecs-resource-ids) for more details. * **aws:** Remove `region` from `aws_docdb_global_clusters` as global tables are visible from many regions. Additionally, `aws_docdb_global_clusters` now uses account-level multiplexing instead of region-level, so only 1 entry would be returned when previously it could produce several entries. * **aws:** GuardDuty detector findings PK should include detector ([#9555](#9555)) * **aws:** Remove `region` from `aws_neptune_global_clusters` as global clusters are visible from many regions. Additionally, `aws_neptune_global_clusters` now uses account-level multiplexing instead of region-level, so only 1 entry would be returner when previously it could produce several entries. * **aws:** Missing partial/parent PK for `aws_ssm_instance_patches` ([#8437](#8437)) * **aws:** Add `updated_at` to `aws_securityhub_findings` PK ([#8914](#8914)) * **aws:** Missing partial PK for aws_ram_resource_share_permissions ([#8433](#8433)) * **aws:** Replace the type of columns ending with `_at` to `Timestamp`. Initially it wasn't the case as the API markup of AWS doesn't include the proper annotations. See aws/aws-sdk#492 for more details. The following tables are affected: * `aws_guardduty_detector_findings`: `created_at`, `updated_at` * `aws_guardduty_detector_members`: `invited_at`, `updated_at` * `aws_guardduty_detectors`: `created_at`, `updated_at` * `aws_securityhub_findings`: `created_at`, `first_observed_at`, `last_observed_at`, `updated_at` ### This Release has the Following Changes to Tables - Table `aws_ec2_subnets`: column `account_id` removed from table (:warning: breaking) - Table `aws_ec2_subnets`: column `region` removed from table (:warning: breaking) - Table `aws_ec2_subnets`: column added with name `request_account_id (PK)` and type `String` (:warning: breaking) - Table `aws_ec2_subnets`: column added with name `request_region (PK)` and type `String` (:warning: breaking) - Table `aws_iam_group_last_accessed_details`: column `arn` removed from table (:warning: breaking) - Table `aws_iam_group_last_accessed_details`: column added with name `group_arn (PK)` and type `String` (:warning: breaking) - Table `aws_iam_group_last_accessed_details`: primary key constraint added to column `account_id` (:warning: breaking) - Table `aws_iam_group_policies`: column `group_id` removed from table (:warning: breaking) - Table `aws_iam_group_policies`: primary key constraint added to column `account_id` (:warning: breaking) - Table `aws_iam_group_policies`: primary key constraint added to column `group_arn` (:warning: breaking) - Table `aws_iam_group_policies`: primary key constraint added to column `policy_name` (:warning: breaking) - Table `aws_iam_group_policies`: primary key constraint removed from column `_cq_id` (:warning: breaking) - Table `aws_iam_groups`: column `id` removed from table (:warning: breaking) - Table `aws_iam_groups`: primary key constraint added to column `arn` (:warning: breaking) - Table `aws_iam_role_last_accessed_details`: column `arn` removed from table (:warning: breaking) - Table `aws_iam_role_last_accessed_details`: column added with name `role_arn (PK)` and type `String` (:warning: breaking) - Table `aws_iam_role_last_accessed_details`: primary key constraint added to column `account_id` (:warning: breaking) - Table `aws_iam_role_policies`: column order changed for `policy_name` - Table `aws_iam_roles`: column `id` removed from table (:warning: breaking) - Table `aws_iam_roles`: primary key constraint added to column `arn` (:warning: breaking) - Table `aws_iam_signing_certificates`: primary key constraint added to column `account_id` (:warning: breaking) - Table `aws_iam_ssh_public_keys`: primary key constraint added to column `account_id` (:warning: breaking) - Table `aws_iam_ssh_public_keys`: primary key constraint added to column `user_arn` (:warning: breaking) - Table `aws_iam_user_groups`: primary key constraint added to column `account_id` (:warning: breaking) - Table `aws_iam_user_groups`: primary key constraint added to column `user_arn` (:warning: breaking) - Table `aws_iam_user_groups`: primary key constraint removed from column `user_id` (:warning: breaking) - Table `aws_iam_user_last_accessed_details`: column `arn` removed from table (:warning: breaking) - Table `aws_iam_user_last_accessed_details`: column added with name `user_arn (PK)` and type `String` (:warning: breaking) - Table `aws_iam_user_last_accessed_details`: primary key constraint added to column `account_id` (:warning: breaking) - Table `aws_iam_user_policies`: primary key constraint added to column `account_id` (:warning: breaking) - Table `aws_iam_user_policies`: primary key constraint added to column `policy_name` (:warning: breaking) - Table `aws_iam_user_policies`: primary key constraint added to column `user_arn` (:warning: breaking) - Table `aws_iam_user_policies`: primary key constraint removed from column `_cq_id` (:warning: breaking) - Table `aws_iam_users`: column `id` removed from table (:warning: breaking) - Table `aws_iam_users`: primary key constraint added to column `arn` (:warning: breaking) - Table `aws_organizations_accounts`: column `account_id` removed from table (:warning: breaking) - Table `aws_organizations_accounts`: column added with name `request_account_id (PK)` and type `String` (:warning: breaking) - Table `aws_organizations_roots`: column `account_id` removed from table (:warning: breaking) - Table `aws_organizations_roots`: column added with name `request_account_id (PK)` and type `String` (:warning: breaking) ### Features * **aws-services:** Support newly added regions ([#9654](#9654)) ([69d5f35](69d5f35)) * **aws:** ELBv2 Listener Rules ([#9484](#9484)) ([c235fca](c235fca)) * **aws:** More Redshift resources ([#9486](#9486)) ([dacc048](dacc048)) * **aws:** Move Role and Group Attached Policies to Separate tables ([#9508](#9508)) ([18c5351](18c5351)) ### Bug Fixes * **aws:** Add `cluster_arn` to `aws_ecs_cluster_services` primary key ([#9559](#9559)) ([3f0c7ee](3f0c7ee)) * **aws:** Add `request_account_id` to `aws_organizations_accounts` primary key ([#9733](#9733)) ([3c7d7a8](3c7d7a8)) * **aws:** Add `request_account_id` to `aws_organizations_roots` primary key ([#9732](#9732)) ([2fbcb6b](2fbcb6b)) * **aws:** Add `updated_at` to `aws_securityhub_findings` PK ([#8914](#8914)) ([59b25d3](59b25d3)) * **aws:** Add Columns to `aws_ec2_subnets` Primary Key to handle when subnet is shared ([#9731](#9731)) ([c18928f](c18928f)) * **aws:** Change `aws_inspector2_findings` Primary Key to Include `request_account_id` and `request_region` ([#9650](#9650)) ([e51c680](e51c680)) * **aws:** Change Primary keys for `aws_dynamodb_global_tables` to include `region` ([#9651](#9651)) ([63baf42](63baf42)) * **aws:** GuardDuty detector findings PK should include detector ([#9555](#9555)) ([71f0826](71f0826)) * **aws:** Implement proper tag pagination ([#9646](#9646)) ([bcb82d6](bcb82d6)) * **aws:** Missing partial PK for aws_ram_resource_share_permissions ([#8433](#8433)) ([bf48d05](bf48d05)) * **aws:** Missing partial/parent PK for `aws_ssm_instance_patches` ([#8437](#8437)) ([49bdcd3](49bdcd3)) * **aws:** Only send images once ([#9644](#9644)) ([d883e84](d883e84)) * **aws:** Primary Key changed for `aws_iam_groups` ([#9677](#9677)) ([03b80a1](03b80a1)) * **aws:** Primary Key changed for `aws_iam_signing_certificates` ([#9677](#9677)) ([03b80a1](03b80a1)) * **aws:** Primary Key changed for `aws_iam_ssh_public_keys` ([#9677](#9677)) ([03b80a1](03b80a1)) * **aws:** Primary Key changed for `aws_iam_user_groups` ([#9677](#9677)) ([03b80a1](03b80a1)) * **aws:** Primary Key changed for `aws_iam_user_policies` ([#9677](#9677)) ([03b80a1](03b80a1)) * **aws:** Primary Key changed for `aws_iam_users` ([#9677](#9677)) ([03b80a1](03b80a1)) * **aws:** Remove `group_id` from `aws_iam_group_policies` ([#9677](#9677)) ([03b80a1](03b80a1)) * **aws:** Remove `id` from `aws_iam_groups` ([#9677](#9677)) ([03b80a1](03b80a1)) * **aws:** Remove `id` from `aws_iam_roles` ([#9677](#9677)) ([03b80a1](03b80a1)) * **aws:** Remove `id` from `aws_iam_users` ([#9677](#9677)) ([03b80a1](03b80a1)) * **aws:** Remove `region` from `aws_docdb_global_clusters` ([#9558](#9558)) ([ea9750b](ea9750b)) * **aws:** Remove `region` from `aws_neptune_global_clusters` ([#9556](#9556)) ([f09e767](f09e767)) * **aws:** Remove `tags` from `aws_eventbridge_event_bus_targets` ([#9648](#9648)) ([fa64254](fa64254)) * **aws:** Rename `arn` field in `aws_iam_group_last_accessed_details` ([#9677](#9677)) ([03b80a1](03b80a1)) * **aws:** Rename `arn` field in `aws_iam_role_last_accessed_details` ([#9677](#9677)) ([03b80a1](03b80a1)) * **aws:** Rename `arn` field in `aws_iam_user_last_accessed_details` ([#9677](#9677)) ([03b80a1](03b80a1)) * **aws:** Replace `arn` with `id` in `aws_eventbridge_event_bus_targets` primary key ([#9648](#9648)) ([fa64254](fa64254)) * **aws:** Replace the type of columns ending with `_at` to `Timestamp` ([#8912](#8912)) ([d3b9f71](d3b9f71)) * **aws:** Tag Pagination Properly Handled ([#9611](#9611)) ([1af31aa](1af31aa)) * **aws:** Update Primary key for `aws_iam_group_policies` ([#9677](#9677)) ([03b80a1](03b80a1)) * **deps:** Update github.com/gocarina/gocsv digest to 9a18a84 ([#9563](#9563)) ([4c5432b](4c5432b)) * **deps:** Update golang.org/x/exp digest to 10a5072 ([#9587](#9587)) ([31f913f](31f913f)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/feature/s3/manager to v1.11.60 ([#9616](#9616)) ([d155d28](d155d28)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/ecrpublic to v1.15.6 ([#9617](#9617)) ([710b564](710b564)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/efs to v1.19.9 ([#9618](#9618)) ([399e56e](399e56e)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/eks to v1.27.8 ([#9619](#9619)) ([9ad0589](9ad0589)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticache to v1.26.6 ([#9620](#9620)) ([f33de4f](f33de4f)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticbeanstalk to v1.15.6 ([#9621](#9621)) ([3c144ba](3c144ba)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticloadbalancing to v1.15.6 ([#9622](#9622)) ([da307f7](da307f7)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 to v1.19.7 ([#9623](#9623)) ([1ca1a82](1ca1a82)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elasticsearchservice to v1.18.7 ([#9624](#9624)) ([efa8c2c](efa8c2c)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/elastictranscoder to v1.14.6 ([#9625](#9625)) ([3322c94](3322c94)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/firehose to v1.16.8 ([#9626](#9626)) ([279dd25](279dd25)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/fsx to v1.28.7 ([#9627](#9627)) ([3e0491a](3e0491a)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/glacier to v1.14.7 ([#9628](#9628)) ([9d10696](9d10696)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/iam to v1.19.8 ([#9629](#9629)) ([807fc7c](807fc7c)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/identitystore to v1.16.6 ([#9630](#9630)) ([e3dc54c](e3dc54c)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/inspector to v1.13.6 ([#9631](#9631)) ([55706fc](55706fc)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/inspector2 to v1.11.7 ([#9632](#9632)) ([791f9a4](791f9a4)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery to v1.7.25 ([#9634](#9634)) ([16a220f](16a220f)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/kafka to v1.19.7 ([#9635](#9635)) ([540952d](540952d)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/kinesis to v1.17.8 ([#9636](#9636)) ([f08edc4](f08edc4)) * **deps:** Update module github.com/aws/aws-sdk-go-v2/service/kms to v1.20.8 ([#9637](#9637)) ([e2c1a79](e2c1a79)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.44.1 ([#9520](#9520)) ([202c31b](202c31b)) * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.44.2 ([#9661](#9661)) ([a27dc84](a27dc84)) * **deps:** Update module github.com/mattn/go-isatty to v0.0.18 ([#9609](#9609)) ([5b2908e](5b2908e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BEGIN_COMMIT_OVERRIDE
fix(aws): Rename
arnfield inaws_iam_group_last_accessed_details(#9677)BREAKING-CHANGE: In table
aws_iam_group_last_accessed_details, replacearnwithgroup_arnto indicate the ARN is the value of thegroup_arnthat is providing access. Along with changing this value primary key also changed from(arn, service_namespace)to(account_id, group_arn, service_namespace)because thegroup_arndoesn't include a value that indicates theaccount_id.fix(aws): Rename
arnfield inaws_iam_role_last_accessed_details(#9677)BREAKING-CHANGE: In table
aws_iam_role_last_accessed_details, replacearnwithrole_arnto indicate the ARN is the value of therole_arnthat is providing access. Along with changing this value primary key also changed from(arn, service_namespace)to(account_id, role_arn, service_namespace)because therole_arndoesn't include a value that indicates theaccount_idfix(aws): Rename
arnfield inaws_iam_user_last_accessed_details(#9677)BREAKING-CHANGE: In table
aws_iam_user_last_accessed_details, replacearnwithrole_arnto indicate the ARN is the value of theuser_arnthat is providing access. Along with changing this value primary key also changed from(arn, service_namespace)to(account_id, user_arn, service_namespace)because theuser_arndoesn't include a value that indicates theaccount_idfix(aws): Update Primary key for
aws_iam_group_policies(#9677)BREAKING-CHANGE: In table
aws_iam_group_policies, primary Key was changed from(_cq_id)to(account_id, group_arn, policy_name)so inoverwritepolicies are updated and not always appendedfix(aws): Remove
group_idfromaws_iam_group_policies(#9677)BREAKING-CHANGE: In table
aws_iam_group_policies, thegroup_idis not used as part of the PK, so it was removed from the table. If a you need the value you can join on the parent tablefix(aws): Remove
idfromaws_iam_groups(#9677)BREAKING-CHANGE: In
aws_iam_groups, theidcolumn was removed. The value is still available in thegroup_idcolumnfix(aws): Primary Key changed for
aws_iam_groups(#9677)BREAKING-CHANGE: In
aws_iam_groups, primary Key was changed from(account_id, id)to(account_id, arn)fix(aws): Remove
idfromaws_iam_roles(#9677)BREAKING-CHANGE: In
aws_iam_roles, theidcolumn was removed. The value is still available in therole_idcolumnfix(aws): Remove
idfromaws_iam_users(#9677)BREAKING-CHANGE: In
aws_iam_users, theidcolumn was removed. The value is still available in theuser_idcolumnfix(aws): Primary Key changed for
aws_iam_users(#9677)BREAKING-CHANGE: In
aws_iam_users, primary Key was changed from(id, account_id)to(account_id, arn)fix(aws)!: Primary Key changed for
aws_iam_signing_certificates(#9677)BREAKING-CHANGE: In
aws_iam_signing_certificates, primary Key was changed from(user_arn, certificate_id)to(account_id, user_arn, certificate_id)becauseuser_arnis not unique across accountsfix(aws)!: Primary Key changed for
aws_iam_ssh_public_keys(#9677)BREAKING-CHANGE: In
aws_iam_ssh_public_keys, primary Key was changed from(ssh_public_key_id)to(account_id, user_arn, ssh_public_key_id)becausessh_public_key_idis not unique across users or accountsfix(aws)!: Primary Key changed for
aws_iam_user_groups(#9677)BREAKING-CHANGE: In
aws_iam_user_groups, primary Key was changed from(user_id, arn)to(account_id, user_arn, arn)in order to stay consistent with the parent table primary keysfix(aws)!: Primary Key changed for
aws_iam_user_policies(#9677)BREAKING-CHANGE: In
aws_iam_user_policies, primary Key was changed from(_cq_id)to(account_id, user_arn, policy_name)so inoverwritepolicies are updated and not always appendedEND_COMMIT_OVERRIDE