Updating detection filters. #10

swhite-oreilly · 2023-08-17T14:18:46Z

Description

Adding Completed to the job status types that are ignored upon cleanup for all Comprehend job types.

Adding support for the following job types: ComprehendEventsDetectionJob, ComprehendPiiEntititesDetectionJob, and ComprehendTargetedSentimentDetectionJob.

Note

document-classification-job and topics-detection-job do not have stop functions in the CLI nor Boto3. As any personal identifying data is deleted on S3 cleanup, and these jobs technically don't store anything but metadata links to the files in S3 and naming/tags, the team has decided support for these can be left in and it isn't a hard requirement to add support for these at this time.

Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.295 to 1.44.307. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Commits](aws/aws-sdk-go@v1.44.295...v1.44.307) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.307 to 1.44.313. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Commits](aws/aws-sdk-go@v1.44.307...v1.44.313) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Support for removing TGW Attachment Peering resources, as only VPC attachments where supported before. Co-authored-by: Björn Häuser <b.haeuser@rebuy.com>

…de#1046) * kms-keys: Skip keys already in pending replica deletion state Multi-region KMS keys enter state KeyStatePendingReplicaDeletion when deleted, they should be filtered out in the list operation. * Filter out PendingReplicaDeletion in the filter state rather than list state

…ebuy-de#936) Co-authored-by: Björn Häuser <b.haeuser@rebuy.com>

Co-authored-by: Björn Häuser <b.haeuser@rebuy.com>

* Add formatting check to GitHub Actions * Fix pre-existing formatting issues

Signed-off-by: Taylor Barrella <tbarrella@gmail.com> Co-authored-by: Björn Häuser <b.haeuser@rebuy.com>

Signed-off-by: Taylor Barrella <tbarrella@gmail.com>

* Adding CloudFrontOriginRequestPolicy resources. --------- Co-authored-by: Philipp Trulson <der-eismann@users.noreply.github.com>

Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.313 to 1.44.318. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Commits](aws/aws-sdk-go@v1.44.313...v1.44.318) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

> This is an automatically generated PR. @rebuy-de/ FYI

* Filter main route table * use `Filter` instead of custom method

Bumps the golang group with 1 update: [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go). - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Commits](aws/aws-sdk-go@v1.44.318...v1.44.323) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-patch dependency-group: golang ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: der-eismann <der-eismann@users.noreply.github.com>

Adding `Completed` to the job status types that are ignored upon cleanup.

gsoria

These changes make sense to me. I've tried it locally and I see that the completed SentimentDetectionJob and EntitiesDetectionJob jobs were skipped.

main

us-east-1 - ComprehendEntityRecognizer - arn:aws:comprehend:us-east-1:318480998194:entity-recognizer/test-3241623422e062a1c5f9b9199432bff80d4109be - [EntityRecognizerArn: "arn:aws:comprehend:us-east-1:318480998194:entity-recognizer/test-3241623422e062a1c5f9b9199432bff80d4109be", LanguageCode: "en"] - would remove
us-east-1 - ComprehendDocumentClassifier - arn:aws:comprehend:us-east-1:318480998194:document-classifier/testDelete-3241623422e062a1c5f9b9199432bff80d4109be - [DocumentClassifierArn: "arn:aws:comprehend:us-east-1:318480998194:document-classifier/testDelete-3241623422e062a1c5f9b9199432bff80d4109be", LanguageCode: "en"] - would remove
us-east-1 - ComprehendSentimentDetectionJob - sentiment-analysis-3241623422e062a1c5f9b9199432bff80d4109be - [JobId: "c08aabeafd134afef71126260046ada6", JobName: "sentiment-analysis-3241623422e062a1c5f9b9199432bff80d4109be"] - would remove
us-east-1 - ComprehendEntitiesDetectionJob - entities-analysis-3241623422e062a1c5f9b9199432bff80d4109be - [JobId: "d7f4050116539b3ac90baf530c22807e", JobName: "entities-analysis-3241623422e062a1c5f9b9199432bff80d4109be"] - would remove
Scan complete: 4 total, 4 nukeable, 0 filtered.

this branch

us-east-1 - ComprehendEntityRecognizer - arn:aws:comprehend:us-east-1:318480998194:entity-recognizer/test-3241623422e062a1c5f9b9199432bff80d4109be - [EntityRecognizerArn: "arn:aws:comprehend:us-east-1:318480998194:entity-recognizer/test-3241623422e062a1c5f9b9199432bff80d4109be", LanguageCode: "en"] - would remove
us-east-1 - ComprehendDocumentClassifier - arn:aws:comprehend:us-east-1:318480998194:document-classifier/testDelete-3241623422e062a1c5f9b9199432bff80d4109be - [DocumentClassifierArn: "arn:aws:comprehend:us-east-1:318480998194:document-classifier/testDelete-3241623422e062a1c5f9b9199432bff80d4109be", LanguageCode: "en"] - would remove
Scan complete: 2 total, 2 nukeable, 0 filtered.

I see there are other resources not supported by aws-nuke nor cloud control API:

document-classification-job
document-classifier-endpoint
entity-recognizer-endpoint
events-detection-job
pii-entities-detection-job
targeted-sentiment-detection-job
topics-detection-job

Are these going to be covered as part of your changes?

corybekk

Just to make sure I am understanding the change, aws-nuke was trying to delete resources that were in the "completed" state? i.e resources that have already been deleted?

swhite-oreilly · 2023-08-21T23:30:24Z

Just to make sure I am understanding the change, aws-nuke was trying to delete resources that were in the "completed" state? i.e resources that have already been deleted?

@corybekk These don't actually get deleted as they are just metadata pointing to the trained files in S3. The original function goal was to stop in progress jobs, but they missed filtering out those in the Completed status so they were causing errors because they were in the wrong state.

swhite-oreilly · 2023-08-21T23:31:03Z

These changes make sense to me. I've tried it locally and I see that the completed SentimentDetectionJob and EntitiesDetectionJob jobs were skipped.

main

us-east-1 - ComprehendEntityRecognizer - arn:aws:comprehend:us-east-1:318480998194:entity-recognizer/test-3241623422e062a1c5f9b9199432bff80d4109be - [EntityRecognizerArn: "arn:aws:comprehend:us-east-1:318480998194:entity-recognizer/test-3241623422e062a1c5f9b9199432bff80d4109be", LanguageCode: "en"] - would remove
us-east-1 - ComprehendDocumentClassifier - arn:aws:comprehend:us-east-1:318480998194:document-classifier/testDelete-3241623422e062a1c5f9b9199432bff80d4109be - [DocumentClassifierArn: "arn:aws:comprehend:us-east-1:318480998194:document-classifier/testDelete-3241623422e062a1c5f9b9199432bff80d4109be", LanguageCode: "en"] - would remove
us-east-1 - ComprehendSentimentDetectionJob - sentiment-analysis-3241623422e062a1c5f9b9199432bff80d4109be - [JobId: "c08aabeafd134afef71126260046ada6", JobName: "sentiment-analysis-3241623422e062a1c5f9b9199432bff80d4109be"] - would remove
us-east-1 - ComprehendEntitiesDetectionJob - entities-analysis-3241623422e062a1c5f9b9199432bff80d4109be - [JobId: "d7f4050116539b3ac90baf530c22807e", JobName: "entities-analysis-3241623422e062a1c5f9b9199432bff80d4109be"] - would remove
Scan complete: 4 total, 4 nukeable, 0 filtered.

this branch

us-east-1 - ComprehendEntityRecognizer - arn:aws:comprehend:us-east-1:318480998194:entity-recognizer/test-3241623422e062a1c5f9b9199432bff80d4109be - [EntityRecognizerArn: "arn:aws:comprehend:us-east-1:318480998194:entity-recognizer/test-3241623422e062a1c5f9b9199432bff80d4109be", LanguageCode: "en"] - would remove
us-east-1 - ComprehendDocumentClassifier - arn:aws:comprehend:us-east-1:318480998194:document-classifier/testDelete-3241623422e062a1c5f9b9199432bff80d4109be - [DocumentClassifierArn: "arn:aws:comprehend:us-east-1:318480998194:document-classifier/testDelete-3241623422e062a1c5f9b9199432bff80d4109be", LanguageCode: "en"] - would remove
Scan complete: 2 total, 2 nukeable, 0 filtered.

I see there are other resources not supported by aws-nuke nor cloud control API:

* document-classification-job

* document-classifier-endpoint

* entity-recognizer-endpoint

* events-detection-job

* pii-entities-detection-job

* targeted-sentiment-detection-job

* topics-detection-job

Are these going to be covered as part of your changes?

@gsoria added support for all but document-classification-job or topics-detection-job. Waiting on team response in slack.

corybekk · 2023-08-21T23:37:34Z

Just to make sure I am understanding the change, aws-nuke was trying to delete resources that were in the "completed" state? i.e resources that have already been deleted?

@corybekk These don't actually get deleted as they are just metadata pointing to the trained files in S3. The original function goal was to stop in progress jobs, but they missed filtering out those in the Completed status so they were causing errors because they were in the wrong state.

thanks for the clarification!

Bumps the golang group with 2 updates: [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) and [github.com/google/uuid](https://github.com/google/uuid). Updates `github.com/aws/aws-sdk-go` from 1.44.323 to 1.44.328 - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Commits](aws/aws-sdk-go@v1.44.323...v1.44.328) Updates `github.com/google/uuid` from 1.3.0 to 1.3.1 - [Release notes](https://github.com/google/uuid/releases) - [Changelog](https://github.com/google/uuid/blob/master/CHANGELOG.md) - [Commits](google/uuid@v1.3.0...v1.3.1) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-patch dependency-group: golang - dependency-name: github.com/google/uuid dependency-type: direct:production update-type: version-update:semver-patch dependency-group: golang ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Adding filters for key-phrases and dominant-language

Co-authored-by: der-eismann <der-eismann@users.noreply.github.com>

* Adding Elasticache User and UserGroup Support Adding go modules for elasticache users and groups. Adding filtering for subnet groups to ignore the default elasticache subnet group. * Create opensearchservice-packages.go Adding working code for packages cleanup. * Delete opensearchservice-packages.go Moving opensearch changes to separate branch. * Updating elasticache user/group list calls with pagination. * Reverting versions to match oreilly-main Reverting versions to match oreilly-main * Updating go version to match upstream. * Updating to more closely match style of other resource types. * Adding properties to EC user/usergroups.

…uy-de#873) (rebuy-de#1065)

feat: MemoryDBCluster resource support feat: MemoryDBParameterGroup resource support feat: MemoryDBSubnetGroup resource support feat: MemoryDBUser resource support

…uy-de#1053) * Added Deletion protection disable feature in cognito user-pool * Minor Changes * nit * spaces * spaces 2 * stopping executions * Stopped executiins * nit * Formatted --------- Co-authored-by: Suleman Sohail <Suleman@Suleman-Sohail-SWE.local> Co-authored-by: Philipp Trulson <der-eismann@users.noreply.github.com>

* Add RedshiftScheduledAction resource --------- Co-authored-by: Björn Häuser <b.haeuser@rebuy.com>

gsoria

Nice updates! I've tried it locally and it correctly cleanup the resources. Initially the DocumentClassifier failed, but eventually was cleanup up as well.

us-east-1 - ComprehendDocumentClassifier - arn:aws:comprehend:us-east-1:430037344291:document-classifier/my-doc-classifier-21ef490ae3601ac28f88 - [DocumentClassifierArn: "arn:aws:comprehend:us-east-1:430037344291:document-classifier/my-doc-classifier-21ef490ae3601ac28f88", LanguageCode: "en"] - failed
...

us-east-1 - ComprehendDocumentClassifier - arn:aws:comprehend:us-east-1:430037344291:document-classifier/my-doc-classifier-21ef490ae3601ac28f88 - [DocumentClassifierArn: "arn:aws:comprehend:us-east-1:430037344291:document-classifier/my-doc-classifier-21ef490ae3601ac28f88", LanguageCode: "en"] - removed

Removal requested: 0 waiting, 0 failed, 0 skipped, 9 finished

Don't forget to sync with the upstream repo before merging this PR 🙂

…etection-filters Bring upstream changes from rebuy.

dependabot bot and others added 19 commits July 25, 2023 10:26

Support for TGW Attachment Peering resources (rebuy-de#996)

529f885

Support for removing TGW Attachment Peering resources, as only VPC attachments where supported before. Co-authored-by: Björn Häuser <b.haeuser@rebuy.com>

Enable aws-cn by setting the default partition correctly (rebuy-de#1042)

06a0dfe

feat: Adding tag properties to EC2DefaultSecurityGroupRule resource (r…

8ecf7cc

…ebuy-de#936) Co-authored-by: Björn Häuser <b.haeuser@rebuy.com>

Fix date format for ASG creationDate (rebuy-de#1014)

34899fb

Co-authored-by: Björn Häuser <b.haeuser@rebuy.com>

Add formatting check to GitHub Actions (rebuy-de#1063)

47fc6c7

* Add formatting check to GitHub Actions * Fix pre-existing formatting issues

appconfig: add support for five resources (rebuy-de#1055)

2bef99a

Signed-off-by: Taylor Barrella <tbarrella@gmail.com> Co-authored-by: Björn Häuser <b.haeuser@rebuy.com>

ecs: list all services, not just 10 (rebuy-de#1056)

90ad6c4

Signed-off-by: Taylor Barrella <tbarrella@gmail.com>

Adding CloudFrontOriginRequestPolicy resources. (rebuy-de#1051)

d01537a

* Adding CloudFrontOriginRequestPolicy resources. --------- Co-authored-by: Philipp Trulson <der-eismann@users.noreply.github.com>

Group Golang dependabot updates (rebuy-de#1070)

42a68a3

> This is an automatically generated PR. @rebuy-de/ FYI

Filter main route tables (rebuy-de#953)

df499d5

* Filter main route table * use `Filter` instead of custom method

Update readme for v2.24.0 release (rebuy-de#1072)

96d3e60

Co-authored-by: der-eismann <der-eismann@users.noreply.github.com>

app config: fix max results 100 -> 50 (rebuy-de#1075)

b873fb3

Update readme for v2.24.1 release (rebuy-de#1076)

f4030b9

Co-authored-by: der-eismann <der-eismann@users.noreply.github.com>

Updating detection filters.

b044caf

Adding `Completed` to the job status types that are ignored upon cleanup.

swhite-oreilly changed the base branch from main to oreilly-main August 18, 2023 19:25

gsoria reviewed Aug 18, 2023

View reviewed changes

corybekk approved these changes Aug 21, 2023

View reviewed changes

dependabot bot and others added 5 commits August 22, 2023 10:56

Adding filters for key-phrases and dominant-language

db11768

Adding filters for key-phrases and dominant-language

Adding pii entitites and sentiment detection job support.

7b51b7f

Adding events detection job support.

eeb2e17

Fix: Don't delete predefined deployment strategies (rebuy-de#1081)

88607f3

der-eismann and others added 10 commits August 23, 2023 17:39

Stop building ARM containers for PRs (rebuy-de#1082)

468e5e3

Defuse unnecessary errors for unavailable services (rebuy-de#1083)

158ca36

Build with Go 1.21 (rebuy-de#1084)

ef87e80

Update readme for v2.24.2 release (rebuy-de#1086)

a3fa4aa

Co-authored-by: der-eismann <der-eismann@users.noreply.github.com>

Add EC2InstanceConnectEndpoint resource (rebuy-de#1087)

a552489

Add support for CloudFront public keys and CloudFront key groups (reb…

f8fc2e5

…uy-de#873) (rebuy-de#1065)

feat: MemoryDBACL resource support (rebuy-de#1079)

f1aff70

feat: MemoryDBCluster resource support feat: MemoryDBParameterGroup resource support feat: MemoryDBSubnetGroup resource support feat: MemoryDBUser resource support

Add RedshiftScheduledAction resource (rebuy-de#1047)

150eb13

* Add RedshiftScheduledAction resource --------- Co-authored-by: Björn Häuser <b.haeuser@rebuy.com>

gsoria approved these changes Aug 24, 2023

View reviewed changes

swhite-oreilly and others added 2 commits August 24, 2023 11:35

Merge remote-tracking branch 'upstream/main' into Update-comprehend-d…

dd94ae0

…etection-filters Bring upstream changes from rebuy.

Merge branch 'oreilly-main' into Update-comprehend-detection-filters

b598367

swhite-oreilly merged commit ea07a10 into oreilly-main Aug 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Updating detection filters. #10

Updating detection filters. #10

Uh oh!

swhite-oreilly commented Aug 17, 2023 •

edited

Loading

Uh oh!

gsoria left a comment

Uh oh!

corybekk left a comment

Uh oh!

swhite-oreilly commented Aug 21, 2023

Uh oh!

swhite-oreilly commented Aug 21, 2023 •

edited

Loading

Uh oh!

corybekk commented Aug 21, 2023

Uh oh!

gsoria left a comment

Uh oh!

Uh oh!

Updating detection filters. #10

Updating detection filters. #10

Uh oh!

Conversation

swhite-oreilly commented Aug 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Note

Uh oh!

gsoria left a comment

Choose a reason for hiding this comment

Uh oh!

corybekk left a comment

Choose a reason for hiding this comment

Uh oh!

swhite-oreilly commented Aug 21, 2023

Uh oh!

swhite-oreilly commented Aug 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

corybekk commented Aug 21, 2023

Uh oh!

gsoria left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

swhite-oreilly commented Aug 17, 2023 •

edited

Loading

swhite-oreilly commented Aug 21, 2023 •

edited

Loading