feat: add `AwsS3EntityProvider` as replacement for `AwsS3DiscoveryProcessor` #10480

pjungermann · 2022-03-28T13:33:46Z

Add a new provider AwsS3EntityProvider as a replacement for the now deprecated
AwsS3DiscoveryProcessor.

The new provider will scan configured S3 buckets (with optional) prefix and
add Location entities for all discovered catalog files.

These Location entities will then be processed as usual.

At each execution, the provider will apply a full mutation, replacing all previous
entities with the new entities/state.

Relates-to: #10183
Signed-off-by: Patrick Jungermann Patrick.Jungermann@gmail.com

Hey, I just made a Pull Request!

✔️ Checklist

A changeset describing the change and affected packages. (more info)
Added or updated documentation
Tests for new functionality and regression tests for bug fixes
Screenshots attached (for UI changes)
All your commits have a Signed-off-by line in the message. (more info)

github-actions · 2022-03-28T13:34:11Z

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage/plugin-catalog-backend-module-aws	plugins/catalog-backend-module-aws	patch	`v0.1.4-next.1`

jhaals

Nice work Patrick! 🙌 I've added some suggestions :)

.changeset/friendly-hairs-happen.md

plugins/catalog-backend-module-aws/api-report.md

pjungermann · 2022-03-29T07:44:54Z

I rebased on master since PR #10454 was merged to make it a bit more clean here

plugins/catalog-backend-module-aws/src/credentials/AwsCredentials.ts

benjdlambert · 2022-03-31T10:53:50Z

Hmm just wondering if we should skip the deprecation of the older stuff for now, just until we can confirm that the provider API's that we've settled on are right and do everything that we want before losing support for things that might have been supported in the Processor approach? Does that make sense for now at least?

pjungermann · 2022-03-31T11:05:43Z

Hmm just wondering if we should skip the deprecation of the older stuff for now, just until we can confirm that the provider API's that we've settled on are right and do everything that we want before losing support for things that might have been supported in the Processor approach? Does that make sense for now at least?

@benjdlambert the provider does the same thing as the processor -- just that changes will not cause orphans. Also, deprecating it will not remove it yet. This breaking change could come at a later step.

But of course, deprecating it suggests users to migrate.

At the moment, I don't see a value of the processor anymore though. But maybe I miss something.

If you prefer to remove the deprecation it can be done, of course. Not sure about the docs then though. I would find it a bit confusing to have both there (as user).

benjdlambert · 2022-03-31T11:17:23Z

Yeah sorry, I didn't mean in your implementation, I just meant from our side let's not get people to migrate until we're settled on Providers for the catalog. Maybe you could break the documentation up into Recommended and Other? @freben thoughts here on what we should be encouraging?

pjungermann · 2022-03-31T14:25:19Z

@benjdlambert as a user, I would prefer providers due to the orphans created when using processors.

Also a reason why I contributed this one and plan to contribute a few more.

As next level, I would see event-based updates vs those (full) syncs. A combination like initial full sync + event-based updates + full syncs once in a while as auto-healing would work fine.

For event-based updates, webhook-based change events as from SCM systems or other evens like S3 events can be interesting, maybe buffered in a queue; so general JMS support might make sense. Inside backstage, this could be solved with adaptors for the different providers converting events into a unified event schema or triggering APIs directly.
I know there are a couple of thoughts floating around for this and all these need to go through an RFC first.
Just for context.

pjungermann · 2022-03-31T14:26:49Z

processors are still great to enhance/modify entities. Ingestion I would leave to providers.

docs/integrations/aws-s3/discovery.md

freben

Man, it's getting way too late, some early comments before reading through fully

.changeset/friendly-hairs-happen.md

plugins/catalog-backend-module-aws/src/providers/AwsS3EntityProviderFactory.ts

docs/integrations/aws-s3/discovery.md

plugins/catalog-backend-module-aws/config.d.ts

plugins/catalog-backend-module-aws/src/providers/AwsS3EntityProvider.ts

plugins/catalog-backend-module-aws/src/providers/AwsS3EntityProviderFactory.ts

plugins/catalog-backend-module-aws/src/providers/AwsS3EntityProvider.ts

pjungermann · 2022-04-05T23:32:19Z

just added the option to omit the provider ID in the config for the single config use case using "default" as ID.

# app-config.yaml

catalog:
  providers:
    awsS3:
      # uses "default" as provider ID
      bucketName: sample-bucket
      prefix: prefix/ # optional
      region: us-east-2 # optional, uses the default region otherwise

plugins/catalog-backend-module-aws/api-report.md

…cessor` Add a new provider `AwsS3EntityProvider` as a replacement for the now deprecated `AwsS3DiscoveryProcessor`. The new provider will scan configured S3 buckets (with optional) prefix and add `Location` entities for all discovered catalog files. These `Location` entities will then be processed as usual. At each execution, the provider will apply a full mutation, replacing all previous entities with the new entities/state. Relates-to: backstage#10183 Signed-off-by: Patrick Jungermann <Patrick.Jungermann@gmail.com>

pjungermann · 2022-04-07T08:34:33Z

@Rugvip @freben I believe it is ready to be merged :-)

freben

Let's merge and tweak!

freben · 2022-04-07T12:22:00Z

plugins/catalog-backend-module-aws/src/providers/AwsS3EntityProvider.ts

+    // and the default integration will be added as second integration.
+    // In this case, we still want the first one though, but have no means to select it
+    // just from the bucket name (and region).
+    const integration = ScmIntegrations.fromConfig(configRoot).awsS3.list()[0];


thought for the future: this actually can never be undefined at the current point in time, but if we ever changed the behavior to not add a default entry silently, it might have been wise to be defensive here and throw an error if integration turned out to be undefined

good point. I can create a PR to add this.

follow-up PR #10697

freben · 2022-04-07T12:26:39Z

plugins/catalog-backend-module-aws/src/providers/AwsS3EntityProvider.ts

+      return `${endpoint}/${bucketName}/${key}`;
+    }
+
+    return `https://${bucketName}.s3.${this.config.region}.amazonaws.com/${key}`;


we would have to check if the target was really an amazon one right?

We already check these cases in the lines before. If we don't point to the real AWS S3, we need to provide the endpoint config property (e.g., when LocalStack is used, etc.). So, I'd say it is already covered.

Relates-to: PR backstage#10480 Signed-off-by: Patrick Jungermann <Patrick.Jungermann@gmail.com>

pjungermann · 2022-04-27T19:02:29Z

I just realized that applying the "allow" catalog rule to locations emitted by an entity provider only supports the global config.

For the awsS3 processor, this was not really the case as the processor's Location can have a the Location-specific "allow" rule and itself only emits the entities contained within the S3 objects/catalog files. Hence, the rule was applied successfully.

For other processors which emit Locations as well like the SCM-related processors for github, gitlab, bitbucket, etc. should be impacted by this limitation as well.

In order to mitigate this, you need to ensure that to be added entities are allowed by the global rule.

Not yet sure what would be the right strategy to have a custom configuration for the provider (instance) and be more restrictive at the global rule. 🤔

pjungermann · 2022-07-28T14:41:14Z

issue #12880 as follow up related to the "allow" rule

pjungermann requested a review from a team as a code owner March 28, 2022 13:33

github-actions bot added awaiting-review area:catalog Related to the Catalog Project Area documentation Improvements or additions to documentation labels Mar 28, 2022

pjungermann force-pushed the PJ_awsS3-provider branch 2 times, most recently from 09399fc to b05d156 Compare March 28, 2022 14:23

jhaals reviewed Mar 29, 2022

View reviewed changes

pjungermann force-pushed the PJ_awsS3-provider branch from b05d156 to 20813e6 Compare March 29, 2022 07:43

pjungermann force-pushed the PJ_awsS3-provider branch from 20813e6 to 760f6fe Compare March 29, 2022 08:06

pjungermann requested a review from jhaals March 29, 2022 09:34

pjungermann force-pushed the PJ_awsS3-provider branch from 760f6fe to 3c45683 Compare March 29, 2022 10:21

backstage-goalie bot removed the awaiting-review label Mar 29, 2022

jhaals reviewed Mar 29, 2022

View reviewed changes

plugins/catalog-backend-module-aws/src/credentials/AwsCredentials.ts Outdated Show resolved Hide resolved

pjungermann force-pushed the PJ_awsS3-provider branch from 3c45683 to 66a129e Compare March 29, 2022 13:33

github-actions bot added the awaiting-review label Mar 29, 2022

pjungermann requested review from jhaals and Rugvip March 29, 2022 13:34

pjungermann force-pushed the PJ_awsS3-provider branch from 66a129e to bf341ab Compare March 30, 2022 12:38

backstage-goalie bot removed the awaiting-review label Mar 31, 2022

pjungermann commented Mar 31, 2022

View reviewed changes

docs/integrations/aws-s3/discovery.md Show resolved Hide resolved

freben reviewed Mar 31, 2022

View reviewed changes

pjungermann force-pushed the PJ_awsS3-provider branch from bf341ab to 1ffbca6 Compare March 31, 2022 17:16

github-actions bot added the awaiting-review label Mar 31, 2022

Rugvip reviewed Apr 4, 2022

View reviewed changes

plugins/catalog-backend-module-aws/src/providers/AwsS3EntityProviderFactory.ts Outdated Show resolved Hide resolved

plugins/catalog-backend-module-aws/src/providers/AwsS3EntityProvider.ts Outdated Show resolved Hide resolved

pjungermann force-pushed the PJ_awsS3-provider branch from 00d918f to dc6c5cf Compare April 4, 2022 17:12

github-actions bot added the awaiting-review label Apr 4, 2022

pjungermann requested review from freben and Rugvip April 4, 2022 17:22

pjungermann force-pushed the PJ_awsS3-provider branch from dc6c5cf to 5c56632 Compare April 5, 2022 23:30

anicke mentioned this pull request Apr 6, 2022

New plugin: "plugin-catalog-backend-module-gerrit" #10669

Merged

5 tasks

pjungermann commented Apr 6, 2022

View reviewed changes

plugins/catalog-backend-module-aws/api-report.md Outdated Show resolved Hide resolved

pjungermann force-pushed the PJ_awsS3-provider branch from 5c56632 to 09ce01b Compare April 6, 2022 22:06

pjungermann force-pushed the PJ_awsS3-provider branch from 09ce01b to 5969c4b Compare April 7, 2022 06:44

freben approved these changes Apr 7, 2022

View reviewed changes

freben merged commit 9f78f91 into backstage:master Apr 7, 2022

pjungermann deleted the PJ_awsS3-provider branch April 7, 2022 12:29

pjungermann added a commit to Bonial-International-GmbH/backstage that referenced this pull request Apr 7, 2022

feat: handle potential future case of no awsS3 integration

6d67ea6

Relates-to: PR backstage#10480 Signed-off-by: Patrick Jungermann <Patrick.Jungermann@gmail.com>

pjungermann mentioned this pull request Apr 7, 2022

feat: handle potential future case of no awsS3 integration #10697

Merged

5 tasks

pjungermann added a commit to Bonial-International-GmbH/backstage that referenced this pull request Apr 7, 2022

feat: handle potential future case of no awsS3 integration

4daa2fa

Relates-to: PR backstage#10480 Signed-off-by: Patrick Jungermann <Patrick.Jungermann@gmail.com>

pjungermann added a commit to Bonial-International-GmbH/backstage that referenced this pull request Apr 7, 2022

feat: handle potential future case of no awsS3 integration

c07ef1a

Relates-to: PR backstage#10480 Signed-off-by: Patrick Jungermann <Patrick.Jungermann@gmail.com>

pjungermann added a commit to Bonial-International-GmbH/backstage that referenced this pull request Apr 7, 2022

feat: handle potential future case of no awsS3 integration

e604e9d

Relates-to: PR backstage#10480 Signed-off-by: Patrick Jungermann <Patrick.Jungermann@gmail.com>

pjungermann added a commit to Bonial-International-GmbH/backstage that referenced this pull request Apr 8, 2022

feat: handle potential future case of no awsS3 integration

d6faec6

Relates-to: PR backstage#10480 Signed-off-by: Patrick Jungermann <Patrick.Jungermann@gmail.com>

pjungermann added a commit to Bonial-International-GmbH/backstage that referenced this pull request Apr 8, 2022

feat: handle potential future case of no awsS3 integration

3f801d1

Relates-to: PR backstage#10480 Signed-off-by: Patrick Jungermann <Patrick.Jungermann@gmail.com>

goenning mentioned this pull request May 18, 2022

feat: add AzureDevOpsEntityProvider as replacement for AzureDevOpsDiscoveryProcessor #11604

Merged

5 tasks

ivangonzalezacuna mentioned this pull request Jun 7, 2022

feat: add GitlabDiscoveryEntityProvider as replacement for GitlabDiscoveryProcessor #11886

Merged

5 tasks

Rugvip mentioned this pull request Jun 14, 2022

Convert discovery processors into entity providers #10183

Closed

5 tasks

pjungermann mentioned this pull request Jul 28, 2022

feat: add new BitbucketCloudEntityProvider and common/shared client lib #11345

Merged

5 tasks

pjungermann mentioned this pull request Mar 23, 2023

Org Member: pjungermann backstage/community#82

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add `AwsS3EntityProvider` as replacement for `AwsS3DiscoveryProcessor` #10480

feat: add `AwsS3EntityProvider` as replacement for `AwsS3DiscoveryProcessor` #10480

pjungermann commented Mar 28, 2022 •

edited

Loading

github-actions bot commented Mar 28, 2022 •

edited

Loading

jhaals left a comment

pjungermann commented Mar 29, 2022 •

edited

Loading

benjdlambert commented Mar 31, 2022

pjungermann commented Mar 31, 2022

benjdlambert commented Mar 31, 2022

pjungermann commented Mar 31, 2022

pjungermann commented Mar 31, 2022

freben left a comment

pjungermann commented Apr 5, 2022

pjungermann commented Apr 7, 2022

freben left a comment

freben Apr 7, 2022

pjungermann Apr 7, 2022

pjungermann Apr 7, 2022

freben Apr 7, 2022

pjungermann Apr 7, 2022

pjungermann commented Apr 27, 2022

pjungermann commented Jul 28, 2022

feat: add AwsS3EntityProvider as replacement for AwsS3DiscoveryProcessor #10480

feat: add AwsS3EntityProvider as replacement for AwsS3DiscoveryProcessor #10480

Conversation

pjungermann commented Mar 28, 2022 • edited Loading

Hey, I just made a Pull Request!

✔️ Checklist

github-actions bot commented Mar 28, 2022 • edited Loading

Changed Packages

jhaals left a comment

Choose a reason for hiding this comment

pjungermann commented Mar 29, 2022 • edited Loading

benjdlambert commented Mar 31, 2022

pjungermann commented Mar 31, 2022

benjdlambert commented Mar 31, 2022

pjungermann commented Mar 31, 2022

pjungermann commented Mar 31, 2022

freben left a comment

Choose a reason for hiding this comment

pjungermann commented Apr 5, 2022

pjungermann commented Apr 7, 2022

freben left a comment

Choose a reason for hiding this comment

freben Apr 7, 2022

Choose a reason for hiding this comment

pjungermann Apr 7, 2022

Choose a reason for hiding this comment

pjungermann Apr 7, 2022

Choose a reason for hiding this comment

freben Apr 7, 2022

Choose a reason for hiding this comment

pjungermann Apr 7, 2022

Choose a reason for hiding this comment

pjungermann commented Apr 27, 2022

pjungermann commented Jul 28, 2022

feat: add `AwsS3EntityProvider` as replacement for `AwsS3DiscoveryProcessor` #10480

feat: add `AwsS3EntityProvider` as replacement for `AwsS3DiscoveryProcessor` #10480

pjungermann commented Mar 28, 2022 •

edited

Loading

github-actions bot commented Mar 28, 2022 •

edited

Loading

pjungermann commented Mar 29, 2022 •

edited

Loading