Adding the incremental entity provider backend #14356

dekoding · 2022-10-26T19:50:29Z

Signed-off-by: Damon Kaswell damon.kaswell1@hp.com

Hey, I just made a Pull Request!

This PR introduces the concept of incremental entity providers to Backstage. An incremental entity provider is a form of entity provider that can ingest from very large data sources where a full mutation to achieve initial ingestion may be impossible.

Incremental providers split a large ingestion task into smaller chunks which can be streamed into the catalog (using deltas under the hood). The size and length of these chunks are configurable, as are the lengths of the pauses between them and the amount of time to rest after ingestion is complete before starting a fresh run.

The incremental entity provider tracks a cursor to indicate its current location in the data stream. This allows the provider's work to pick up where it leaves off after a burst of activity, on any replica running the provider's task.

In order to accomplish this, the incremental provider introduces a new schema called ingestion to the Postgres database, with three tables:

ingestions - Contains the status record for an incremental entity provider. Over time, there will always be two records per provider: One for the previous run, and one for the current run. Older ingestion records are purged automatically.
ingestion_marks - Contains the cursor records used by the incremental entity provider. They have a many-to-one relationship with the ingestions table records. Ingestion marks for old ingestion records are purged automatically as well.
ingestion_mark_entities - Contains an entry for each discovered entity, with a many-to-one relationship with records from the ingestion_marks table. Again, older records are purged automatically. This table is used for comparisons to the final_entities table, to allow entities for assets that have been removed to be deleted.

It's important to note that due to the fact that any replica of Backstage could pick up the task of ingestion, stateful protocols such as LDAP will not work, and should be ingested either via a proxy or with a standard entity provider.

✔️ Checklist

A changeset describing the change and affected packages. (more info)
Added or updated documentation
Tests for new functionality and regression tests for bug fixes
Screenshots attached (for UI changes)
All your commits have a Signed-off-by line in the message. (more info)

github-actions · 2022-10-26T19:50:50Z

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage/plugin-catalog-backend-module-incremental-ingestion	plugins/catalog-backend-module-incremental-ingestion	minor	`v0.0.0`

plugins/incremental-ingestion-backend/README.md

taras

Awesome work. I added a few comments for the readme.

freben · 2022-10-27T09:27:59Z

Whoa! Nice :) Review to be had later on. Regarding the DI error, note that all our in-repo dependencies (to other packages/plugins packages) are on the form "workspace:^" now, instead of actual versions. Way more convenient, I'm sure you'll find. I think that should let the compilation progress more.

dekoding · 2022-10-27T15:23:29Z

Whoa! Nice :) Review to be had later on. Regarding the DI error, note that all our in-repo dependencies (to other packages/plugins packages) are on the form "workspace:^" now, instead of actual versions. Way more convenient, I'm sure you'll find. I think that should let the compilation progress more.

Thanks for the heads-up! I also need to trim some other dependencies that are unneeded in this version.

plugins/incremental-ingestion-backend/src/database/migrations.ts

plugins/incremental-ingestion-backend/src/engine/IncrementalIngestionEngine.ts

freben

Alright! Sorry it took some time, we were away and I also spent some real focus time on this :) I hope the amount of comments doesn't feel overwhelming. Let me know if you prefer some form of arrangement with an eager merge, with the most important things addressed and deferring the rest, or so

packages/backend/package.json

plugins/incremental-ingestion-backend/package.json

plugins/incremental-ingestion-backend/src/service/IncrementalCatalogBuilder.ts

plugins/incremental-ingestion-backend/README.md

freben · 2022-11-08T13:42:28Z

plugins/incremental-ingestion-backend/src/database/IncrementalIngestionDatabaseManager.ts

+} from '../types';
+
+/** @public */
+export class IncrementalIngestionDatabaseManager {


There are no tests for this class (might be good for at least specifically this one since it might be the riskier one in future refactors)

I'll look into writing one. Is this a showstopper?

Nah. Would be good to get them in eventually of course.

plugins/incremental-ingestion-backend/src/database/IncrementalIngestionDatabaseManager.ts

freben · 2022-11-08T13:44:25Z

plugins/incremental-ingestion-backend/src/database/IncrementalIngestionDatabaseManager.ts

+
+      const ingestionsDeleted = await tx('ingestion.ingestions')
+        .delete()
+        .where('provider_name', provider);


yeah a lot of the cleanup would go away if you had on cascade delete foreign relations as suggested in the migration

I couldn't find a way to do this without losing the information about how many of each type of record was deleted. I'm still trying to decide if I actually care about that data, though.

plugins/incremental-ingestion-backend/src/database/IncrementalIngestionDatabaseManager.ts

freben · 2022-11-08T13:46:33Z

plugins/incremental-ingestion-backend/src/database/IncrementalIngestionDatabaseManager.ts

+  }
+
+  /**
+   * Performs a lookup of all providers that have duplicate active ingestion records.


Should there be a UNIQUE constraint on the table instead?

This method is left over from before we added the unique constraint that is present in the current migration. I'll have to think about whether this is still a concern.

github-actions · 2022-11-17T08:35:18Z

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

freben · 2022-11-22T12:02:30Z

Just closing and opening to try to nudge the builds

freben

Alright, some followup. Do note that I pushed a PR to strip out all of the database related code from the API report; you want to pull this branch before you continue work on it. I hope that's not a problem. 🙏

freben · 2022-11-22T10:05:04Z

plugins/incremental-ingestion-backend/README.md

+  // before incremental builder migrations are executed
+  const { incrementalAdminRouter } = await incrementalBuilder.build();
+
+  router.use('/incremental', incrementalAdminRouter);


Alright, let's leave it as is then for now. We will want to write a module for the new backend system for this eventually, and at that point I think we'll discover that some similar deferral mechanism as suggested above might come into play under the hood too. So we can worry about that then.

freben · 2022-11-22T10:05:46Z

plugins/incremental-ingestion-backend/src/database/IncrementalIngestionDatabaseManager.ts

+} from '../types';
+
+/** @public */
+export class IncrementalIngestionDatabaseManager {


Nah. Would be good to get them in eventually of course.

plugins/incremental-ingestion-backend/api-report.md

plugins/incremental-ingestion-backend/src/database/IncrementalIngestionDatabaseManager.ts

plugins/incremental-ingestion-backend/src/engine/IncrementalIngestionEngine.ts

plugins/incremental-ingestion-backend/src/router/paths.ts

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Signed-off-by: Fredrik Adelöw <freben@gmail.com>

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Signed-off-by: Fredrik Adelöw <freben@gmail.com>

packages/cli/src/commands/start/command.ts

Signed-off-by: Fredrik Adelöw <freben@gmail.com>

freben

Alright, I think this is in a good shape now to merge and iterate on!

One large remaining note that I'd like to have addressed in a follow-up, is that it directly accesses catalog tables. That should ideally not be necessary. Since we already know from our own tables what the previous ingestion round returned, we should consult with that to compute the deleted items. The catalog, especially since it goes via the search and final_entities tables, is not a good source of truth since there may be problems and delays related to processing and stitching meaning that the final state of the db isn't what you think it is, next time that the burst starts over again.

dekoding requested review from a team and backstage-service as code owners October 26, 2022 19:50

backstage-goalie bot assigned freben Oct 26, 2022

taras reviewed Oct 27, 2022

View reviewed changes

plugins/incremental-ingestion-backend/README.md Outdated Show resolved Hide resolved

taras reviewed Oct 27, 2022

View reviewed changes

plugins/incremental-ingestion-backend/README.md Outdated Show resolved Hide resolved

taras reviewed Oct 27, 2022

View reviewed changes

plugins/incremental-ingestion-backend/README.md Outdated Show resolved Hide resolved

taras suggested changes Oct 27, 2022

View reviewed changes

dagda1 mentioned this pull request Nov 1, 2022

sync @frontside/backstage-plugin-incremental-ingestion-backend with backstage PR thefrontside/playhouse#139

Merged

taras reviewed Nov 3, 2022

View reviewed changes

plugins/incremental-ingestion-backend/src/database/migrations.ts Outdated Show resolved Hide resolved

dagda1 reviewed Nov 4, 2022

View reviewed changes

plugins/incremental-ingestion-backend/src/engine/IncrementalIngestionEngine.ts Outdated Show resolved Hide resolved

freben reviewed Nov 8, 2022

View reviewed changes

freben added the area:catalog Related to the Catalog Project Area label Nov 10, 2022

github-actions bot added the stale label Nov 17, 2022

freben removed the stale label Nov 17, 2022

taras mentioned this pull request Nov 17, 2022

📁 Catalog Improvement Meta Issue #14574

Open

31 tasks

github-actions bot added area:catalog Related to the Catalog Project Area and removed area:catalog Related to the Catalog Project Area labels Nov 17, 2022

freben force-pushed the dekoding/incremental-entity-provider branch 3 times, most recently from d74979c to 227e882 Compare November 22, 2022 11:13

github-actions bot removed the area:catalog Related to the Catalog Project Area label Nov 22, 2022

freben closed this Nov 22, 2022

freben reopened this Nov 22, 2022

freben reviewed Nov 22, 2022

View reviewed changes

dekoding and others added 18 commits November 23, 2022 15:05

Handle error edge cases

21f0ae7

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Remove from package.json

261e73c

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Code hygiene and cleanup

d0e85ef

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Fix defect in DB manager identified internally

8a23545

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Consolidate migrations and use default schema

6608e08

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Fix migrations

58b627c

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Some type cleanup

5b850ac

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Updated API report

25da47e

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Use foreign key constraints

b6d0c26

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Fixes for migrations and routes

d6dde46

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Refine routes

1a3ae5a

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Prettier formatting

6404ad1

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

shave down the api report, some final fixes

35455be

Signed-off-by: Fredrik Adelöw <freben@gmail.com>

Use stringifyerror

87cabfe

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Update yarn.lock

e426002

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Group router paths in a less risky way

de022e5

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

Fix removal logic

7deebb9

Signed-off-by: Damon Kaswell <damon.kaswell1@hp.com>

rename to be a catalog backend module, implement backend system module

3f5d620

Signed-off-by: Fredrik Adelöw <freben@gmail.com>

freben force-pushed the dekoding/incremental-entity-provider branch from de21a79 to 3f5d620 Compare November 23, 2022 14:09

github-actions bot added the area:catalog Related to the Catalog Project Area label Nov 23, 2022

add module tests

ed48b42

Signed-off-by: Fredrik Adelöw <freben@gmail.com>

Rugvip reviewed Nov 24, 2022

View reviewed changes

packages/cli/src/commands/start/command.ts Outdated Show resolved Hide resolved

freben force-pushed the dekoding/incremental-entity-provider branch from b198472 to 72b1847 Compare November 24, 2022 14:10

add dev app, more testing

d7209f1

Signed-off-by: Fredrik Adelöw <freben@gmail.com>

freben force-pushed the dekoding/incremental-entity-provider branch from 72b1847 to d7209f1 Compare November 24, 2022 14:16

fix racy test

2b8d1d3

Signed-off-by: Fredrik Adelöw <freben@gmail.com>

freben approved these changes Nov 24, 2022

View reviewed changes

freben merged commit 6ecb455 into backstage:master Nov 24, 2022

freben mentioned this pull request Nov 28, 2022

Make incremental entity providers more resilient to failed data sources #14904

Merged

5 tasks

jhaals mentioned this pull request Dec 20, 2022

Version Packages #15218

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding the incremental entity provider backend #14356

Adding the incremental entity provider backend #14356

dekoding commented Oct 26, 2022

github-actions bot commented Oct 26, 2022 •

edited

Loading

taras left a comment

freben commented Oct 27, 2022 •

edited

Loading

dekoding commented Oct 27, 2022

freben left a comment

freben Nov 8, 2022

dekoding Nov 22, 2022

freben Nov 22, 2022

freben Nov 8, 2022

dekoding Nov 21, 2022

freben Nov 8, 2022

dekoding Nov 21, 2022

github-actions bot commented Nov 17, 2022

freben commented Nov 22, 2022

freben left a comment

freben Nov 22, 2022

freben Nov 22, 2022

freben left a comment

Adding the incremental entity provider backend #14356

Adding the incremental entity provider backend #14356

Conversation

dekoding commented Oct 26, 2022

Hey, I just made a Pull Request!

✔️ Checklist

github-actions bot commented Oct 26, 2022 • edited Loading

Changed Packages

taras left a comment

Choose a reason for hiding this comment

freben commented Oct 27, 2022 • edited Loading

dekoding commented Oct 27, 2022

freben left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 17, 2022

freben commented Nov 22, 2022

freben left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freben left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 26, 2022 •

edited

Loading

freben commented Oct 27, 2022 •

edited

Loading