Use elegant way to get all features for a project and fix potential concurrency issues #368
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses a few comments/issues:
search_entities
method, which might not be scalable when there are tons of entities (or might take some time)This PR solves those two issues, by doing the following:
lineage
API to get all the entities belonging to a certain project at once, so that we don't have to do search, rather we rely on the pre-computed purview lineage to get all the entitiesget_features_from_registry()
, we don't use the attributes since there might be concurrency issues; Instead, we useAtlasProcess
to avoid potential concurrency issues.Also fixed an issue brought by #380 where additional schema is added.