Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Indexing improvements #480

Kirill888 opened this issue Jun 7, 2018 · 0 comments


Copy link

commented Jun 7, 2018

This is a meta-issue where we collate various dataset add/update issues raised recently. A lot of those should be addressed together. Even though they might appear as independent concerns, the refactoring needed to address them is best done as one unit of work.

  • Performance
    • #478 processing duplicates
  • Usability
    • #477 spam warnings about lineage data
    • #446 location modifications with "no change needed message"
    • #475 non-atomic add (lineage data might be added)
    • #481 updating lineage dataset metadata shouldn't be a default action
    • #450 auto-product-matching for datasets that already exist in DB shouldn't be needed
    • #451 source_policy confusion
    • #471 light-weight sources metadata (might not be addressed immediately, but will benefit from other changes)

Some of these stem from the way dataset add processes lineage data. It treats lineage datasets in exactly the same way as top-level datasets, as it just performs depth-first traversal of the lineage tree adding datasets that are missing, updating those that already exist.

I would argue that lineage data should be treated differently by dataset add

  • It is expected that lineage datasets are in the DB already, inverse is true for top-level datasets
  • Might want different update/verification behaviours for top-level datasets vs lineage datasets
  • When errors occur user should be able to tell whether the problem is due to top-level dataset or due to lineage dataset. We are not making this clear currently.
@Kirill888 Kirill888 referenced this issue Jun 27, 2018
9 of 9 tasks complete

@omad omad closed this in #485 Jul 13, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
1 participant
You can’t perform that action at this time.