Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added an ID index and delete old view rows #919

Merged
merged 3 commits into from
Jan 16, 2024
Merged

Conversation

bashir2
Copy link
Collaborator

@bashir2 bashir2 commented Jan 15, 2024

Description of what I changed

Adds an index on the id columns (and makes that column mandatory) and deletes record with the same resource IDs before adding them to the DB. This is required for #916 to be able to update old view rows.

E2E test

TESTED:

Ran the pipeline for both small and large datasets (~16M observations). The impact of the index/deletion on performance seems to be ~3% or less (depending on various scenarios tested).

Checklist: I completed these to help reviewers :)

  • I have read and will follow the review process.

  • I am familiar with Google Style Guides for the language I have coded in.

    No? Please take some time and review Java and Python style guides.

  • My IDE is configured to follow the Google code styles.

    No? Unsure? -> configure your IDE.

  • I have added tests to cover my changes. (If you refactored existing code that was well tested you do not have to add tests)

  • I ran mvn clean package right before creating this pull request and added all formatting changes to my commit.

  • All new and existing tests passed.

  • My pull request is based on the latest changes of the master branch.

    No? Unsure? -> execute command git pull --rebase upstream master

Copy link
Collaborator

@chandrashekar-s chandrashekar-s left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes @bashir2, there are few only few comments which are not that critical. I am approving the changes.

Copy link
Collaborator Author

@bashir2 bashir2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @chandrashekar-s for the review; I marked some of your comments to address in a follow-up PR [soon].

@bashir2 bashir2 merged commit 06267bc into google:master Jan 16, 2024
5 checks passed
@bashir2 bashir2 deleted the viewsDB branch January 18, 2024 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants