-
Notifications
You must be signed in to change notification settings - Fork 83
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
added an ID index and delete old view rows #919
Conversation
pipelines/batch/src/main/java/com/google/fhir/analytics/JdbcResourceWriter.java
Show resolved
Hide resolved
pipelines/batch/src/main/java/com/google/fhir/analytics/JdbcResourceWriter.java
Show resolved
Hide resolved
pipelines/batch/src/main/java/com/google/fhir/analytics/JdbcResourceWriter.java
Show resolved
Hide resolved
pipelines/batch/src/main/java/com/google/fhir/analytics/JdbcResourceWriter.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the changes @bashir2, there are few only few comments which are not that critical. I am approving the changes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @chandrashekar-s for the review; I marked some of your comments to address in a follow-up PR [soon].
Description of what I changed
Adds an index on the
id
columns (and makes that column mandatory) and deletes record with the same resource IDs before adding them to the DB. This is required for #916 to be able to update old view rows.E2E test
TESTED:
Ran the pipeline for both small and large datasets (~16M observations). The impact of the index/deletion on performance seems to be ~3% or less (depending on various scenarios tested).
Checklist: I completed these to help reviewers :)
I have read and will follow the review process.
I am familiar with Google Style Guides for the language I have coded in.
No? Please take some time and review Java and Python style guides.
My IDE is configured to follow the Google code styles.
No? Unsure? -> configure your IDE.
I have added tests to cover my changes. (If you refactored existing code that was well tested you do not have to add tests)
I ran
mvn clean package
right before creating this pull request and added all formatting changes to my commit.All new and existing tests passed.
My pull request is based on the latest changes of the master branch.
No? Unsure? -> execute command
git pull --rebase upstream master