Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revise saveConcept method to use save() instead of saveAndFlush() #575

Merged

Conversation

@joelsch
Copy link

@joelsch joelsch commented Mar 1, 2017

Closes #574

Revise saveConcept method to use myConceptDao.save() instead of myConceptDao.saveAndFlush(), to avoid excessive CPU utilization when persisting a large CodeSystem (e.g. one with 399837 concepts).

Commentary: It seems the Hibernate flush operation contains code which performs an O(N^2) operation on the cache, and saveAndFlush causes that to be performed N times. Not good when N is 399837.

I think this change is reasonably safe, because saveConcept is only used within a transaction context. However, this may also benefit from a review by someone more familiar with HAPI's inner workings.

…ptDao.saveAndFlush (to avoid overloading CPU with flush computations when persisting large codesystems, e.g. one with 399837 concepts).
@coveralls
Copy link

@coveralls coveralls commented Mar 1, 2017

Coverage Status

Coverage remained the same at 81.75% when pulling 2f1662b on joelsch:large-codesystem-persistence-fix into 0d06627 on jamesagnew:master.

@jamesagnew
Copy link
Owner

@jamesagnew jamesagnew commented Mar 1, 2017

Does your great big transaction bundle save correctly with this change?

I think the original intent of the constant flushing was to avoid having the JPA session keeping all of the entities in memory as it saves. Presumably that was an unneccesary "optimization" if this is working with your proposed change. :)

@joelsch
Copy link
Author

@joelsch joelsch commented Mar 1, 2017

Yes, my great big transaction bundle saves correctly with this change. Here's an abbreviated log file excerpt ...

2017-03-01 05:07:40.647 [http-nio-8080-exec-8] INFO  c.u.f.j.dao.dstu3.FhirSystemDaoDstu3 [FhirSystemDaoDstu3.java:281] Beginning Transaction with 2 resources
2017-03-01 05:07:40.648 [http-nio-8080-exec-8] INFO  c.u.f.j.dao.dstu3.FhirSystemDaoDstu3 [FhirSystemDaoDstu3.java:333] Processed 0 non-GET entries out of 2
2017-03-01 05:07:45.334 [http-nio-8080-exec-8] INFO  c.u.f.j.d.d.FhirResourceDaoCodeSystemDstu3 [FhirResourceDaoCodeSystemDstu3.java:204] CodeSystem CodeSystem/1/_history/0 has a status of complete, going to store concepts in terminology tables
2017-03-01 05:07:46.449 [http-nio-8080-exec-8] INFO  c.u.f.j.d.d.FhirResourceDaoCodeSystemDstu3 [FhirResourceDaoCodeSystemDstu3.java:216] Code system has 399837 concepts
2017-03-01 05:07:46.450 [http-nio-8080-exec-8] INFO  c.u.f.j.term.BaseHapiTerminologySvc [BaseHapiTerminologySvc.java:470] Storing code system
...
2017-03-01 05:34:37.111 [http-nio-8080-exec-8] INFO  c.u.f.j.dao.dstu3.FhirSystemDaoDstu3 [FhirSystemDaoDstu3.java:602] Transaction completed in 1616464ms
2017-03-01 05:40:02.044 [scheduledExecutorService-5] INFO  c.u.f.j.term.BaseHapiTerminologySvc [BaseHapiTerminologySvc.java:427] Saving 2000 deferred concepts...
...
2017-03-01 05:49:04.738 [scheduledExecutorService-2] INFO  c.u.f.j.term.BaseHapiTerminologySvc [BaseHapiTerminologySvc.java:458] All deferred concepts and relationships have now been synchronized to the database
@joelsch
Copy link
Author

@joelsch joelsch commented Mar 2, 2017

Additionally, to process the large bundle, I increased the Java heap size to 2048m. 1536m might also work.

jamesagnew added a commit that referenced this pull request Mar 17, 2017
@jamesagnew
Copy link
Owner

@jamesagnew jamesagnew commented Mar 17, 2017

Fantastic, merging this now! Thanks again!

@jamesagnew jamesagnew merged commit 899ed25 into jamesagnew:master Mar 17, 2017
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage remained the same at 81.75%
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.