New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Avoid fetching work-chunk data #4622
Conversation
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## rel_6_4 #4622 +/- ##
==========================================
Coverage ? 81.26%
Complexity ? 23863
==========================================
Files ? 1433
Lines ? 87607
Branches ? 11808
==========================================
Hits ? 71198
Misses ? 11154
Partials ? 5255 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
Avoid fetching work-chunk data (#4622)
* Use java event names for work chunk transitions. * Cherry-pick d5ebd1f from rel_6_4 Avoid fetching work-chunk data (#4622) * add end time to reduction step (#4640) * add end time to reduction step * add changelog --------- Co-authored-by: Long Ma <long@smilecdr.com> (cherry picked from commit 37f5e59) * Cancel processing Provide error message in cancelled jobs, and avoid transitions in final states. * Apply tx boundary to starting job and first chunk. * cleanup * Apply tx boundary to work chunk processing * Delete BatchWorkChunk * Introduce events for job create, and chunk dequeue * Apply tx boundary to chunk handler * Move instance cancellation to database * tx boundary around stats collection and completion * tx boundary around stats collection and completion * Extend tx boundary to error, fail, and cancel * Move failure into status calc * ERROR is not an "ended" state. * Revert generics cleanup to avoid noise * Avoid sending gated chunks twice. * Make no-data path safer. Cleanup * Fix mock test for step advance. * Delete unsafe updateInstace() call * Cleanup * Changelog and notes * Fix cancel boundary. Cleanups * Cleanup * Sort mongo chunks for stable paging. Other cleanup * Document error handling * Cleanup * Update hapi-fhir-jpaserver-test-utilities/src/main/java/ca/uhn/fhir/jpa/test/Batch2JobHelper.java Co-authored-by: StevenXLi <stevenli_8118@hotmail.com> --------- Co-authored-by: longma1 <32119004+longma1@users.noreply.github.com> Co-authored-by: StevenXLi <stevenli_8118@hotmail.com>
* Use java event names for work chunk transitions. * Cherry-pick d5ebd1f from rel_6_4 Avoid fetching work-chunk data (#4622) * add end time to reduction step (#4640) * add end time to reduction step * add changelog --------- Co-authored-by: Long Ma <long@smilecdr.com> (cherry picked from commit 37f5e59) * Cancel processing Provide error message in cancelled jobs, and avoid transitions in final states. * Apply tx boundary to starting job and first chunk. * cleanup * Apply tx boundary to work chunk processing * Delete BatchWorkChunk * Introduce events for job create, and chunk dequeue * Apply tx boundary to chunk handler * Move instance cancellation to database * tx boundary around stats collection and completion * tx boundary around stats collection and completion * Extend tx boundary to error, fail, and cancel * Move failure into status calc * ERROR is not an "ended" state. * Revert generics cleanup to avoid noise * Avoid sending gated chunks twice. * Make no-data path safer. Cleanup * Fix mock test for step advance. * Delete unsafe updateInstace() call * Cleanup * Changelog and notes * Fix cancel boundary. Cleanups * Cleanup * Sort mongo chunks for stable paging. Other cleanup * Document error handling * Cleanup * Update hapi-fhir-jpaserver-test-utilities/src/main/java/ca/uhn/fhir/jpa/test/Batch2JobHelper.java Co-authored-by: StevenXLi <stevenli_8118@hotmail.com> --------- Co-authored-by: longma1 <32119004+longma1@users.noreply.github.com> Co-authored-by: StevenXLi <stevenli_8118@hotmail.com>
* Use java event names for work chunk transitions. * Cherry-pick d5ebd1f from rel_6_4 Avoid fetching work-chunk data (#4622) * add end time to reduction step (#4640) * add end time to reduction step * add changelog --------- Co-authored-by: Long Ma <long@smilecdr.com> (cherry picked from commit 37f5e59) * Cancel processing Provide error message in cancelled jobs, and avoid transitions in final states. * Apply tx boundary to starting job and first chunk. * cleanup * Apply tx boundary to work chunk processing * Delete BatchWorkChunk * Introduce events for job create, and chunk dequeue * Apply tx boundary to chunk handler * Move instance cancellation to database * tx boundary around stats collection and completion * tx boundary around stats collection and completion * Extend tx boundary to error, fail, and cancel * Move failure into status calc * ERROR is not an "ended" state. * Revert generics cleanup to avoid noise * Avoid sending gated chunks twice. * Make no-data path safer. Cleanup * Fix mock test for step advance. * Delete unsafe updateInstace() call * Cleanup * Changelog and notes * Fix cancel boundary. Cleanups * Cleanup * Sort mongo chunks for stable paging. Other cleanup * Document error handling * Cleanup * Update hapi-fhir-jpaserver-test-utilities/src/main/java/ca/uhn/fhir/jpa/test/Batch2JobHelper.java Co-authored-by: StevenXLi <stevenli_8118@hotmail.com> --------- Co-authored-by: longma1 <32119004+longma1@users.noreply.github.com> Co-authored-by: StevenXLi <stevenli_8118@hotmail.com>
The batch maintenance job loop is fetching all chunk data. This change omits this unneeded I/O.