New batch framework / Bulk Import

[jamesagnew]

Note: The QueryCount logic used to count SQL statements has changed so that batched statements are unrolled (e.g. if 3 SQL statements were identical and submitted as a batch that used to be counted as 1 and it's now counted as 3). A number of SQL statement counts have therefore increased in this ticket. I have confirmed every one of them and they are all not actually increases. This new way seems more accurate.

[lgtm-com]

This pull request introduces 2 alerts when merging 32de383 into 700221a - view on LGTM.com

new alerts:

• 1 for Information exposure through a stack trace
• 1 for Dereferenced variable may be null

[lgtm-com]

This pull request introduces 3 alerts when merging 9b6dbae into fb342cc - view on LGTM.com

new alerts:

• 1 for Information exposure through a stack trace
• 1 for Dereferenced variable may be null
• 1 for Potential input resource leak

[lgtm-com]

This pull request introduces 3 alerts when merging 19db798 into ae33cf8 - view on LGTM.com

new alerts:

• 1 for Information exposure through a stack trace
• 1 for Dereferenced variable may be null
• 1 for Potential input resource leak

[lgtm-com]

This pull request introduces 3 alerts when merging e46be36 into 6313918 - view on LGTM.com

new alerts:

• 1 for Information exposure through a stack trace
• 1 for Dereferenced variable may be null
• 1 for Potential input resource leak

[lgtm-com]

This pull request introduces 3 alerts when merging 40575dd into 0c5d868 - view on LGTM.com

new alerts:

• 1 for Information exposure through a stack trace
• 1 for Dereferenced variable may be null
• 1 for Potential input resource leak

[lgtm-com]

This pull request introduces 3 alerts when merging acc93ba into 0c5d868 - view on LGTM.com

new alerts:

• 1 for Information exposure through a stack trace
• 1 for Dereferenced variable may be null
• 1 for Potential input resource leak

[lgtm-com]

This pull request introduces 3 alerts when merging 2a70c72 into 0c5d868 - view on LGTM.com

new alerts:

• 1 for Information exposure through a stack trace
• 1 for Dereferenced variable may be null
• 1 for Potential input resource leak

[lgtm-com]

This pull request introduces 1 alert when merging 4af35ae into 1e2e177 - view on LGTM.com

new alerts:

• 1 for Potential input resource leak

[lgtm-com]

This pull request introduces 1 alert when merging b01d6a6 into 1e2e177 - view on LGTM.com

new alerts:

• 1 for Potential input resource leak

[lgtm-com]

This pull request introduces 1 alert when merging 2f8e055 into 1e2e177 - view on LGTM.com

new alerts:

• 1 for Potential input resource leak

[lgtm-com]

This pull request introduces 1 alert when merging 816c050 into cc44dee - view on LGTM.com

new alerts:

• 1 for Potential input resource leak

[codecov]

Codecov Report

Merging #3387 (8aa9269) into master (65776bb) will increase coverage by 0.03%.
The diff coverage is 89.00%.

❗ Current head 8aa9269 differs from pull request most recent head ae12b8d. Consider uploading reports for the commit ae12b8d to get more accurate results

@@             Coverage Diff              @@
##             master    #3387      +/-   ##
============================================
+ Coverage     82.98%   83.02%   +0.03%     
- Complexity    20746    20859     +113     
============================================
  Files          1396     1400       +4     
  Lines         74540    74875     +335     
  Branches      11163    11176      +13     
============================================
+ Hits          61860    62166     +306     
- Misses         8377     8415      +38     
+ Partials       4303     4294       -9     

Impacted Files	Coverage Δ
...-fhir-base/src/main/java/ca/uhn/fhir/i18n/Msg.java	100.00% <ø> (ø)
...src/main/java/ca/uhn/fhir/util/ParametersUtil.java	92.62% <0.00%> (-2.74%)	⬇️
.../uhn/fhir/jpa/cache/ResourceVersionSvcDaoImpl.java	90.74% <ø> (ø)
.../main/java/ca/uhn/fhir/jpa/dao/HistoryBuilder.java	100.00% <ø> (ø)
.../java/ca/uhn/fhir/jpa/dao/LegacySearchBuilder.java	52.61% <ø> (ø)
...hn/fhir/jpa/dao/index/DaoResourceLinkResolver.java	91.74% <ø> (ø)
...ndex/SearchParamWithInlineReferencesExtractor.java	86.70% <ø> (ø)
...java/ca/uhn/fhir/jpa/dao/mdm/MdmLinkExpandSvc.java	10.71% <ø> (ø)
...r/jpa/dao/predicate/PredicateBuilderReference.java	71.06% <ø> (ø)
.../jpa/dao/predicate/PredicateBuilderResourceId.java	71.11% <ø> (ø)
... and 66 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 65776bb...ae12b8d. Read the comment docs.

[michaelabuckley]

Very nice. Using types for chunk input and output is much clearer.

Need to schedule kanban items for:

• error retry limits
• avoiding $reindex stalling all other activity

[lgtm-com]

This pull request introduces 2 alerts and fixes 1 when merging 8aa9269 into 65776bb - view on LGTM.com

new alerts:

• 1 for HTTP response splitting
• 1 for Potential input resource leak

fixed alerts:

• 1 for HTTP response splitting

[lgtm-com]

This pull request introduces 2 alerts and fixes 1 when merging ae12b8d into 65776bb - view on LGTM.com

new alerts:

• 1 for HTTP response splitting
• 1 for Potential input resource leak

fixed alerts:

• 1 for HTTP response splitting