-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce storage required for indexing - stop writing sp_name, res_type, and sp_updated to hfj_spidx_* tables #5941
Reduce storage required for indexing - stop writing sp_name, res_type, and sp_updated to hfj_spidx_* tables #5941
Conversation
…_type and sp_updated columns of HFJ_SPIDX tables nullable
Formatting check succeeded! |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #5941 +/- ##
============================================
+ Coverage 83.39% 83.51% +0.11%
- Complexity 26927 27324 +397
============================================
Files 1681 1701 +20
Lines 103965 105751 +1786
Branches 13189 13351 +162
============================================
+ Hits 86702 88315 +1613
- Misses 11613 11729 +116
- Partials 5650 5707 +57 ☔ View full report in Codecov by Sentry. |
...ver-model/src/main/java/ca/uhn/fhir/jpa/model/listener/IndexStorageOptimizationListener.java
Outdated
Show resolved
Hide resolved
… RES_TYPE after update/load
…-required-for-indexing-tables # Conflicts: # hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/migrate/tasks/HapiFhirJpaMigrationTasks.java
…eters if IndexMissingFields and optimizeIndexStorage are both enabled
…rect configuration handling
… SP recovery, documentation updates
…-required-for-indexing-tables
…-required-for-indexing-tables
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approved pending various comments.
...main/resources/ca/uhn/hapi/fhir/changelog/7_4_0/5937-reduce-storage-for-sp-index-tables.yaml
Outdated
Show resolved
Hide resolved
hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/config/SearchConfig.java
Outdated
Show resolved
Hide resolved
...ir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/migrate/tasks/HapiFhirJpaMigrationTasks.java
Show resolved
Hide resolved
.../src/main/java/ca/uhn/fhir/jpa/search/builder/predicate/BaseSearchParamPredicateBuilder.java
Show resolved
Hide resolved
...ver-model/src/main/java/ca/uhn/fhir/jpa/model/listener/IndexStorageOptimizationListener.java
Outdated
Show resolved
Hide resolved
...ver-model/src/main/java/ca/uhn/fhir/jpa/model/listener/IndexStorageOptimizationListener.java
Show resolved
Hide resolved
hapi-fhir-jpaserver-model/src/main/java/ca/uhn/fhir/jpa/model/util/SearchParamHash.java
Show resolved
Hide resolved
…-required-for-indexing-tables
…-required-for-indexing-tables # Conflicts: # hapi-fhir-jpaserver-model/src/test/java/ca/uhn/fhir/jpa/model/entity/ResourceIndexedSearchParamCoordsTest.java # hapi-fhir-jpaserver-model/src/test/java/ca/uhn/fhir/jpa/model/entity/ResourceIndexedSearchParamDateTest.java # hapi-fhir-jpaserver-model/src/test/java/ca/uhn/fhir/jpa/model/entity/ResourceIndexedSearchParamQuantityNormalizedTest.java # hapi-fhir-jpaserver-model/src/test/java/ca/uhn/fhir/jpa/model/entity/ResourceIndexedSearchParamQuantityTest.java # hapi-fhir-jpaserver-model/src/test/java/ca/uhn/fhir/jpa/model/entity/ResourceIndexedSearchParamStringTest.java # hapi-fhir-jpaserver-model/src/test/java/ca/uhn/fhir/jpa/model/entity/ResourceIndexedSearchParamTokenTest.java # hapi-fhir-jpaserver-model/src/test/java/ca/uhn/fhir/jpa/model/entity/ResourceIndexedSearchParamUriTest.java
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good work.
Reading your work exposed some missing tests around the entity equals/hashCode work.
Made some minor suggestions.
Remember to restore settings changed Spring contexts after the test.
.../src/main/java/ca/uhn/fhir/jpa/search/builder/predicate/BaseSearchParamPredicateBuilder.java
Show resolved
Hide resolved
hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/docs/server_jpa/performance.md
Outdated
Show resolved
Hide resolved
...aserver-model/src/main/java/ca/uhn/fhir/jpa/model/entity/BaseResourceIndexedSearchParam.java
Outdated
Show resolved
Hide resolved
hapi-fhir-jpaserver-model/src/main/java/ca/uhn/fhir/jpa/model/util/SearchParamHash.java
Show resolved
Hide resolved
...ver-model/src/main/java/ca/uhn/fhir/jpa/model/listener/IndexStorageOptimizationListener.java
Show resolved
Hide resolved
hapi-fhir-server/src/main/java/ca/uhn/fhir/rest/server/util/FhirContextSearchParamRegistry.java
Outdated
Show resolved
Hide resolved
...test-r5/src/test/java/ca/uhn/fhir/jpa/dao/r5/FhirResourceDaoR5IndexStorageOptimizedTest.java
Show resolved
Hide resolved
.../ca/uhn/fhir/jpa/searchparam/matcher/InMemoryResourceMatcherR5IndexStorageOptimizedTest.java
Show resolved
Hide resolved
...test-r4/src/test/java/ca/uhn/fhir/jpa/dao/r4/FhirResourceDaoR4IndexStorageOptimizedTest.java
Show resolved
Hide resolved
...-jpaserver-test-r4/src/test/java/ca/uhn/fhir/jpa/dao/r4/FhirResourceDaoR4SearchNoFtTest.java
Outdated
Show resolved
Hide resolved
Co-authored-by: Michael Buckley <michaelabuckley@gmail.com>
Co-authored-by: Michael Buckley <michaelabuckley@gmail.com>
…-required-for-indexing-tables # Conflicts: # hapi-fhir-jpaserver-model/src/main/java/ca/uhn/fhir/jpa/model/entity/StorageSettings.java # hapi-fhir-jpaserver-test-r4/src/test/java/ca/uhn/fhir/jpa/dao/r4/FhirResourceDaoR4SearchNoFtTest.java
hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/docs/server_jpa/performance.md
Outdated
Show resolved
Hide resolved
…-required-for-indexing-tables # Conflicts: # hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/migrate/tasks/HapiFhirJpaMigrationTasks.java
…-required-for-indexing-tables
Migration:
HFJ_SPIDX
tables to allowSP_NAME
andRES_TYPE
columns to be nullable.failureAllowed()
, as bothSP_NAME
andRES_TYPE
could be included in custom indexes, and SQL Server won't let us change nullability on columns with indexes pointing to them.Optimization changes:
HFJ_SPIDX
tables - nullingSP_NAME
,RES_TYPE
SP_UPDATED
if this feature is enabled.JpaSearchParamCache
to recoverSP_NAME
,RES_TYPE
from hash_identity after loading from DB.HASH_IDENTITY
field was added toBaseResourceIndexedSearchParam
entity and removed from all inheritor entities.ResourceIndexedSearchParam
now usingHASH_IDENTITY
instead ofsp_name
andres_type
. This is required to makeResourceIndexedSearchParam
objects with and without optimization to be equal - to not cause unnecessaryResourceIndexedSearchParams
updates. (as we are comparing db version of entities with in-memory built Search params)DaoSearchParamSynchronizer
logic to check whether it is needed to update existing search parameters afterisIndexStorageOptimized
changeisIncludePartitionInSearchHashes
andisIndexStorageOptimized
are enabled on server. (isIncludePartitionInSearchHashes
is not supported ifisIndexStorageOptimized
is set totrue
)InMemoryResourceMatcher
now useshashIdentity
to filter SearchParams instead ofsp_name
. (only if optimization is enabled)hashIdentity
instead ofSP_NAME
,RES_TYPE
to build a query. This way new optimization could work in pair with EnabledIndexMissingFields
setting. (only if optimization is enabled)