Merge pull request #3440 from IBM/issue-2886

issue #2886 - remove useStoredCompartmentParam property

docs/src/pages/guides/FHIRPerformanceGuide.md

@@ -277,16 +277,6 @@ The values for PARAMETER_NAMES and RESOURCE_TYPES are supposed to be fully cache
 
 Resources are assigned to various compartments using expressions with multiple terms. In the IBM FHIR Server JDBC persistence layer, these expressions are translated to SQL predicates with multiple `OR` statements. These `ORs` make it more difficult for the query optimizer to compute the most efficient execution plan resulting in a slow query. To address this, the IBM FHIR Server evaluates the compartment membership expression during ingestion and stores the results. The SQL query can then be written using a single value predicate resulting in faster query.
 
-To enable this optimization, set the `fhirServer/search/useStoredCompartmentParam` configuration parameter to `true` in the fhir-server-config.json file:
-
-``` json
-        "search": {
-            "useStoredCompartmentParam": true
-        },
-```
-
-Enabling this optimization is recommended. See the IBM FHIR Server release notes for more details.
-
 ## 3.6. Usage of Server Resource Provider
 
 The IBM FHIR Server has a dynamic registry of conformance resources. The built-in "ServerRegistryResourceProvider" can be used to bridge conformance resources from the tenant data store (uploaded through the REST API) to the registry.

docs/src/pages/guides/FHIRSearchConfiguration.md

@@ -193,54 +193,17 @@ The IBM FHIR Server supports configurable handling of unknown or unsupported sea
 Filtered search parameters are handled exactly the same as undefined search parameters, meaning that searches which include these parameters will fail in `strict` mode.
 
 #### 1.2.2 Compartment search considerations with filtering
-It is possible that results returned by compartment searches can be affected by filtering. For each compartment type, the rules for determining if a resource is a member of a compartment of that type, and thus if that resource should be returned on an associated compartment search, are based on inclusion criteria. Inclusion criteria is defined to be one or more search parameters whose value is a reference to the compartment resource type. These membership rules are defined by the FHIR specification for the following compartments:
-- [Device](https://www.hl7.org/fhir/compartmentdefinition-device.html)
-- [Encounter](https://www.hl7.org/fhir/compartmentdefinition-encounter.html)
-- [Patient](https://www.hl7.org/fhir/compartmentdefinition-patient.html)
-- [Practitioner](https://www.hl7.org/fhir/compartmentdefinition-practitioner.html)
-- [RelatedPerson](https://www.hl7.org/fhir/compartmentdefinition-relatedperson.html)
+For each compartment type, the rules for determining if a resource is a member of a compartment of that type, and thus if that resource should be returned on an associated compartment search, are based on inclusion criteria. Inclusion criteria is one or more search parameters whose value is a reference to the compartment resource type. These membership rules are defined by the FHIR specification for the following compartments:
+- [Device](https://www.hl7.org/fhir/R4/compartmentdefinition-device.html)
+- [Encounter](https://www.hl7.org/fhir/R4/compartmentdefinition-encounter.html)
+- [Patient](https://www.hl7.org/fhir/R4/compartmentdefinition-patient.html)
+- [Practitioner](https://www.hl7.org/fhir/R4/compartmentdefinition-practitioner.html)
+- [RelatedPerson](https://www.hl7.org/fhir/R4/compartmentdefinition-relatedperson.html)
 
 For example, for the `Patient` compartment, to determine if an `Observation` is a member, the inclusion criteria search parameters are `subject` and `performer`. If the `Observation` resource fields associated with these search parameters reference a `Patient` resource, the `Observation` resource is a member of that `Patient` compartment.
 
-However, in order for a resource to be returned as expected on a compartment search, the search index information associated with the inclusion criteria search parameters for that resource must get generated and stored at the time the resource is created or updated. If these search parameters have been filtered out based on the filtering configuration in `fhir-server-config.json`, the IBM FHIR Server will not store the necessary search index information, thus the resource will not be returned as expected on a compartment search. Using the example above, if the `subject` and `performer` search parameters have been filtered out for the `Observation` resource, it will not be returned as expected on a `Patient` compartment search. The following snippet illustrates a search parameter configuration in which the `subject` and `performer` search parameters have been filtered out:
-
-```
-"resources": {
-    "open": true,
-    "Observation": {
-        "searchParameters": {
-            "_id": "http://hl7.org/fhir/SearchParameter/Resource-id",
-            "_lastUpdated": "http://hl7.org/fhir/SearchParameter/Resource-lastUpdated",
-            "patient": "http://hl7.org/fhir/SearchParameter/clinical-patient",
-            "category": "http://hl7.org/fhir/SearchParameter/Observation-category",
-            "code": "http://hl7.org/fhir/SearchParameter/clinical-code",
-            "status": "http://hl7.org/fhir/SearchParameter/Observation-status",
-            "date": "http://hl7.org/fhir/SearchParameter/clinical-date"
-        }
-    }
-}
-```
-
-In order to avoid this issue, inclusion criteria search parameters should not be filtered out. If any filtering is configured in `fhir-server-config.json` for resources that may be members of a compartment, their inclusion criteria search parameters should be included in the list of allowed search parameters. Again using the example above, if search parameter filtering is specified for the `Observation` resource type, the `subject` and `performer` search parameters must be specified in the `searchParameters` list (assuming an entry of `"*": "*"` is not specified) in order for `Observation` resources to be returned in `Patient` compartment searches. The following snippet illustrates a search parameter configuration in which the `subject` and `performer` search parameters have been included in the list of allowed search parameters:
-
-```
-"resources": {
-    "open": true,
-    "Observation": {
-        "searchParameters": {
-            "_id": "http://hl7.org/fhir/SearchParameter/Resource-id",
-            "_lastUpdated": "http://hl7.org/fhir/SearchParameter/Resource-lastUpdated",
-            "patient": "http://hl7.org/fhir/SearchParameter/clinical-patient",
-            "category": "http://hl7.org/fhir/SearchParameter/Observation-category",
-            "code": "http://hl7.org/fhir/SearchParameter/clinical-code",
-            "status": "http://hl7.org/fhir/SearchParameter/Observation-status",
-            "date": "http://hl7.org/fhir/SearchParameter/clinical-date",
-            "subject": "http://hl7.org/fhir/SearchParameter/Observation-subject",
-            "performer": "http://hl7.org/fhir/SearchParameter/Observation-performer"
-        }
-    }
-}
-```
+As of version 5.0.0, compartment membership is always evaluated during ingestion, even if the search parameters that define compartment membership have been filtered out in fhir-server-config.json.
+However, in cases where the inclusion criteria parameters have been overridden, it is still possible for server config to affect compartment membership.
 
 ##  2 Re-index
 Reindexing is implemented as a custom operation that tells the IBM FHIR Server to read a set of resources and replace the existing search parameters with those newly extracted from the resource body.

docs/src/pages/guides/FHIRServerUsersGuide.md

@@ -208,22 +208,6 @@ Search parameters are handled like a single configuration properly; providing a
 
 More information about multi-tenant support can be found in [Section 4.9 Multi-tenancy](#49-multi-tenancy).
 
-### 3.2.3 Compartment Search Performance
-
-The IBM FHIR Server supports the ability to compute and store compartment membership values during ingestion. Once stored, these values can help accelerate compartment-related search queries. To use this feature, update the IBM FHIR Server to at least version 4.5.1 and run a reindex operation as described in the [fhir-bucket](https://github.com/IBM/FHIR/tree/main/fhir-bucket) project [README](https://github.com/IBM/FHIR/blob/main/fhir-bucket/README.md). The reindex operation reprocesses the resources stored in the database, computing and storing the new compartment reference values. After the reindex operation has completed, add the `useStoredCompartmentParam` configuration element to the relevant tenant fhir-server-config.json file to allow the search queries to use the pre-computed values:
-
-```
-    {
-        "fhirServer": {
-            "search": {
-                "useStoredCompartmentParam": true
-            }
-        }
-    }
-```
-
-Note that this parameter only enables or disables the compartment search query optimization feature. The compartment membership values are always computed and stored during ingestion or reindexing, regardless of the setting of this value. After the reindex operation is complete, it is recommended to set `useStoredCompartmentParam` to true. No reindex is required if this value is subsequently set to false.
-
 ## 3.3 Persistence layer configuration
 The IBM FHIR Server allows deployers to select a persistence layer implementation that fits their needs. Currently, the server includes a JDBC persistence layer which supports Apache Derby, IBM Db2, and PostgreSQL.  However, Apache Derby is not recommended for production usage.
 
@@ -2194,7 +2178,6 @@ This section contains reference information about each of the configuration prop
 |`fhirServer/audit/hostname`|string|A string used to identify the Hostname, useful in containerized environments|
 |`fhirServer/audit/ip`|string|A string used to identify the IP address, useful to identify only one IP|
 |`fhirServer/search/useBoundingRadius`|boolean|If true then the bounding area is a radius, else the bounding area is a box.|
-|`fhirServer/search/useStoredCompartmentParam`|boolean|True, compute and store parameter to accelerate compartment searches. Requires reindex using at least IBM FHIR Server version 4.5.1 before this feature is enabled |
 |`fhirServer/search/enableLegacyWholeSystemSearchParams`|boolean|True, searches specifying whole-system search parameters `_profile`, `_tag`, and `_security` run against legacy search index data, else those searches run against new search index data. This property can be set to `true` before a reindex operation is run, and after migrating to IBM FHIR Server version 4.9.0, to allow searches to work while the reindex operation is in progress. After the reindex has completed successfully, the property should be set to `false` or removed from the configuration. |
 |`fhirServer/bulkdata/enabled`| string|Enabling the BulkData operations |
 |`fhirServer/bulkdata/core/api/url`|string|The URL to access the FHIR server hosting the batch web application |
@@ -2338,7 +2321,6 @@ This section contains reference information about each of the configuration prop
 |`fhirServer/persistence/datasources/<datasourceId>/searchOptimizerOptions/from_collapse_limit`|16|
 |`fhirServer/persistence/datasources/<datasourceId>/searchOptimizerOptions/join_collapse_limit`|16|
 |`fhirServer/search/useBoundingRadius`|false|
-|`fhirServer/search/useStoredCompartmentParam`|boolean|true|
 |`fhirServer/search/enableLegacyWholeSystemSearchParams`|false|
 |`fhirServer/security/cors`|true|
 |`fhirServer/security/basic/enabled`|false|
@@ -2498,7 +2480,6 @@ must restart the server for that change to take effect.
 |`fhirServer/persistence/datasources/<datasourceId>/searchOptimizerOptions/from_collapse_limit`|Y|Y|
 |`fhirServer/persistence/datasources/<datasourceId>/searchOptimizerOptions/join_collapse_limit`|Y|Y|
 |`fhirServer/search/useBoundingRadius`|Y|Y|
-|`fhirServer/search/useStoredCompartmentParam`|Y|Y|
 |`fhirServer/search/enableLegacyWholeSystemSearchParams`|Y|Y|
 |`fhirServer/security/cors`|Y|Y|
 |`fhirServer/security/basic/enabled`|Y|Y|

fhir-config/src/main/java/com/ibm/fhir/config/FHIRConfiguration.java

@@ -125,8 +125,7 @@ public class FHIRConfiguration {
 
     // Search config properties
     public static final String PROPERTY_SEARCH_BOUNDING_AREA_RADIUS_TYPE = "fhirServer/search/useBoundingRadius";
-    public static final String PROPERTY_USE_STORED_COMPARTMENT_PARAM = "fhirServer/search/useStoredCompartmentParam";
-
+    @Deprecated
     public static final String PROPERTY_SEARCH_ENABLE_LEGACY_WHOLE_SYSTEM_SEARCH_PARAMS = "fhirServer/search/enableLegacyWholeSystemSearchParams";
 
     // bulkdata

fhir-search/src/main/java/com/ibm/fhir/search/util/SearchHelper.java

@@ -1204,18 +1204,6 @@ public FHIRSearchContext parseCompartmentQueryParameters(String compartmentName,
         return parseCompartmentQueryParameters(compartmentName, compartmentLogicalId, resourceType, queryParameters, true, true);
     }
 
-    /**
-     * Check the configuration to see if the flag enabling the compartment search
-     * optimization. Defaults to false so the behavior won't change unless it
-     * is explicitly enabled in fhir-server-config. This is important, because
-     * existing data must be reindexed (see $reindex custom operation) to
-     * generate values for the ibm-internal compartment relationship params.
-     * @return
-     */
-    public static boolean useStoredCompartmentParam() {
-        return FHIRConfigHelper.getBooleanProperty(FHIRConfiguration.PROPERTY_USE_STORED_COMPARTMENT_PARAM, true);
-    }
-
     /**
      * @param compartmentName
      * @param compartmentLogicalId
@@ -1260,19 +1248,10 @@ public QueryParameter buildInclusionCriteria(String compartmentName, Set<String>
         if (compartmentName != null && compartmentLogicalIds != null && !compartmentLogicalIds.isEmpty()) {
             // The inclusion criteria are represented as a chain of parameters, each with a value of the
             // compartmentLogicalId.
-            // The query parsers will OR these parameters to achieve the compartment search.
             List<String> inclusionCriteria;
 
-            if (useStoredCompartmentParam()) {
-                // issue #1708. When enabled, use the ibm-internal-... compartment parameter. This
-                // results in faster queries because only a single parameter is used to represent the
-                // compartment membership.
-                compartmentHelper.checkValidCompartmentAndResource(compartmentName, resourceType);
-                inclusionCriteria = Collections.singletonList(CompartmentHelper.makeCompartmentParamName(compartmentName));
-            } else {
-                // pre #1708 behavior
-                inclusionCriteria = compartmentHelper.getCompartmentResourceTypeInclusionCriteria(compartmentName, resourceType);
-            }
+            compartmentHelper.checkValidCompartmentAndResource(compartmentName, resourceType);
+            inclusionCriteria = Collections.singletonList(CompartmentHelper.makeCompartmentParamName(compartmentName));
 
             for (String criteria : inclusionCriteria) {
                 QueryParameter parameter  = new QueryParameter(Type.REFERENCE, criteria, null, null, true);