Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extract compartment search parameters regardless of search parameter configuration #1777

Closed
michaelwschroeder opened this issue Dec 2, 2020 · 4 comments
Closed

Extract compartment search parameters regardless of search parameter configuration #1777

michaelwschroeder opened this issue Dec 2, 2020 · 4 comments
Assignees
@lmsurpre
Labels
deprecation P2 Priority 2 - Should Have search

Comments

@michaelwschroeder
Copy link
Contributor

michaelwschroeder commented Dec 2, 2020 •
edited
Loading

Is your feature request related to a problem? Please describe.
If compartment inclusion criteria search parameters are not specified in the fhirServer/resources/<resourceType>/searchParameters list for a given resource type, then the new generated compartment search token (if the fhirServer/search/useStoredCompartmentParam value is set to true - see issue #1708) or the tokens for the inclusion criteria search parameters, are not created/stored at resource create/update time, and thus that resource will not get returned on a compartment search as it should.

Example:

  1. fhir-server-config.json only specifies _id and _lastUpdated as valid search parameters for the Procedure resource type, not the patient and performer inclusion criteria search parameters:

        "resources": {
      "Procedure": {
        "searchParameters": {
          "_id": "http://hl7.org/fhir/SearchParameter/Resource-id",
          "_lastUpdated": "http://hl7.org/fhir/SearchParameter/Resource-lastUpdated"
        }
      }
    },

  2. A Procedure resource is created which contains the following Patient reference, meaning it should be associated with that Patient compartment:

        "subject": {
      "reference": "Patient/17575"
    },

    However, the Patient compartment inclusion criteria search parameters for the Procedure resource type, patient and performer, are not in the fhirServer/resources/<resourceType>/searchParameters list, so are not considered valid search parameters. This means the generated compartment search token (if enabled) or the inclusion criteria search tokens will not be created and indexed when the Procedure resource is stored.

  3. The following compartment search is performed:

    Patient/17575/Procedure

    The Procedure created in step 2 will not be returned in the search results because the necessary compartment search token(s), which are what's used in the compartment search SQL, were not created and indexed.

Describe the solution you'd like
When extracting compartment inclusion criteria search parameters for a resource when persisting the resource, we should assume the search parameters are valid rather than checking them against the configured search parameters for the tenant, which might not have included the compartment inclusion criteria search parameters. Then when validating search parameters on a compartment search request, again, we should not validate the compartment inclusion criteria search parameters against the configured search parameters for the tenant, but instead assume they are valid.

Describe alternatives you've considered
See issue #1707 for alternatives considered.

Acceptance Criteria
At least one acceptance criteria is included.
1.
GIVEN the compartment inclusion criteria search parameters for a resource type are not configured as valid search parameters
WHEN a compartment search against the resource type is executed
THEN all valid resources are returned
2.
GIVEN the compartment inclusion criteria search parameters for a resource type are not configured as valid search parameters
WHEN a standard search against the resource type is executed which contains one of the inclusion criteria search parameters
THEN an error will be returned indicating the search parameter is not valid for the given resource type

Additional context
Add any other context or screenshots about the feature request here.
This was referenced Dec 2, 2020
Closed
Closed
@lmsurpre
Copy link
Member

lmsurpre commented Jun 4, 2021

When we do this one, we should deprecate the fhirServer/search/useStoredCompartmentParam variable and simply always use the internal search param

@lmsurpre lmsurpre mentioned this issue Oct 29, 2021
Closed
@lmsurpre
Copy link
Member

lmsurpre commented Nov 8, 2021 •
edited
Loading

Investigation: what is the current behavior?
Now that we have useStoredCompartmentParam , does compartment search work or not when the inclusion criteria parameters are filtered out?

@lmsurpre lmsurpre added the P2 Priority 2 - Should Have label Nov 8, 2021
@lmsurpre lmsurpre added this to the Sprint 2021-15 milestone Nov 8, 2021
@lmsurpre lmsurpre self-assigned this Nov 15, 2021
@lmsurpre lmsurpre mentioned this issue Nov 22, 2021
Closed
@lmsurpre lmsurpre removed this from the Sprint 2021-16 milestone Dec 9, 2021
@lmsurpre
Copy link
Member

I confirmed that we do not extract the ibm-internal-Patient-compartment parameter if the corresponding patient search parameter is not enabled.

lmsurpre added a commit that referenced this issue Feb 21, 2022
@lmsurpre
issue #1777 - extract compartment inclusion params even when filtered
2c9ea8c 
Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>
lmsurpre added a commit that referenced this issue Feb 21, 2022
@lmsurpre
issue #1777 - do not store temporary/internal param values
8bef782 
In the previous commit, we started extracting all compartment inclusion
criteria search parameters, even in cases where they are filtered out
via config.
Although the filtered out parameters would not be searchable, we were
still storing these non-searchable parameter values in the db (once with
the inclusion criteria name and once with the corresponding
ibm-internal-x-compartment search parameter name).

This commit updates the search extraction implementation to avoid
storing these intermediate inclusion criteria parameters to the db.
To accomplish this, we now:
1. Add a DO_NOT_STORE extension on the intermediate search parameters
that are used to extract compartment membership information prior to
collecting those values for the ibm-internal-x-compartment search params
2. Propagate that information to the ExtractedParameterValues
3. Filter out parameter values that are not for storage before computing
the extract search parameters hash and returning
ExtractedSearchParameters

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>
lmsurpre added a commit that referenced this issue Feb 21, 2022
@lmsurpre
issue #1777 - do not store temporary/internal param values
36290d6 
In the previous commit, we started extracting all compartment inclusion
criteria search parameters, even in cases where they are filtered out
via config.
Although the filtered out parameters would not be searchable, we were
still storing these non-searchable parameter values in the db (once with
the inclusion criteria name and once with the corresponding
ibm-internal-x-compartment search parameter name).

This commit updates the search extraction implementation to avoid
storing these intermediate inclusion criteria parameters to the db.
To accomplish this, we now:
1. Add a DO_NOT_STORE extension on the intermediate search parameters
that are used to extract compartment membership information prior to
collecting those values for the ibm-internal-x-compartment search params
2. Propagate that information to the ExtractedParameterValues
3. Filter out parameter values that are not for storage before computing
the extract search parameters hash and returning
ExtractedSearchParameters

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>
lmsurpre added a commit that referenced this issue Feb 22, 2022
@lmsurpre
issue #1777 - do not store temporary/internal param values
beb93d1 
In the previous commit, we started extracting all compartment inclusion
criteria search parameters, even in cases where they are filtered out
via config.
Although the filtered out parameters would not be searchable, we were
still storing these non-searchable parameter values in the db (once with
the inclusion criteria name and once with the corresponding
ibm-internal-x-compartment search parameter name).

This commit updates the search extraction implementation to avoid
storing these intermediate inclusion criteria parameters to the db.
To accomplish this, we now:
1. Add a DO_NOT_STORE extension on the intermediate search parameters
that are used to extract compartment membership information prior to
collecting those values for the ibm-internal-x-compartment search params
2. Propagate that information to the ExtractedParameterValues
3. Filter out parameter values that are not for storage before computing
the extract search parameters hash and returning
ExtractedSearchParameters

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>
lmsurpre added a commit that referenced this issue Feb 22, 2022
@lmsurpre
issue #1777 - do not store temporary/internal param values
a58fc4c 
In the previous commit, we started extracting all compartment inclusion
criteria search parameters, even in cases where they are filtered out
via config.
Although the filtered out parameters would not be searchable, we were
still storing these non-searchable parameter values in the db (once with
the inclusion criteria name and once with the corresponding
ibm-internal-x-compartment search parameter name).

This commit updates the search extraction implementation to avoid
storing these intermediate inclusion criteria parameters to the db.
To accomplish this, we now:
1. Add a DO_NOT_STORE extension on the intermediate search parameters
that are used to extract compartment membership information prior to
collecting those values for the ibm-internal-x-compartment search params
2. Propagate that information to the ExtractedParameterValues
3. Filter out parameter values that are not for storage before computing
the extract search parameters hash and returning
ExtractedSearchParameters

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>
lmsurpre added a commit that referenced this issue Feb 22, 2022
@lmsurpre
issue #1777 - do not store temporary/internal param values
c10c55d 
In the previous commit, we started extracting all compartment inclusion
criteria search parameters, even in cases where they are filtered out
via config.
Although the filtered out parameters would not be searchable, we were
still storing these non-searchable parameter values in the db (once with
the inclusion criteria name and once with the corresponding
ibm-internal-x-compartment search parameter name).

This commit updates the search extraction implementation to avoid
storing these intermediate inclusion criteria parameters to the db.
To accomplish this, we now:
1. Add a DO_NOT_STORE extension on the intermediate search parameters
that are used to extract compartment membership information prior to
collecting those values for the ibm-internal-x-compartment search params
2. Propagate that information to the ExtractedParameterValues
3. Filter out parameter values that are not for storage before computing
the extract search parameters hash and returning
ExtractedSearchParameters

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>
lmsurpre added a commit that referenced this issue Feb 22, 2022
@lmsurpre
issue #1777 - do not store temporary/internal param values
21eeb11 
In the previous commit, we started extracting all compartment inclusion
criteria search parameters, even in cases where they are filtered out
via config.
Although the filtered out parameters would not be searchable, we were
still storing these non-searchable parameter values in the db (once with
the inclusion criteria name and once with the corresponding
ibm-internal-x-compartment search parameter name).

This commit updates the search extraction implementation to avoid
storing these intermediate inclusion criteria parameters to the db.
To accomplish this, we now:
1. Add a DO_NOT_STORE extension on the intermediate search parameters
that are used to extract compartment membership information prior to
collecting those values for the ibm-internal-x-compartment search params
2. Propagate that information to the ExtractedParameterValues
3. Filter out parameter values that are not for storage before computing
the extract search parameters hash and returning
ExtractedSearchParameters

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>
lmsurpre added a commit that referenced this issue Feb 22, 2022
@lmsurpre
issue #1777 - added tests and minor refactor
20fc22d 
1. added simple unit tests for the new ParametersMap behavior
2. added single e2e test to SearchTest to cover the case where the
compartment inclusion criteria parameter is filtered out in the config

Also, we now add the DO_NOT_STORE extension from SearchUtil instead of
as the parameters are added to the ParametersMap

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>
lmsurpre added a commit that referenced this issue Mar 2, 2022
@lmsurpre
Merge pull request #3376 from IBM/issue-1777
1e4e99f 
issue #1777 - extract compartment inclusion params even when filtered
@d0roppe
Copy link
Collaborator

d0roppe commented Mar 14, 2022

Verified issue with 2 different compartments. closing issue

@d0roppe d0roppe closed this as completed Mar 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lmsurpre lmsurpre

Labels
deprecation P2 Priority 2 - Should Have search
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lmsurpre @michaelwschroeder @d0roppe