Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duplicate Job Parameters fhir.dataSourcesInfo are created #1855

Closed
prb112 opened this issue Jan 8, 2021 · 0 comments · Fixed by #1856
Closed

Duplicate Job Parameters fhir.dataSourcesInfo are created #1855

prb112 opened this issue Jan 8, 2021 · 0 comments · Fixed by #1856
Assignees
Labels
bug Something isn't working

Comments

@prb112
Copy link
Contributor

prb112 commented Jan 8, 2021

Describe the bug
Duplicate Job Parameters are created in

"fhir.dataSourcesInfo": "ClsKICAgIHsKICAgICAgICAidHlwZSI6ICJQYXRpZW50IiwKICAgICAgICAidXJsIjogImh0dHBzOi8vbG9jYWxob3N0OjkwMDAvZmhpcmJ1bGtkYXRhL3Rlc3QtaW1wb3J0Lm5kanNvbiIKICAgIH0KXQ==",

Tried this out and hit this serialization for the JobParameters. I'll fix the duplicate storageType with this issue.

"jobParameters": {
        "cos.credential.ibm": "N",
        "cos.operationoutcomes.bucket.name": "fhirbulkdata",
        "cos.endpoint.internal": "https://localhost:9000",
        "cos.endpoint.external": "https://localhost:9000",
        "cos.location": "us",
        "fhir.tenant": "default",
        "fhir.datastoreid": "default",
        "fhir.dataSourcesInfo": "ClsKICAgIHsKICAgICAgICAidHlwZSI6ICJQYXRpZW50IiwKICAgICAgICAidXJsIjogImh0dHBzOi8vbG9jYWxob3N0OjkwMDAvZmhpcmJ1bGtkYXRhL3Rlc3QtaW1wb3J0Lm5kanNvbiIKICAgIH0KXQ==",
        "fhir.dataSourcesInfo": "ClsKICAgIHsKICAgICAgICAidHlwZSI6ICJQYXRpZW50IiwKICAgICAgICAidXJsIjogImh0dHBzOi8vbG9jYWxob3N0OjkwMDAvZmhpcmJ1bGtkYXRhL3Rlc3QtaW1wb3J0Lm5kanNvbiIKICAgIH0KXQ==",
        "import.fhir.storagetype": "aws-s3"
    }

curl -k -v -u "fhiruser:change-password" -H 'Content-Type: application/fhir+json' 'https://localhost:9443/fhir-server/api/v4/$import' -d '{
"resourceType": "Parameters",
"id": "30321130-5032-49fb-be54-9b8b82b2445a",
"parameter": [
{
"name": "inputFormat",
"valueString": "application/fhir+ndjson"
},
{
"name": "inputSource",
"valueUri": "https://localhost:9443/source-fhir-server"
},
{
"name": "input",
"part": [
{
"name": "type",
"valueString": "Patient"
},
{
"name": "url",
"valueUrl": "https://localhost:9000/fhirbulkdata/test-import.ndjson"
}
]
},
{
"name": "storageDetail",
"valueString": "aws-s3"
}
]
}'

curl -k -v -u "fhiruser:change-password" -H 'Content-Type: application/fhir+json' 'https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=Zhkno6biH1Qwh2S6VDBT6w'

Additional context
Related to #1854

@prb112 prb112 added the bug Something isn't working label Jan 8, 2021
@prb112 prb112 added this to the Sprint 2021-01 milestone Jan 8, 2021
@prb112 prb112 self-assigned this Jan 8, 2021
prb112 added a commit that referenced this issue Jan 8, 2021
- Removed the duplicate serialization of the fhir.dataSourcesInfo

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>
prb112 added a commit that referenced this issue Jan 8, 2021
Duplicate Job Parameters fhir.dataSourcesInfo are created #1855
JohnTimm added a commit that referenced this issue Jan 21, 2021
* ci: introduce integration tests for fhir-audit

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* ci: introduce integration tests for fhir-audit

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* ci: add integration tests for fhir-audit feature and fix one bug with use of/from

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: pseudo tty

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: small change to the timeout length to 120s

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: alternative method for getting the results from the kafka-1 container

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: alternative method for getting the results from the kafka-1 container

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: add get_results.sh creation of the directory

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: update

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: update

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: for privileged execution

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: difference running ci local and remote

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: audit

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: audit with docker copy

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: adding tty support and stdin support

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: work around tty issue

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* removing the tty references -it

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* changed the execution pattern

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* changed the execution pattern

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: update to predefine output file

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: update to predefine output file

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix permissions

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix permissions

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix permissions

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix permissions

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix permissions

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix permissions

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* issues #1839 and #1743 - support search parameter disambiguation

1. Update ParametersMap to support storing multiple search parameters
with the same code
2. Address #1743 by collecting to a map instead of a list
3. Update SearchUtil.getSearchParameter to lookup the search parameter
by URI from the config if possible (instead of applying the filter to
the full set of built-in parameters).
4. Update the docs to reflect that search parameter filtering now
applies to tenant-specific search parameters as well. This should help
us move toward #1596

Also fixed a bad trace message and did some minor formatting / javadoc.

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>

* ci: work around issue with tty

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: test that doesn't account for year shifts

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: test that doesn't account for year shifts

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* ci: work around issue with tty

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* remove hardcoded year from SearchLastUpdatedIdTest

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>

* fix: change the integration pattern slightly for tty

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: change the integration pattern slightly for tty

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: change the integration pattern slightly for tty

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* fix: last two tests to update with dynamic year

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* issues #1839 and #1743 - support search parameter disambiguation

1. Update ParametersMap to support storing multiple search parameters
with the same code
2. Address #1743 by collecting to a map instead of a list
3. Update SearchUtil.getSearchParameter to lookup the search parameter
by URI from the config if possible (instead of applying the filter to
the full set of built-in parameters).
4. Update the docs to reflect that search parameter filtering now
applies to tenant-specific search parameters as well. This should help
us move toward #1596

Also fixed a bad trace message and did some minor formatting / javadoc.

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>

* Apply suggestions from code review

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>

* Update build/audit/README.md

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

Co-authored-by: Lee Surprenant <lmsurpre@us.ibm.com>

* add info on accessing the bulk operation job logs

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>

* Add unit tests for the ParametersMap

Also made a minor change to insertAll so it gets the code from the
existing ParametersMap instead of from the SearchParameters in the map.
Usually these are the same, but they can differ.

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>

* Issue #1849 - AuthZ interceptor validate/convert search requests

Signed-off-by: Mike Schroeder <mschroed@us.ibm.com>

* Issue #1849 - address review comments

Signed-off-by: Mike Schroeder <mschroed@us.ibm.com>

* Duplicate Job Parameters fhir.dataSourcesInfo are created #1855

- Removed the duplicate serialization of the fhir.dataSourcesInfo

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* Confusing error when request is targetted for an invalid tenant id #1792

- Disambiguates the Error Messages that are bubbled up through the
Persistence Layer
- Add Integration Test

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* Confusing error when request is targetted for an invalid tenant id #1792

1 - added layer at the rest level

Signed-off-by: Paul Bastide <pbastide@us.ibm.com>

* Issue #1615 - Enforce configured interactions in REST layer

Signed-off-by: Mike Schroeder <mschroed@us.ibm.com>

* Update FHIRValidationGuide.md

1. update the version references for the packaged implementation guides
2. add a section at the top to describe where to get the validation module

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>

* Issue #1615 - address review comments

Signed-off-by: Mike Schroeder <mschroed@us.ibm.com>

* Issue #1615 - fix whitespace

Signed-off-by: Mike Schroeder <mschroed@us.ibm.com>

* Issue #1615 - add enum for interaction types

Signed-off-by: Mike Schroeder <mschroed@us.ibm.com>

* Modify davinci-pdex CapabilityStatement-pdex-server.json

updated the searchRevInclude value for the Coverage resource in
pdex-server to work around https://jira.hl7.org/browse/FHIR-30338

Signed-off-by: Lee Surprenant <lmsurpre@us.ibm.com>

* Issue #1494 - add Bundle.entry.search to search results

Signed-off-by: Mike Schroeder <mschroed@us.ibm.com>

* Issue #1494 - add documentation

Signed-off-by: Mike Schroeder <mschroed@us.ibm.com>

* Issue #1494 - address review comments

Signed-off-by: Mike Schroeder <mschroed@us.ibm.com>

Co-authored-by: Paul Bastide <pbastide@us.ibm.com>
Co-authored-by: Lee Surprenant <lmsurpre@us.ibm.com>
Co-authored-by: Mike Schroeder <mschroed@us.ibm.com>
Co-authored-by: Michael W Schroeder <66479070+michaelwschroeder@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant