Missing extension in the parquet file compared to source FHIR resource.

[vbothe23]

Extensions in the resources are dropped while transforming FHIR resource into parquet file.

Patient resource:

{
    "resourceType": "Patient",
    "id": "4b8d1d94-d77c-4bf0-b82f-0a73c8e966e4",
    "meta": {
        "versionId": "1",
        "lastUpdated": "2023-07-04T04:26:09.189+00:00",
        "source": "#ElxTECreuoOe7wN7"
    },
    "text": {
        "status": "generated",
        "div": "<div xmlns=\"http://www.w3.org/1999/xhtml\"><div class=\"hapiHeaderText\">TEST <b>PATIENT </b></div><table class=\"hapiPropertyTable\"><tbody><tr><td>Identifier</td><td>P123456</td></tr><tr><td>Address</td><td><span>Abc </span><br/><span>Oyo </span></td></tr><tr><td>Date of birth</td><td><span>02 July 1991</span></td></tr></tbody></table></div>"
    },
    "extension": [
        {
            "url": "http://hl7.org/fhir/StructureDefinition/patient-religion",
            "valueCoding": {
                "system": "http://terminology.hl7.org/CodeSystem/v3-ReligiousAffiliation",
                "code": "1013",
                "display": "Christian"
            }
        }
    ],
    "identifier": [
        {
            "system": "http://iprdgroup.com/identifiers/patient-card",
            "value": "P123456"
        }
    ],
    "name": [
        {
            "family": "PATIENT",
            "given": [
                "TEST"
            ]
        }
    ],
    "telecom": [
        {
            "system": "phone",
            "value": "+91-989898989898"
        }
    ],
    "gender": "female",
    "birthDate": "1991-07-02",
    "address": [
        {
            "line": [
                "Abc"
            ],
            "district": "Afijio",
            "state": "Oyo"
        }
    ],
    "maritalStatus": {
        "coding": [
            {
                "system": "http://terminology.hl7.org/CodeSystem/v3-MaritalStatus",
                "code": "M",
                "display": "Married"
            }
        ],
        "text": "Married"
    },
    "contact": [
        {
            "relationship": [
                {
                    "coding": [
                        {
                            "code": "father",
                            "display": "Father"
                        }
                    ],
                    "text": "Father"
                }
            ],
            "name": {
                "family": "LNAME",
                "given": [
                    "FNAME"
                ]
            },
            "telecom": [
                {
                    "system": "phone",
                    "value": "+91-0870707007080"
                }
            ]
        }
    ]
}

Corresponding content in the parquet file:

{
    "id": "82652",
    "meta": {
        "versionId": "1",
        "lastUpdated": "2023-07-04T04:26:09.000+00:00"
    },
    "text": {
        "status": "generated"
    },
    "identifier": {
        "array": [
            {
                "system": "http://iprdgroup.com/identifiers/patient-card",
                "value": "P123456"
            }
        ]
    },
    "name": {
        "array": [
            {
                "family": "PATIENT",
                "given": {
                    "array": [
                        "TEST"
                    ]
                }
            }
        ]
    },
    "telecom": {
        "array": [
            {
                "system": "phone",
                "value": "+91-989898989898"
            }
        ]
    },
    "gender": "female",
    "birthDate": "1991-07-02",
    "address": {
        "array": [
            {
                "line": {
                    "array": [
                        "Abc"
                    ]
                },
                "district": "Afijio",
                "state": "Oyo"
            }
        ]
    },
    "maritalStatus": {
        "coding": {
            "array": [
                {
                    "system": "http://terminology.hl7.org/CodeSystem/v3-MaritalStatus",
                    "code": "M",
                    "display": "Married"
                }
            ]
        },
        "text": "Married"
    },
    "contact": {
        "array": [
            {
                "relationship": {
                    "array": [
                        {
                            "coding": {
                                "array": [
                                    {
                                        "code": "father",
                                        "display": "Father"
                                    }
                                ]
                            },
                            "text": "Father"
                        }
                    ]
                },
                "name": {
                    "family": "LNAME",
                    "given": {
                        "array": [
                            "FNAME"
                        ]
                    }
                },
                "telecom": {
                    "array": [
                        {
                            "system": "phone",
                            "value": "+91-0870707007080"
                        }
                    ]
                }
            }
        ]
    }
}

[bashir2]

Thanks @vbothe23 for filing this issue; this is a known problem and the ultimate fix is tracked under #560. That said, extensions are currently half-supported if you are willing to hard-code the extension URL and recompile the code. An example for this is described in PR #562.

[LovjeetVyas]

@bashir2 can u help me like in where is our data from fhir server is getting transformed in batch
i got some extended keys in my subject like group id, identifier, with nulls

[bashir2]

@bashir2 can u help me like in where is our data from fhir server is getting transformed in batch i got some extended keys in my subject like group id, identifier, with nulls

Sorry @LovjeetVyas for the late reply; can you please elaborate more on your issue? Are you saying that you have some FHIR extensions and you are not getting those values in the Parquet files? If that is the case, then this is expected behavior unless if you hardcode the extension URL (as described in #562). The fix for this is in #924 which @chandrashekar-s is working on and is expected to be merged this week.

[LovjeetVyas]

Hello @chandrashekar-s, @bashir2 I trust this message finds you well. I am reaching out to seek your assistance regarding a matter concerning the compatibility of Google FHIR Data Pipes with the Microsoft FHIR server. Our team has been conducting research and encountering challenges in establishing a connection between the two platforms. I would greatly appreciate your guidance on the requirements for connecting with the Microsoft FHIR server. Additionally, I am attaching a screenshot illustrating the specific issue we are currently facing. Your prompt response and insights would be highly valuable. Thank you, and I look forward to your reply.

BELOW IS THE COMMAND WE ARE USING:

PS D:\ccare\datapipes\fhir-data-pipes> java -jar ./pipelines/batch/target/batch-bundled.jar --fhirServerUrl=https://tenantfhirwrapperdata-tenantfhirservice.fhir.azurehealthcareapis.com --fhirServerOAuthTokenEndpoint=https://login.microsoftonline.com/xysd-sdasdan-xsdasd/oauth2/token --fhirServerOAuthClientId=fhune-s3434n3-sd3dgf45 --fhirServerOAuthClientSecret=uib3b4_34onin23 --resourceList=Patient --batchSize=10 --targetParallelism=2 --fhirSinkPath="" --sinkUserName="" --sinkPassword="" --outputParquetPath=/DATA_WAREHOUSE --jdbcModeEnabled=false --jdbcMaxPoolSize=50 --fhirDatabaseConfigPath=/workspace/utils/hapi-postgres-config.json --jdbcInitialPoolSize=10 --jdbcFetchSize=10000 --jdbcModeHapi=false --runner=DirectRunner --activePeriod="" --since=""

[above is the cli command we are using for connection'

13:07:49.074 [main] INFO com.google.fhir.analytics.FhirEtl com.google.fhir.analytics.FhirEtl.main:389 - Flags: Current Settings:
activePeriod:
appName: FhirEtl
batchSize: 10
fhirDatabaseConfigPath: /workspace/utils/hapi-postgres-config.json
fhirServerOAuthClientId: #############################
fhirServerOAuthClientSecret: #############################
fhirServerOAuthTokenEndpoint: https://login.microsoftonline.com/#######################/oauth2/token
fhirServerUrl: https://ccarefhirwrapperdata-ccarefhirservice.fhir.azurehealthcareapis.com
fhirSinkPath:
jdbcFetchSize: 10000
jdbcInitialPoolSize: 10
jdbcMaxPoolSize: 50
jdbcModeEnabled: false
jdbcModeHapi: false
optionsId: 0
outputParquetPath: /DATA_WAREHOUSE
resourceList: Patient
runner: class org.apache.beam.runners.direct.DirectRunner
since:
sinkPassword:
sinkUserName:
stableUniqueNames: WARNING
targetParallelism: 2

13:07:49.119 [main] INFO ca.uhn.fhir.util.VersionUtil ca.uhn.fhir.util.VersionUtil.initialize:84 - HAPI FHIR version 6.6.2 - Rev 8d55781507
13:07:49.127 [main] INFO ca.uhn.fhir.context.FhirContext ca.uhn.fhir.context.FhirContext.:210 - Creating new FHIR context for FHIR version [R4]
13:07:49.200 [main] INFO c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:371 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/profile/profiles-resources.xml
13:07:49.225 [main] INFO ca.uhn.fhir.util.XmlUtil ca.uhn.fhir.util.jar.DependencyLogImpl.logStaxImplementation:74 - FHIR XML procesing will use StAX implementation 'null' version 'null'
13:07:51.619 [main] INFO c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:371 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/profile/profiles-types.xml
13:07:51.727 [main] INFO c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:371 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/profile/profiles-others.xml
13:07:51.935 [main] INFO c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:371 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/extension/extension-definitions.xml
13:07:52.736 [main] INFO com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.:82 - Fetching access tokens from https://login.microsoftonline.com/2466764f-5f33-4d63-be86-052226a72c9a/oauth2/token
13:07:53.567 [main] INFO c.g.fhir.analytics.FhirSearchUtil com.google.fhir.analytics.FhirSearchUtil.createSegments:221 - Fetching first batch of Patient
13:07:54.383 [main] ERROR com.google.fhir.analytics.FhirEtl com.google.fhir.analytics.FhirEtl.buildFhirSearchPipeline:141 - Either the date format in the active period is wrong or none of the resources support 'date' featurekey expires_in
Exception in thread "main" java.lang.IllegalArgumentException: key expires_in
at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:360)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:335)
at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:79)
at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:73)
at com.google.api.client.http.HttpResponse.parseAs(HttpResponse.java:460)
at com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:324)
at com.google.fhir.analytics.FetchUtil$ClientCredentialsAuthInterceptor.requestAccessToken(FetchUtil.java:196)
at com.google.fhir.analytics.FetchUtil$ClientCredentialsAuthInterceptor.getToken(FetchUtil.java:172)
at com.google.fhir.analytics.FetchUtil$ClientCredentialsAuthInterceptor.interceptRequest(FetchUtil.java:188)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at ca.uhn.fhir.interceptor.executor.BaseInterceptorService$HookInvoker.invoke(BaseInterceptorService.java:517)
at ca.uhn.fhir.interceptor.executor.BaseInterceptorService.doCallHooks(BaseInterceptorService.java:281)
at ca.uhn.fhir.interceptor.executor.BaseInterceptorService.callHooks(BaseInterceptorService.java:269)
at ca.uhn.fhir.interceptor.executor.BaseInterceptorService.callHooks(BaseInterceptorService.java:63)
at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:319)
at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:538)
at ca.uhn.fhir.rest.client.impl.GenericClient$FetchConformanceInternal.execute(GenericClient.java:830)
at ca.uhn.fhir.rest.client.impl.RestfulClientFactory.validateServerBase(RestfulClientFactory.java:320)
at ca.uhn.fhir.rest.client.impl.RestfulClientFactory.validateServerBaseIfConfiguredToDoSo(RestfulClientFactory.java:283)
at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:245)
at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:538)
at ca.uhn.fhir.rest.client.impl.GenericClient$SearchInternal.execute(GenericClient.java:1997)
at com.google.fhir.analytics.FhirSearchUtil.createSegments(FhirSearchUtil.java:224)
at com.google.fhir.analytics.FhirEtl.buildFhirSearchPipeline(FhirEtl.java:139)
at com.google.fhir.analytics.FhirEtl.setupAndBuildPipelines(FhirEtl.java:377)
at com.google.fhir.analytics.FhirEtl.main(FhirEtl.java:395)
Caused by: java.lang.IllegalArgumentException: key expires_in, field private java.lang.Long com.google.api.client.auth.oauth2.TokenResponse.expiresInSeconds
at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:451)
at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:787)
... 29 more
Caused by: java.lang.IllegalArgumentException: number field formatted as a JSON string must use the @JsonString annotation at com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
at com.google.api.client.util.Preconditions.checkArgument(Preconditions.java:47)
at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:863)
... 31 more

[chandrashekar-s]

Hi @LovjeetVyas, sorry for the late response. The auth implementation that is followed OAuth2 standards and hence a successful auth response must receive a token in this format, i.e. the expires_in field must be a Long. Even the microsoft implementation follows the same.
Also, can you please create a new issue for this, as the context for this is different.

[chandrashekar-s]

Support for extensions has been provided with the PR #924, now the fhir profile structure definitions can be configured with the parameter structureDefinitionsDir and the pipelines will use these extensions during conversion to Parquet format. By default, the US Core profile FHIR extensions is supported.

One limitation with the current solution is that, if multiple fhir profiles are configured for the same resource type, then at any given point in time only one profile is applied. A more generic solution is being tracked as part of this issue #980

[LovjeetVyas]

hey @chandrashekar-s , sorry for this late response I was indulged in some activities and didnt got the time
but so far from our side to authenticate our request we made some changes for that string value that we were getting
and we were getting a url in resource field too that was required for authentication that was not in fhir-datapipes auth files so added that too
and its working fine
another issue that we faced was with pagination
casue ms fhir doesnt support the pagination as hapi fhir and none was specified in the repo and "fhirsearchutil" thus we made some changes into that too
now we are facing a problem with the extension field and currently trying to resolve that too

thankyou for your response and help earlier, if again i got stuck in something will definitely reach out to you guys