Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

trouble parsing an HL7 message with a repeating group #325

Closed
ppc001 opened this issue Oct 14, 2021 · 25 comments
Closed

trouble parsing an HL7 message with a repeating group #325

ppc001 opened this issue Oct 14, 2021 · 25 comments
Assignees
@yankunhuang-pku @ginalee-dotcom

Comments

@ppc001
Copy link

ppc001 commented Oct 14, 2021

I am trying to parse an HL7 ORU^R01 message with a repeating group. The message has the following format:

MSH
PID
[{NTE}]
[{NK1}]
PV1
{
ORC
OBR
[{NTE}]
{
[
OBX
[{NTE}]
]
}
}

MSH|^&|BKR|NHCIEBC|||20210927003525|LABBACKGROUND|ORU^R01|38335214|P|2.3|||||||||PHLabReport-Ack^^2.16.840.1.114222.4.10.3^ISO||
PID|1||59999999^^^NOVANT^CI||JXXXXX^TXXXX^CXXXXXX||19010101|F||2|200 HAWTHORNE LANE^^CHARLOTTE^NC^28204^USA^P^^MECKLENBUR||(999)999-9999^PRN^PH^^1^999^9999999^NET^Internet^none@noemail.com||ENGLISH|||999999999999|999999999|||Not Hispanic||||||||N|||20190617163909|1||PID|1||59999999^^^NOVANT^CI59999999^^^NOVANT^CI||JXXXXX^TXXXX^CXXXXXX^JXXXXX^TXXXX^CXXXXXX||19010101|F||2|200 HAWTHORNE LANE^^CHARLOTTE^NC^28204^USA^P^^MECKLENBUR||(999)999-9999^PRN^PH^^1^999^9999999^NET^Internet^none@noemail.com||ENGLISH|||999999999999|999999999|||Not Hispanic||||||||N|||20190617163909|1|
NTE|1|O|Signature - Acct: 99999999|
NTE|2|O|Allscripts - Gill Ortho, ChartID: 9999 EHR Mrn: 99999999|
NK1|1|ALMOND^CHRIS^^|Son|6225 SWIMMING DUCK RD^^CHARLOTTE^NC^28214^USA|(704)999-9999^^CP^^^704^9999999|||||||||||||||||||||||||||
NK1|2|BERRY^TAI^^|Daughter-in-|6225 SWIMMING DUCK RD^^CHARLOTTE^NC^28214^USA|(704)999-9999^^PH^^^704^9999999~(704)988-8888^^CP^^^704^9888888|||||||||||||||||||||||||||
PV1|1|OC|NHCIEBC^^^NHCIEBC^^^^^^^|||||9999999999^A^KETAN^B^^^^^^^^^NPI|||||||||||999999999999|Medicare Man||||||||||||||||||||||||20210923091309|20210923111059|||||8400112385
ORC|RE|936013155||21LCA-266S0345^Beaker|||^^^20210927^^R|935055050^56102788&Beaker|20210927003519|INFBLCRI^ACUTE INTERFACE^INCOMING^BEAKER LC RES^||8888888888^S^KIM^S^^^^^^^^^NPI|NHCIEBC^^^NHCIEBC^^^^^|(999)999-6500^^^^^999^9996500|20210927003519|||DKT2UA2490PL0^DKT2UA2490PL0^^100101001^LSC FAMILY MEDICINE||||||9 Queens^Ste 520^CHARLOTTE^NC^28204-3215^USA^C^^MECKLENBUR|||||||LAB304838^CULTURE, URINE^NH BEAKER EAP^^^^^^CULTURE, URINE
OBR|1|936013155||LAB3041225^URINE CULTURE RESULT^NH BEAKER EAP^^^^^^URINE CULTURE RESULT||20210927|20210923093255|||SES091^STARK^SHARON^E^|G||R30.0^Dysuria^ICD-10-CM^^^^^^Dysuria|20210923000000|Urine&Urine^^^Urine, Clean&Urine, Clean Catch|1376800698^STRICKLAND^KIMBERLY^S^^^^^^^^^NPI|(980)302-6500^^^^^980^3026500|||||20210927003500||Lab Only|F||^^^20210927^^R||935055050^56102788&Beaker|||&Acute Interface&Incoming&Beaker Lc Res&||||20210927||||||||||||||LAB304838^CULTURE, URINE^NH BEAKER EAP^^^^^^CULTURE, URINE
NTE|1|L|Performed at: 01 - LabCorp Burlington|
NTE|2|L|1447 York Court, Burlington, NC 272153361|
NTE|3|L|Lab Director: Sanjai Nagendra MD, Phone: 8007624344|
TQ1|1||||||||R
OBX|1|ST|10094^C URINE 1^NH BEAKER LRR^^^^^^C URINE 1||Klebsiella pneumoniae|||A|||F|||20210923093255|||||20210927003500||||NH LABCORP^D|SEE NOTES^^^^^^B||||||
NTE|1|L|Cefazolin <=4 ug/mL|
NTE|2|L|Cefazolin with an MIC <=16 predicts susceptibility to the oral agents|
NTE|3|L|cefaclor, cefdinir, cefpodoxime, cefprozil, cefuroxime, cephalexin,|
NTE|4|L|and loracarbef when used for therapy of uncomplicated urinary tract|
NTE|5|L|infections due to E. coli, Klebsiella pneumoniae, and Proteus|
NTE|6|L|mirabilis.|
NTE|7|L|50,000-100,000 colony forming units per mL|
OBX|2|ST|10098^C URINE SUS^NH BEAKER LRR^^^^^^C URINE SUS||Comment||||||F|||20210923093255|||||20210927003500||||NH LABCORP^D|SEE NOTES^^^^^^B||||||
NTE|1|L| ** S = Susceptible; I = Intermediate; R = Resistant **|
NTE|2|L| P = Positive; N = Negative|
NTE|3|L| MICS are expressed in micrograms per mL|
NTE|4|L| Antibiotic RSLT#1 RSLT#2 RSLT#3 RSLT#4|
NTE|5|L|Amoxicillin/Clavulanic Acid S|
NTE|6|L|Ampicillin R|
NTE|7|L|Cefepime S|
NTE|8|L|Ceftriaxone S|
NTE|9|L|Cefuroxime S|
NTE|10|L|Ciprofloxacin S|
NTE|11|L|Ertapenem S|
NTE|12|L|Gentamicin S|
NTE|13|L|Imipenem S|
NTE|14|L|Levofloxacin S|
NTE|15|L|Meropenem S|
NTE|16|L|Nitrofurantoin S|
NTE|17|L|Piperacillin/Tazobactam S|
NTE|18|L|Tetracycline S|
NTE|19|L|Tobramycin S|
NTE|20|L|Trimethoprim/Sulfa S|
SPM|1|||Urine^Urine^^^^^^^Urine||||Urine, Clean^Urine, Clean Catch^^^^^^^Urine, Clean Catch|||||||||20210923093255|20210923000000||||||

The messages has multiple OBX segments with multiple NTE segments following each OBX segment. The FHIR converter is only mapping the first OBX segment to an Observation resource. I need to map the NTE comments to the Observation.note.text FHIR attribute as well.

FYI: I am fairly new to the liquid markup language. Any help is greatly appreciate
ORU_R01_liquid_snippet
d.
@BoyaWu10
Copy link
Contributor

Hi @ppc001, could you please share more context of your question? Does the converter work fail to get the OBX segment or the NTE segments? BTW, could you please share the raw liquid template you created instead of the snapshot?

@ppc001
Copy link
Author

ppc001 commented Oct 20, 2021 via email

@ppc001
Copy link
Author

ppc001 commented Oct 20, 2021

ORU_R01.txt

@ppc001
Copy link
Author

ppc001 commented Oct 20, 2021 via email

@BoyaWu10
Copy link
Contributor

Got it. The current behavior of get_related_segment_list returns a series of consecutive segments following a specific segment. When we obtain the OBX segments which follow OBR segment with this filter, as they are not consecutive (each OBX is followed by several NTE segments), the filter only returns the first OBX segment.

Currently, the segment filters in converter lack the context of grouping and have limited functions. We will investigate more about how to support it in future.

As a workaround for this special case, we can use get_segment_lists to get the OBX segments instead. After that, we can get the NTE segments with get_related_segment_list, as they are consecutive.

{% assign obxSegmentLists = hl7v2Data | get_segment_lists: 'OBX' -%}
    {% for obxSegment in obxSegmentLists.OBX -%}
        {% assign nteSegmentLists = hl7v2Data | get_related_segment_list: obxSegment, 'NTE' -%}
        {% for nteSegment in nteSegmentLists.NTE -%}
            {% if nteSegment.1.Value == 1 -%}
                {% assign comments = nteSegment.3.Value -%}
            {% else %}
                {% assign comments = comments | append: "^" -%}
                {% assign comments = comments | append: nteSegment.3.Value -%}
            {% endif %}
        {% endfor -%}
...

@ydequin21
Copy link

I've noticed that after OBX, the function will search for an NTE segment(s) until it gets to one rather than only looking at the next line. As you can see below, there are multiple lines between OBX and NTE but it still gets converted and added as a comment.
ba2d
bad.

I have also attached a correct scenario for reference.
good2
good

Optimally, we would want the function to only look at one line after the OBX line. If the next few consecutive lines are NTE then they should be added as a comment and appended to each other but if it is anything other than NTE there should not be a comment at all.

The liquid code surrounding this issue is the same one you recommended which I have reattached and the ORU_R01.hl7 is still a valid file to reference.
Screenshot 2021-11-16 152833

@BoyaWu10 BoyaWu10 assigned yankunhuang-pku and unassigned BoyaWu10 Nov 17, 2021
@ydequin21
Copy link

Hey @yankunhuang-pku,

Wanted to check-in on the progress of this issue.

@yankunhuang-pku
Copy link
Contributor

We are implementing a new filter to solve this problem. I'll let you know if it's ready.

@ydequin21
Copy link

good afternoon,
checking on progress

@yankunhuang-pku
Copy link
Contributor

yankunhuang-pku commented Dec 13, 2021 •
edited
Loading

We plan to release it this week or next week if nothing goes wrong.

@ydequin21
Copy link

great to hear! do you have a more precise timeline at this time?

@yankunhuang-pku
Copy link
Contributor

We released a filter called split_data_by_segments for now. I think you can try using this filter to remove the confusing segments to solve this problem.

@ginalee-dotcom
Copy link
Contributor

Closing per Yankun's solution above. Please re-open if you have additional questions, thanks!

@ydequin21
Copy link

Thank you for creating the changes and releasing the filter @yankunhuang-pku.
I have been trying to use the filter to fix my current issue but I am not succeeding. I have attached two sample HL7v2 files that I am using to test the filter, bkr_998302 and bkr_998330_2.

For bkr_998302, the intended result would be to have the NTE segments(notes/comments) of line 23-32 in the sample data file to follow OBX |9| (Observation) from line 22 as seen in MSFT_2 below
MSFT_2

Currently, with the most recent version of the FHIR converter, the NTE segments follow every single OBX instances from line 14 through 22 plus an additional instance in the ServiceRequest resource group. (line 12).

We added the filter at the obrSegment level as you can see in MSFT_4 below on line 453. This deleted all nteSegments under Observations and only kept the nteSegments in the ServiceRequest resource group as seen in MSFT_3 below
MSFT_3
MSFT_4

For bkr_998330_2, the intended result is for the NTE segments to populate the correct Observation resources and for all the Observations to get converted.

With the most recent converter update, we have noticed that the notes/comments are populating the correct Observation resources as seen in MSFT_5 below. However, the following OBX and NTE segments (lines 14-42) end up getting lost and not converted. I believe this issue also stems from the NTE filtering that ends up ignoring OBXsegments that follow NTE segments.
MSFT_5

please advise on how we can achieve our intended results and move forward. Please let me know if further documentation would be necessary or helpful.

MSFT_issue.zip

@yankunhuang-pku
Copy link
Contributor

yankunhuang-pku commented Jan 20, 2022 •
edited
Loading

Are these tests based on the VS Code extension? At present, we can only use the command line tool in this project to try the new filter, which has not been released on the VS Code side.

Could you test these cases with the command line tool first?

@ppc001
Copy link
Author

ppc001 commented Jan 20, 2022

We tried testing the split_data_by_segments filter from the command line and got the following error

C:\tmp\Microsoft.Health.Fhir.Liquid.Converter.Tool>.\Microsoft.Health.Fhir.Liquid.Converter.Tool.exe convert -d C:\Users\ppc001\source\repos\fhir_liquidtemplates -r ORU_R01 -i c:\tmp\fhir_input -o c:\tmp\fhir_output
Processing c:\tmp\fhir_input\bkr_998302.hl7
Process failed: Error happened when rendering templates: Error - Filter 'split_data_by_segments' does not have a default value for 'segmentIdSeparators' and no value was supplied

Here is the code snippet that was called.

{% assign obxSegmentLists = hl7v2Data | split_data_by_segments 'OBX|NTE' -%}

Not sure if we did something wrong there

@ydequin21
Copy link

Are these tests based on the VS Code extension? At present, we can only use the command line tool in this project to try the new filter, which has not been released on the VS Code side.

Could you test these cases with the command line tool first?

Do you have a timeline for the VS Code release?
In future updates, will the VS Code release always be delayed? How long is that delay?
thank you

@yankunhuang-pku
Copy link
Contributor

We tried testing the split_data_by_segments filter from the command line and got the following error

C:\tmp\Microsoft.Health.Fhir.Liquid.Converter.Tool>.\Microsoft.Health.Fhir.Liquid.Converter.Tool.exe convert -d C:\Users\ppc001\source\repos\fhir_liquidtemplates -r ORU_R01 -i c:\tmp\fhir_input -o c:\tmp\fhir_output Processing c:\tmp\fhir_input\bkr_998302.hl7 Process failed: Error happened when rendering templates: Error - Filter 'split_data_by_segments' does not have a default value for 'segmentIdSeparators' and no value was supplied

Here is the code snippet that was called.

{% assign obxSegmentLists = hl7v2Data | split_data_by_segments 'OBX|NTE' -%}

Not sure if we did something wrong there

This case may be because you wrote it wrong. Missing symbol : after split_data_by_segments

@yankunhuang-pku
Copy link
Contributor

Are these tests based on the VS Code extension? At present, we can only use the command line tool in this project to try the new filter, which has not been released on the VS Code side.
Could you test these cases with the command line tool first?

Do you have a timeline for the VS Code release? In future updates, will the VS Code release always be delayed? How long is that delay? thank you

Yes, it will be released later than this tool. The specific delay time depends on other work.

@ppc001
Copy link
Author

ppc001 commented Jan 21, 2022 via email

@ydequin21
Copy link

ydequin21 commented Jan 21, 2022 •
edited
Loading

we have tested with command line and we are seeing the same issues. I have attached the input sample data files and some images detailing what we are seeing and what we wish to see. I have also attached what current outputs we currently see.
thank you for your assistance.

please re-open this issue thread
sample_data_outputs.zip

@yankunhuang-pku
Copy link
Contributor

@ppc001 @ydequin21 I think it may be the error of your template logic. Here I can provide some usage for your reference. You can try to write your template logic like this sample. Please note that you should use command line tool instead of VS Code extension.
image

If you have other questions, please provide the templates corresponding to your input data and outputs.

@ydequin21
Copy link

@yankunhuang-pku, thank you for looking into our previous issues.

We originally thought the issue above pertaining to bkr_998330 had to do with the new filter but the issue persists without the filter when using the most recent FHIR Converter template and resources.

I have included the ORU_R01.liquid template. it is your current most updated version without custom changes. The input(bkr_998330 input) has NTE under OBR, OBX |1|, and OBX |2|. In our JSON output we clearly see the note under Service Request(OBR) and one Observation (OBX |1|) but there is not another Observation (OBX |2|) so we lose the entire observation with all the notes that followed it.

I have also reattached the expected output for a clearer perspective on what the issue is.

Moving on with our project we are not using the latest upgrade because of these code-breaking issues.
Please look into this issue.
sample_data_outputs.zip

@yankunhuang-pku
Copy link
Contributor

yankunhuang-pku commented Jan 26, 2022 •
edited
Loading

The reason for this problem is that the templates are still old and the behavior is the same as before if you haven't modified the templates. We have not replaced the old filter get_related_segment_list with the new filter split_data_by_segments in the current templates.
@ginalee-dotcom This seems to be a template problem. Could you help to refine the templates? I think we can not use get_related_segment_list: obrsegment, 'OBX', but split the data based on the segments OBR and OBX by using the new filter split_data_by_segments.

@ginalee-dotcom
Copy link
Contributor

Closing this as we have the filter split_data_by_segments as a solution. Will be refining the templates to use this filter.

@DanPaseltiner DanPaseltiner mentioned this issue Jan 11, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yankunhuang-pku yankunhuang-pku

@ginalee-dotcom ginalee-dotcom

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@BoyaWu10 @ppc001 @yankunhuang-pku @ginalee-dotcom @ydequin21