Possible error while parsing structured abstracts.

[RudrakshTuwani]

Hi, first of all big thanks for this life-saver of a package.

I think there is some problem with parsing XML for structured abstracts. Consider the following example:

         <Abstract>
          <AbstractText Label="" NlmCategory="UNASSIGNED">
            <b>Patient: Female, 16</b>
            <b>Final Diagnosis: Pelvic mass</b>
            <b>Symptoms: None</b>
            <b>Medication: None</b>
            <b>Clinical Procedure: CT • MRI</b>
            <b>Specialty: Diagnostic radiology • pediatrics.</b>
          </AbstractText>
          <AbstractText Label="OBJECTIVE" NlmCategory="OBJECTIVE">
            <b>Unusual presentation of unknown etiology, Rare disease, Mistake in diagnosis.</b>
          </AbstractText>
          <AbstractText Label="BACKGROUND" NlmCategory="BACKGROUND">Müllerian anomalies encompass a wide variety of malformations in the female genital tract, usually associated with renal and anorectal malformations. Of these anomalies, approximately 11% are uterus didelphys, which occurs when midline fusion of the müllerian ducts is arrested to a variable extent.</AbstractText>
          <AbstractText Label="CASE REPORT" NlmCategory="METHODS">We report the case of a 16-year-old female with uterine didelphys, jejunal malrotation, hematometra, hematosalpinx, and bilateral subcentimeter homogenous circular cystic-like renal lesions, who initially presented with left lower quadrant abdominal pain, non-bloody vomiting, and a history of irregular menstrual periods. Initial CT was confusing for an adnexal cystic mass, but further imaging disclosed the above müllerian anomalies.</AbstractText>
          <AbstractText Label="CONCLUSIONS" NlmCategory="CONCLUSIONS">Müllerian anomalies may mimic other, more common, adnexal lesions; thus, adequate evaluation of suspicious cystic adnexal masses with multiple and advanced imaging modalities such as MRI is essential for adequate diagnosis and management.</AbstractText>
        </Abstract>

The parse returned by medline_parser is as follows:

'Patient: Female, 16\n            Final Diagnosis: Pelvic mass\n            Symptoms: None\n            Medication: None\n            Clinical Procedure: CT \u2022 MRI\n            Specialty: Diagnostic radiology \u2022 pediatrics.'

As you can see, it completely misses a major portion of the text. I wonder if this is the case for all structured abstracts or only limited ones. As additional info, the file I'm using is ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/medline17n0763.xml.gz and the PMID of the abstract is 23826455.

Thanks!

[titipata]

Thanks for reporting @RudrakshTuwani! Is there anyway that you can upload XML somewhere so that I can try to fix the parser?

[RudrakshTuwani]

See the updated description.

[titipata]

That's perfect. I'll fix that by the weekend. Bug me if I forget tho!

[RudrakshTuwani]

No problem, thanks a lot!

Do you think this is the case for all structured abstracts or it's specific to a few?

[titipata]

There are a lot of abstract like this. I haven't calculated the exact number. I guess it should be roughly a million from full Medline dataset.

[RudrakshTuwani]

Okay, thanks!

[RudrakshTuwani]

I seem to have found the bug.

if article.find('Abstract/AbstractText') is not None:
        abstract = stringify_children(article.find('Abstract/AbstractText')).strip() or ''
elif article.find('Abstract') is not None:
        abstract = stringify_children(article.find('Abstract')).strip() or ''
else:
        abstract = ''

So, the first if block matches the first tag under structured abstract and returns it. This can be solved by removing the condition for matching "AbstractText". Can you tell me why we are even matching AbstractText? Since, just matching on Abstract also returns all the relevant information.

[titipata]

Yes @RudrakshTuwani, you're right. I'm fixing that part now.

[titipata]

I fix it in commit b59e150, let me know if it works for you!

[RudrakshTuwani]

Yes, this seems to be working. Thanks a lot!

One suggestion for future, maybe we can add the corresponding field for the structured abstract in the beginning of the sentence. If you want, I can work on it and send a PR later.

[titipata]

Haha, you read my mind! I did that add just push it, see commit 34dae58

[RudrakshTuwani]

Haha, yes this seems to be working perfectly! Thanks a lot, man!

[titipata]

@RudrakshTuwani, sorry, I still see the problem in my parser atftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/medline17n0843.xml.gz. I'll fix it and close the issue soon.

[RudrakshTuwani]

Oh, can you send me the PMID? I'll have a look as well.

[titipata]

This is 26368793, so now I parsed at got multiple METHODS section as

OBJECTIVE
To examine interleukin-12 (IL-12), IL-18, IFN-γ, intracellular adhesion molecule-1 (ICAM-1), leukemia inhibitory factor (LIF), and migration inhibitory factor (MIF) levels in precisely-timed blood and endometrial tissue samples from women with idiopathic recurrent pregnancy loss (RPL).


METHODS
Case-control study.


METHODS
University hospital.

It should be a better way to concatenate all the same section in one long string.

[titipata]

We are actually doing it correctly, you can see at: https://www.ncbi.nlm.nih.gov/pubmed/?term=26368793. However, I chose to get abstract.attrib.get('NlmCategory', '') instead of says Label="BACKGROUND". Still thinking what is the best way to do that. Label will be good for someone who wants to display the abstract. However, NlmCategory is also good if you want to make each sections consistent.

[RudrakshTuwani]

Yes, there's a dilemma here. Maybe we should simply use Label because` that's the way it's displayed on the website as well?

Or do something like NLMCategory - Label: and reduce it to just one of them in case of repetitions like OBJECTIVE - OBJECTIVE:

What do you think?

[titipata]

So I fixed it in f3fe97f as

dicts_out = pp.parse_medline_xml('data/medline16n0902.xml.gz', year_info_only=False, nlm_category=False)

If you set nlm_category=True, it will give result using NLM category instead.

[RudrakshTuwani]

Okay, I fixed it in a slightly different way. The output I get is the following:

OBJECTIVE: Unusual presentation of unknown etiology, Rare disease, Mistake in diagnosis. 

BACKGROUND: Müllerian anomalies encompass a wide variety of malformations in the female 
genital tract, usually associated with renal and anorectal malformations. Of these anomalies, 
approximately 11% are uterus didelphys, which occurs when midline fusion of the müllerian ducts is 
arrested to a variable extent. 

METHODS "CASE REPORT": We report the case of a 16-year-old female with uterine didelphys, 
jejunal malrotation, hematometra, hematosalpinx, and bilateral subcentimeter homogenous circular 
cystic-like renal lesions, who initially presented with left lower quadrant abdominal pain, non-bloody 
vomiting, and a history of irregular menstrual periods. Initial CT was confusing for an adnexal cystic 
mass, but further imaging disclosed the above müllerian anomalies. 

CONCLUSIONS: Müllerian anomalies may mimic other, more common, adnexal lesions; thus, 
adequate evaluation of suspicious cystic adnexal masses with multiple and advanced imaging 
modalities such as MRI is essential for adequate diagnosis and management.

Label if not equal to NLM category are added in quotation marks in front of the NLMCategory.

I guess we can close this now.

[titipata]

Definitely, we can close it for now. I'll discuss with @daniel-acuna a bit more about this issue. Let me know there is any use cases that you want output to be in particular format. Thanks again for the issue tho!

[RudrakshTuwani]

The check for UNASSIGNED is still failing. Replacing 'is not' by != fixes it. Also, we want to remove the content under UNASSIGNED section right? In which case, we may want to put abstract_list.append(stringify_children(abstract).strip()) inside the if condition.

No problem! Thanks for being so prompt on this. Let me know if I can contribute in any other way.

[titipata]

Fixed is not to !=, I amend the commit!

I think putting abstract_list.append(stringify_children(abstract).strip()) outside is correct cause we always add text into the list. However, if section is UNASSIGNED, I just skip it :)

[RudrakshTuwani]

Okay, that works too :)