Closes #61 [DEPRECATED]

[MFreidank]

Closes issue #61

Relevant details:

• Name: SPL ADR 200 DB
• Description: SPL ADR 200 DB is a dataset for adverse event mentions annotated at entity level in Structured Product Labels of 200 FDA-approved drugs. Annotations were done as part of a partnership between
  the United States Food and Drug Administration (FDA) and the National Library of Medicine.
  These data were used for the adverse event challenge of TAC (Text Analysis Conference) 2017, see also:
  https://bionlp.nlm.nih.gov/tac2017adversereactions/
• Paper: https://www.researchgate.net/publication/322810855_A_dataset_of_200_structured_product_labels_annotated_for_adverse_drug_reactions
• Data: https://bionlp.nlm.nih.gov/tac2017adversereactions/

Checkbox

• Confirm that this PR is linked to the dataset issue.
• Create the dataloader script biodatasets/my_dataset/my_dataset.py (please use only lowercase and underscore for dataset naming).
• Provide values for the _CITATION, _DATASETNAME, _DESCRIPTION, _HOMEPAGE, _LICENSE, _URLs, _SUPPORTED_TASKS, _SOURCE_VERSION, and _BIGBIO_VERSION variables.
• Implement _info(), _split_generators() and _generate_examples() in dataloader script.
• Make sure that the BUILDER_CONFIGS class attribute is a list with at least one BigBioConfig for the source schema and one for a bigbio schema.
• Confirm dataloader script works with datasets.load_dataset function.
• Confirm that your dataloader script passes the test suite run with python -m tests.test_bigbio biodatasets/my_dataset/my_dataset.py.
• If my dataset is local, I have provided an output of the unit-tests in the PR (please copy paste). This is OPTIONAL for public datasets, as we can test these without access to the data files.

Please let me know in case any changes are required. Looking forward to your comments!

[WojciechKusa]

Hi @MFreidank. The code looks great, thank you for your contribution! 🎉

I have noticed that ADR dataset also contains unannotated test data split.

If the schema is similar to the trainset, it would be nice to add it to the loader as data in a split called "unannotated". What do you think?

[MFreidank]

Hi @WojciechKusa

Thank you for your review and encouraging feedback.

I also noticed the additional split, but when I had tried implementing it, I got a unittest-time error for the subset spl_adr_200db_unannotated_bigbio_kb as it requires at least one entity (and as the data is unlabeled, I don't have any entity labels).

At a high level I thought of these options:

• Have two source subsets, one for train and one for unannotated splits and one single bigbio_kb schema that uses only the train split data.
• Implement a source and bigbio subset for train and unannotated respectively and accept that the test for bigbio_unannotated will fail.
• Use a different bigbio schema than bigbio_kb for the unannotated split (could you suggest one, I had a quick look but didn't see anything stick out immediately).
• Implement a single train subset for both (current implementation)

Please let me know if there's a better solution or which option you'd like me to implement and I'll make the necessary changes right away.

[WojciechKusa]

Thanks for the quick reply @MFreidank.

I would vote for option (2) and allow for failing unittest in unannotated split as we might relax this requirement after the hackathon. Alternatively, I would only implement the unannotated schema for source (option 1).

However, it would be great to get a second opinion @galtay @hakunanatasha?

[WojciechKusa]

Hi @MFreidank,
We have a fix coming to the unit-tests that will allow for bypassing specific arguments. This will help with the case of empty entities.

So we agreed that the best way would be to go with option (2) and, for the moment, let bigbio_unannotated fail:

Implement a source and bigbio subset for train and unannotated respectively and accept that the test for bigbio_unannotated will fail.

Please let us know if you have time to implement this :)

[MFreidank]

Hi @WojciechKusa
Thank you for this update!

I'll get to working on the changes to implement option (2) right away.

Update Changes are now implemented (turned out to be a pretty small change, as schemas and generator code are unaffected).

I've run unit tests, this test passes for me:

python -m tests.test_bigbio --subset_id "spl_adr_200db_train" biodatasets/spl_adr_200db/spl_adr_200db.py

While this one fails as expected as the unannotated data does not return entities:

python -m tests.test_bigbio --subset_id "spl_adr_200db_unannotated" biodatasets/spl_adr_200db/spl_adr_200db.py

Please let me know if the changes look good to you.

Hmm, I may have rushed and it seems I accidentally merged in some commits that weren't intended. Should I close this PR and reopen with a single clean commit?

[WojciechKusa]

Thanks @MFreidank, looks good to me, I am happy to merge it!

I agree, let's open a new PR just to make sure that we commit only this dataset.

[MFreidank]

Okay, closing this PR and re-opening a separate one with only this dataset.
Will tag you directly when it is open so we can proceed to merging @WojciechKusa