Skip to content

Assertion status annotator #93

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 58 commits into from
Jan 27, 2018
Merged

Assertion status annotator #93

merged 58 commits into from
Jan 27, 2018

Conversation

albertoandreottiATgmail
Copy link
Contributor

@albertoandreottiATgmail albertoandreottiATgmail commented Jan 26, 2018 •
edited by saif-ellafi
Loading

Description

With this PR we introduce the Assertion Status Annotator.

Motivation and Context

New feature.

How Has This Been Tested?

Tested on real data both at mode and pipeline levels.

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Code improvements with no or little impact
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING page.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@albertoandreottiATgmail
added unit test for windowing - work in progress
6b72c4e
@albertoandreottiATgmail
added script for converting negex dataset to CSV
ddd5a32
@albertoandreottiATgmail
work in progress - fixed problem with embeddings
e7f2b96
@albertoandreottiATgmail
some cleanup
8754a95
@albertoandreottiATgmail
cleanup
2601e8b
@albertoandreottiATgmail
i2b2 reader
dc3f4c4
@albertoandreottiATgmail
fixed problem with windows
fdca19b
@albertoandreottiATgmail
enhancements in i2b2 reader
9a06fbc
@albertoandreottiATgmail
added case for windowing unit test
4dd89f6
@albertoandreottiATgmail
work in progress assertion model in the pipeline
4b60b1b
@albertoandreottiATgmail
work in progress
427fdbd
@albertoandreottiATgmail
modified tokenizer to match the one using in embeddings
29319d2
@albertoandreottiATgmail
work in progress
01973f9
@albertoandreottiATgmail
work in progress
d7c29d8
@albertoandreottiATgmail
deleted negex dataset reader
1e3b516
@albertoandreottiATgmail
deleted negex dataset reader
e28b52a
@albertoandreottiATgmail
little cleanup in test
f807856
@albertoandreottiATgmail
added tokenizers
a274039
@albertoandreottiATgmail
behavior for some tokenizers
bd8e614
@albertoandreottiATgmail
refactor in windowing code
fd99273
@albertoandreottiATgmail
some cleanup in test and comments
9cf8c4e
@albertoandreottiATgmail
work in progress
b72db86
@albertoandreottiATgmail
added commons evaluation metrics class
9f53eab
@albertoandreottiATgmail
some cleanup
8a035dd
@albertoandreottiATgmail
Merge branch 'word_embeddings' into assertion_status
5f1eea7 
src/main/scala/com/johnsnowlabs/nlp/embeddings/AnnotatorWithWordEmbeddings.scala
src/main/scala/com/johnsnowlabs/nlp/embeddings/ModelWithWordEmbeddings.scala
@albertoandreottiATgmail
corrections in tests, annotations, and labels
935b49b
@albertoandreottiATgmail
restored complete i2b2 dataset
4a7fd43
@albertoandreottiATgmail
code for before & after parameters in model
d6049a9
@albertoandreottiATgmail
cosmetic
f319eca
@albertoandreottiATgmail
cosmetic
26f4366
@albertoandreottiATgmail
cosmetic
fbb8106
@albertoandreottiATgmail
minor changes
d2b8dc8
@albertoandreottiATgmail
implemented simple version of the regex tokenizer
35c57e5
@albertoandreottiATgmail
added html documentation for assertion status
abc4d43
@albertoandreottiATgmail
work in progress for notepad
705ae69
@albertoandreottiATgmail
refactor to include Dataset interface in models
fdc59b8
@albertoandreottiATgmail
added reader for negex dataset
e6dfcdb
@albertoandreottiATgmail
added jupyter notebook for assertion status
aa195e6
@albertoandreottiATgmail
some changes to make parameter names match in notebook
51cdf09
@albertoandreottiATgmail
added parquet version of negex dataset for notebook
6e640c4
@albertoandreottiATgmail
added test cases for negex dataset
b2baa0e
@albertoandreottiATgmail
fixed problem with parameters
fc617de
@albertoandreottiATgmail
removed some hard-coded params
748b1e6
@albertoandreottiATgmail
minor changes in parameters
9512e45
@albertoandreottiATgmail
Merge remote-tracking branch 'origin/flexible-word-embeddings' into a…
001a842 
…ssertion_status
@albertoandreottiATgmail
refactor to avoid embeddings serialization
c207608
@albertoandreottiATgmail
missing file
7ce496c
@albertoandreottiATgmail
fixes for serialization
35eb023
@albertoandreottiATgmail
transient lazy pattern
a5132af
@albertoandreottiATgmail
cleanup
a447606
@albertoandreottiATgmail
removed embeddings logic from RawAnnotator
bf944d0
@saif-ellafi
Copy link
Contributor

Please fix python notebook example and we are good to go,

although there are a few things I would like to change sometime:

  1. see if we can better use tokenizer and document assembler in the model
  2. change parquet labelMap into Feature[Map[String, Double]] It will be faster to use binary format than parquet
  3. use spark.read.parquet instead of sqlContext
  4. help me understand windowing trait some day :) It confuses me quite. Also, any chance for any other type of tokenization process might be involved?

saif-ellafi and others added 6 commits January 27, 2018 12:37
@saif-ellafi
- New tokenizer wrap up
8e364e0
@albertoandreottiATgmail
unit test work in progress
214873a
@saif-ellafi
- Fixed bug in word embeddings write process
4d7d973 
- Updated LogReg to use read and write standard traits
@albertoandreottiATgmail
unit test
3b3cad0
@albertoandreottiATgmail
removed hard-coded path
13cc10c
@albertoandreottiATgmail
Merge remote-tracking branch 'origin/assertion-serialization-improvem…
f29a4ca 
…ents' into assertion_status

src/main/scala/com/johnsnowlabs/nlp/annotators/assertion/logreg/AssertionLogRegModel.scala
@saif-ellafi saif-ellafi merged commit 2507dc6 into master Jan 27, 2018
@maziyarpanahi maziyarpanahi deleted the assertion_status branch March 29, 2021 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@albertoandreottiATgmail @saif-ellafi