-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allowed properties overloading for CoreNLPParser tag, Authors.md #2789
base: develop
Are you sure you want to change the base?
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -326,7 +326,7 @@ def tokenize(self, text, properties=None): | |
for token in sentence["tokens"]: | ||
yield token["originalText"] or token["word"] | ||
|
||
def tag_sents(self, sentences): | ||
def tag_sents(self, sentences, properties=None): | ||
""" | ||
Tag multiple sentences. | ||
|
||
|
@@ -339,9 +339,11 @@ def tag_sents(self, sentences): | |
""" | ||
# Converting list(list(str)) -> list(str) | ||
sentences = (" ".join(words) for words in sentences) | ||
return [sentences[0] for sentences in self.raw_tag_sents(sentences)] | ||
if properties is None: | ||
properties = {"tokenize.whitespace": "true"} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I've merged develop into this PR. Because the CoreNLP tests now run on the CI, we can see that some tests fail due to this change. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @tomaarsen apart from automated tests for CoreNLP is there any other test that needs to be added There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The PR should have some tests that show that the new behaviour works like intended, so you would only need to add tests for the functionality that you added. So, just for these functions. |
||
return [sentences[0] for sentences in self.raw_tag_sents(sentences, properties)] | ||
|
||
def tag(self, sentence): | ||
def tag(self, sentence, properties=None): | ||
""" | ||
Tag a list of tokens. | ||
|
||
|
@@ -360,9 +362,9 @@ def tag(self, sentence): | |
('airspeed', 'NN'), ('of', 'IN'), ('an', 'DT'), | ||
('unladen', 'JJ'), ('swallow', 'VB'), ('?', '.')] | ||
""" | ||
return self.tag_sents([sentence])[0] | ||
return self.tag_sents([sentence], properties)[0] | ||
|
||
def raw_tag_sents(self, sentences): | ||
def raw_tag_sents(self, sentences, properties=None): | ||
""" | ||
Tag multiple sentences. | ||
|
||
|
@@ -377,8 +379,13 @@ def raw_tag_sents(self, sentences): | |
"annotators": "tokenize,ssplit,", | ||
} | ||
|
||
default_properties.update(properties or {}) | ||
|
||
# Supports only 'pos' or 'ner' tags. | ||
assert self.tagtype in ["pos", "ner"] | ||
assert self.tagtype in [ | ||
"pos", | ||
"ner", | ||
], "CoreNLP tagger supports only 'pos' or 'ner' tags." | ||
default_properties["annotators"] += self.tagtype | ||
for sentence in sentences: | ||
tagged_data = self.api_call(sentence, properties=default_properties) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a description for the
properties
argument in the docstring, also mention that"tokenize.whitespace"
is set toTrue
by default (but it won't ifproperties
are specified.