[NER] Add support for Chinese Named Entities #2676
Conversation
|
Job PR-2676-f61dc77 is done. |
|
Job PR-2676-8759c75 is done. |
| @@ -35,6 +36,7 @@ def fit(self, y: pd.Series, x: pd.Series): | |||
| _, entity_groups = self.extract_ner_annotations(y) | |||
| self.unique_entity_groups = self.ner_special_tags + entity_groups | |||
| self.entity_map = {entity: index for index, entity in enumerate(self.unique_entity_groups)} | |||
| self.config.entity_map = self.entity_map | |||
There was a problem hiding this comment.
Do we need to put entity_map under self.config? Or we just need to keep self.entity_map?
There was a problem hiding this comment.
need this entity map in dataprocessor, that's why it is put in config. special tags such as "O" is also in config.
There was a problem hiding this comment.
Shall we extend theNerProcessor to include the entity_map keyword then?
There was a problem hiding this comment.
class NerProcessor:
"""
Prepare NER data for the model specified by "prefix".
"""
def __init__(
self,
model: nn.Module,
max_len: Optional[int] = None,
entity_map: Optional[dict] = None,
config: Optional[DictConfig] = None,
):There was a problem hiding this comment.
already add self.config in nerprocessor. entity_map can be accessed by self.config.entity_map.
|
Job PR-2676-eaf21d6 is done. |
| for annot in ner_annotations: | ||
| custom_offset = annot[0] | ||
| custom_label = annot[1] | ||
| b_prefix = "B-" |
There was a problem hiding this comment.
Need to add "entity_map" in the docstring:
def process_ner_annotations(ner_annotations, ner_text, entity_map, tokenizer, is_eval=False):
"""
Generate token-level/word-level labels with given text and NER annotations.
Parameters
----------
ner_annotations
The NER annotations.
ner_text
The corresponding raw text.
entity_map
The entity map between token label to word label.
tokenizer
The tokenizer to be used.
is_eval
Whether it is for evaluation or not, default: False
Returns
-------
Token-level/word-level labels and text features.
"""
| train_data=train_df, | ||
| tuning_data=dev_df, | ||
| hyperparameters={ | ||
| "model.ner_text.checkpoint_name": "microsoft/mdeberta-v3-base", |
There was a problem hiding this comment.
Shall we use a smaller model?
|
Job PR-2676-9350aa1 is done. |
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.