-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix 2078 #2165
Fix 2078 #2165
Changes from 3 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -347,39 +347,77 @@ def chunk_evaluator( | |
excluded_chunk_types=None, ): | ||
""" | ||
Chunk evaluator is used to evaluate segment labelling accuracy for a | ||
sequence. It calculates the chunk detection F1 score. | ||
sequence. It calculates precision, recall and F1 scores for the chunk detection. | ||
|
||
A chunk is correctly detected if its beginning, end and type are correct. | ||
Other chunk type is ignored. | ||
To use chunk evaluator, several concepts need to be clarified firstly. | ||
Chunk type is the type of the whole chunk and a chunk consists of one or several words. (For example in NER, ORG for organization name, PER for person name etc.) | ||
Tag indicates the position of a word in a chunk. (B for begin, I for inside, E for end, S for single) | ||
We can name a label by combining tag type and chunk type. (ie. B-ORG for begining of an organization name) | ||
|
||
For each label in the label sequence, we have: | ||
The construction of label dict should obey the following rules: | ||
(1) Use one of the listed labelling schemes. These schemes differ in ways indicating chunk boundry. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we should define "chunk type", "tag type" before the following table. And we'd better have a running example to show how to label the words using different schemes. In fact, the following table is the protocol for assigning tag types, not the definition of the schemes. Therefore, I think we also need another table for the definitions. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||
|
||
.. code-block:: python | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why code-block is python? It seems a plain text?
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See this documentation. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||
Scheme Description | ||
plain Use the same label for the whole chunk. | ||
IOB Two labels for chunk type X, B-X for chunk begining and I-X for chunk inside. | ||
IOE Two labels for chunk type X, E-X for chunk ending and I-X for chunk inside. | ||
IOBES Four labels for chunk type X, B-X for chunk begining, I-X for chunk inside, E-X for chunk end and S-X for single word chunk. | ||
.. code-block:: python | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 366行可以去掉,只需要一个 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. done |
||
|
||
To make it clear, let's illustrate by an NER example. | ||
Assuming that there are three named entity types including ORG, PER and LOC which are called 'chunk type' here, | ||
if 'IOB' scheme were used, the label set will be extended to a set including B-ORG, I-ORG, B-PER, I-PER, B-LOC, I-LOC and O, | ||
in which B-ORG for begining of ORG and I-ORG for inside of ORG. | ||
Prefixes which are called 'tag type' here are added to chunk types and there are two tag types including B and I. | ||
Of course, the training data should be labeled accordingly. | ||
|
||
tagType = label % numTagType | ||
chunkType = label / numTagType | ||
otherChunkType = numChunkTypes | ||
(2) Mapping is done correctly by the listed equations and assigning protocol. | ||
|
||
The total number of different labels is numTagType*numChunkTypes+1. | ||
We support 4 labelling scheme. | ||
The tag type for each of the scheme is shown as follows: | ||
The following table are equations to extract tag type and chunk type from a label. | ||
|
||
.. code-block:: python | ||
tagType = label % numTagType | ||
chunkType = label / numTagType | ||
otherChunkType = numChunkTypes | ||
.. code-block:: python | ||
|
||
The following table shows the mapping rule between tagType and tag type in each scheme. | ||
|
||
Scheme Begin Inside End Single | ||
plain 0 - - - | ||
IOB 0 1 - - | ||
IOE - 0 1 - | ||
IOBES 0 1 2 3 | ||
.. code-block:: python | ||
Scheme Begin Inside End Single | ||
plain 0 - - - | ||
IOB 0 1 - - | ||
IOE - 0 1 - | ||
IOBES 0 1 2 3 | ||
.. code-block:: python | ||
|
||
'plain' means the whole chunk must contain exactly the same chunk label. | ||
Continue the NER example, and the label dict should look like this to satify above equations: | ||
|
||
.. code-block:: python | ||
B-ORG 0 | ||
I-ORG 1 | ||
B-PER 2 | ||
I-PER 3 | ||
B-LOC 4 | ||
I-LOC 5 | ||
O 6 | ||
.. code-block:: python | ||
|
||
In this example, chunkType has three values: 0 for ORG, 1 for PER, 2 for LOC, because the scheme is | ||
"IOB" so tagType has two values: 0 for B and 1 for I. | ||
Here we will use I-LOC to explain the above mapping rules in detail. | ||
For I-LOC, the label id is 5, so we can get tagType=1 and ChunkType=2, which means I-LOC is a part of NER chunk LOC | ||
and the tag is I. | ||
|
||
The simple usage is: | ||
|
||
.. code-block:: python | ||
|
||
eval = chunk_evaluator(input, label, chunk_scheme, num_chunk_types) | ||
|
||
.. code-block:: python | ||
|
||
:param input: The input layers. | ||
:type input: LayerOutput | ||
:param label: An input layer containing the ground truth label. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shall we push
precision
,recall
,F1-score
directly into `names?It is simpler and more computationally efficient.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think holding precision, recall and F1-score into an unified map could make the code cleaner and easier to maintain and the extra computation cost is trivial.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@reyoung what do you think about this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is not a big deal I think. Making
recall
,F1-score
just a key for value is simple for implementation.But we could also,
But in view of the previous writing has been done, so be it.