[WIP] CoNLL-2003 dataset -Ner, Chunk and POS tagger

[sbmaruf]

As the dataset is smaller, I only add two prompts for each of the tasks.

Two major issue to follow,

[ISSUE 1]:
After writing the template, on the right side, there is a sample shown in the app.
I see <p> tags inside the samples (from the inspect of the browser). I am not sure if this is because of HTML or not.

[ISSUE 2]:
After writing the template I don't see any error generated. But when I go the Prompted dataset viewer I get the following error,

TemplateSyntaxError: unexpected '|'
Traceback:
File "/usr/local/lib/python3.7/site-packages/streamlit/script_runner.py", line 338, in _run_script
    exec(code, module.__dict__)
File "/Users/sbmaruf/Documents/big-science/promptsource/promptsource/promptsource.py", line 294, in <module>
    prompt = template.apply(example, highlight_variables=True)
File "promptsource/templates.py", line 302, in apply
    rtemplate = env.from_string(jinja)
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 941, in from_string
    return cls.from_code(self, self.compile(source), globals, None)
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 638, in compile
    self.handle_exception(source=source_hint)
File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
File "/usr/local/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
File "<unknown>", line 15, in template

[srush]

Oh wow, you have the hardest one. We'll try to help out.

[srush]

This looks very complex. If there a reason you can't use loop index?

https://stackoverflow.com/questions/1567291/get-loop-index-of-outer-loop

[sbmaruf]

Done.

[srush]

lable - > label

[sbmaruf]

Done

[sbmaruf]

@srush

As suggested, changes are in this commit, 6539b83

New Prompt: 3 additional prompt added for ner, pos and chunks by this commit dc8d60e . The prompts intends to solve the following issue,

• It is not expected that user will always provide labels strictly from the dataset. They may provide a subset of labels.
• When user provided a subsample of labels of ner, pos or chunk tags, the model should understand that ONLY the given lables should be used for classification instead of original labels in the dataset. In this way, inference will be more natural.

So I randomly select some labels with the template and use them to label the sample. If the label is not in the selected labels I put O tag for that token.

TODO:

• Randomly shuffle the tags. Now the labels are always in the same order. We should shuffle the lables. But I cannot find shuffle filter in the app.

Issue:

• Test failing because of unused variable. I cannot update a dictionary in jinja without set method. There is a workaround using do extension. But I cannot find do extension in the app.

For convenience, A template that selects label randomly looks like below. I think the test is not passing because of _dummy variable.

{% set _ner_label_dict = ({
0:"O",
1:"B-PER",
2:"I-PER",
3:"B-ORG",
4:"I-ORG",
5:"B-LOC",
6:"I-LOC",
7:"B-MISC",
8:"I-MISC"
}) %}
{% set _random_ner_label_dict = ({}) %}
{% for k,v in _ner_label_dict.items() -%}
    {% set _rand_num=range(0, 100) | random %}
    {% if _rand_num > 50 %}
        {% set _dummy = _random_ner_label_dict.update({k:v}) %}
    {% endif %}
{% endfor %}
Generate named entities from the following sentence. The named entities are
{% for k,v in _random_ner_label_dict.items() -%}
    {{ v }}
    {{- ", " if not loop.last else "" -}}
{% endfor %}
{{""}}
{% for i in tokens -%}
    {{- " " if not loop.last else "" -}}
    {{ i }}
{% endfor %}
|||
{%- for tok in tokens -%}
    {%- set outer_loop = loop -%}
    {%- for label in ner_tags -%}
         {%- if outer_loop.index == loop.index -%}
            {{- "" if outer_loop.first else " " -}}
            {%- if label in _random_ner_label_dict -%}
                {{tok}}:{{ _random_ner_label_dict[label] }}
            {%- else -%}
                {{tok}}:O
            {% endif %}
        {%- endif -%}
    {%- endfor -%}
{%- endfor -%}

Please also look into this issue before merge, #174

[sbmaruf]

More details on random label selection by prompt

We always include "O" tag for ner, pos and chunk labels. "O" lable indicated it is "other" (uncategorized) label.

In case of parts-of-speech tags, there are few labels that are weird. For example see a prompt with all pos labels,

Generate parts of speech from the following sentence. The parts of speech tags are ", '', #, $, (, ), ,, ., :, ``, CC, CD, DT, EX, FW, IN, JJ, JJR, JJS, LS, MD, NN, NNP, NNPS, NNS, NN|SYM, PDT, POS, PRP, PRP$, RB, RBR, RBS, RP, SYM, TO, UH, VB, VBD, VBG, VBN, VBP, VBZ, WDT, WP, WP$, WRB

In my opinion, here first 9 labels should be normalized. However in the regular prompt I didn't change them. But in the random label selection prompt named as pos_flat_question_with_label I did the following normalization,

{% if k>=0 and k<= 9 %}
    {% set _dummy = _random_pos_label_dict.update({0:"O"}) %}
{% else %}
    {% set _dummy = _random_pos_label_dict.update({k:v}) %}
{% endif %}

This will convert the first 9 weird label as "O" tag. Note that, there is not "O" in dataset for pos labels.

[sbmaruf]

Deleting pull request. Creating a new one with updated zip function.