CLI and getting spans directly from a prodigy or a spaCy model

[ivyleavedtoadflax]

This is following on from our discussion in #48.

First off, I think the default CLI should be a very simple wrapper for the Evaluate class, so it doesn't need to be much more complicated than (note I didn't test any of this):

./nervaluate/cli.py:

import typer

from nervaluate import Evaluator

app = typer.Typer()

app.command()
def evaluate(
    true_path: str = typer.Argument(help="Path to true entity labels"),
    pred_path: str = typer.Argument(help="Path to predicted entity labels"),
    tags: str = typer.Argument(
        None, help="Comma separated list of tags to include in the evaluation"
    ),
    loader: str = typer.Option(
        None,
        help="Optional loader when not using prodigy style spans. One of [list, conll]",
    ),
    by_tag: bool = typer.Option(
        None,
        help="If set, will return tag level results instead of aggregated results.",
    ),
    pretty: bool = typer.Option(
        None,
        help="If set, will print the results in a pretty format instead of returning the raw json",
    ),
):

    tags_list = tags.split(",")
    evaluator = Evaluator(true_path, pred_path, tags=tags_list, loader=loader)

    results, results_by_tag = evaluator.evaluate()

    if by_tag:
        output = results_by_tag
    else:
        output = results

    if pretty:
        pass
        # Some code from wasabi to print a pretty table https://pypi.org/project/wasabi/
    else:
        return output


if __name__ == "__main__":
    app()

For handling predictions directly from a spacy/prodigy model, I think we should implement a typer command that does what @Eleni170 implemented in f2841e2. So it would be something like:

@app.command()
def predict(
    model_path: str=typer.Argument(help="Path to spaCy model"),
    data_path: str=typer.Argument(
        help="Path to data in prodigy format (including the raw text)"
    ), 
    by_tag: bool = typer.Option(
        None,
        help="If set, will return tag level results instead of aggregated results.",
    ),
    pretty: bool = typer.Option(
        None,
        help="If set, will print the results in a pretty format instead of returning the raw json",
    ),
):
    spacy_model = spacy.load(model_path)

    true = []
    pred = []
    tags = {}

    with open(data_path) as f:
        for line in f:
            pattern = json.loads(line)
            text = pattern["text"]
            meta = pattern["meta"]
            labels = check_labels(meta)
            for label in labels:
                tags[label] = ''
            true.append(meta)
            doc = spacy_model(text)
            pred.append(create_prodigy_spans(doc))

    # Maybe we want also to pass tags to the CLI as above but default to all tags as below if nothing is passed.

    evaluator = Evaluator(true, pred, tags=list(tags.keys()))
    global_results, aggregation_results = evaluator.evaluate()

# Similar logic as above to print results to console either as raw json or pretty printed.

Let me know what you think! @Eleni170 @nsorros

[ivyleavedtoadflax]

Btw this is worth a look since we will be using typer for the CLI https://typer.tiangolo.com/tutorial/package/ we get free cli completion as a bonus

[ivyleavedtoadflax]

@davidsbatista FYI we are picking this up again. Would be glad to hear your thoughts on our intended direction (here and elsewhere)

[nsorros]

Sounds good, let's do it

[nsorros]

Maybe in the future we want to move the predict function to some utils cli for spacy / prodigy. Additionally there is a minor problem if we want predict to be able to work with spacy 2 and spacy 3 😬

[nsorros]

Let's implement what @ivyleavedtoadflax suggest. I suggest we make the predict code agnostic to spacy 2/3 and resolve this with two extras one for each version so that the user decides based on what model they want to use the prediction function on.

Also @Eleni170 note that the predict command should run as nervaluate predict whereas the evaluate should be the default so just nervaluate

[davidsbatista]

hey everyone, sorry for being a bit out but there's always other things coming in, and since I stopped using nerevalute I also slowly forgot about it

But anyway, I'm supportive of a command line client which should be simple and easy too use. I'm a bit confused with prodigy/spacy format, shouldn't we just provide a public function, something like:

evaluate(pred_labels: List[str], true_labels: List[str], other_options)

which then outputs an evaluation report, and is the concern of the caller to adjust and pass the labels, from spaCy one can then simply import this function, or am I missing something?

[nsorros]

hey everyone, sorry for being a bit out but there's always other things coming in, and since I stopped using nerevalute I also slowly forgot about it

But anyway, I'm supportive of a command line client which should be simple and easy too use. I'm a bit confused with prodigy/spacy format, shouldn't we just provide a public function, something like:

evaluate(pred_labels: List[str], true_labels: List[str], other_options)

which then outputs an evaluation report, and is the concern of the caller to adjust and pass the labels, from spaCy one can then simply import this function, or am I missing something?

hi @davidsbatista, where we are coming from is we want to use nervaluate with a model trained either with spacy or prodigy. As it turns spacy has a command spacy evaluate model_path data_path for which I was inspired to have a similar cli for nervaluate model_path data_path so that we can calculate additional metrics with a similar API.

then as @ivyleavedtoadflax and you rightly pointed out it is probably not the responsibility of nervaluate and since the tool works on a specified format the basic CLI should provide that. this still leaves us with the problem that there is not convenient way to create the format that nervaluate accepts. if only there was a predict function that could create that format give a model (starting with spacy, prodigy models) that would be handy cause in two commands we could get those metrics. this is how the idea for nervaluate predict was born.

Now I recognise that maybe this should leave on a separate tool all together so maybe in the future it will but if we are all happy with having that predict with extras for spacy so someone that wants to use will install pip install nervaluate[spacy] or pip install nervaluate[spacy3] this could be a first step for now. What do you think?

[davidsbatista]

I personally would not care much about that, main responsability of nerevaluate is to compute the differente metrics for a more detailed NER evaluation report. Then probably one can also give the option to the user on how to decide on ambiguous cases - there were a few issues and PR related to these, as far as I remember, can't point out any specific examples right now.

Then, one can also support some simple set of formats - based on string transformations, i.e., transforming them into the internal representation that nerevaluate uses, like detailed here: https://github.com/MantisAI/nervaluate/blob/master/tests/test_loaders.py

Then one can either import this functionalities into code or call it from the command line. I would settle this as the whole scope of this project.

To go even more ahead and create a specific install to be used only by spaCy, I personally think we are starting to overenginer/complicating things.