Evaluator cannot return the expected metrics.

[lcy-seso]

In V2 API, by default, the following codes print a certain metric calculated by an evaluator.

def event_handler(event):
        if isinstance(event, paddle.event.EndIteration):
            if event.batch_id % 100 == 0:
                print "\nPass %d, Batch %d, Cost %f, %s" % (
                    event.pass_id, event.batch_id, event.cost, event.metrics)

In the C++ implementation, an evaluator stores the evaluation results in Argument, and there are also some evaluators that store nothing but just print to stderr.

The above code can only print the evaluated stored in Argument.value. If any of the following two situations happens, the evaluator cannot return a right result.

1. An evaluator stores multiple evaluated metrics in a special format.
2. An evaluator does not store the evaluated metrics but just print, for example, the chuck evaluator.

For the fist case, there should be some documentations to explain how each evaluator stores its results.

For the second case, the original C++ codes need to be modified.

Does anyone test that in V2 API every evaluator returns the right results?

[reyoung]

Could we fix this bug by making Evaluator can return many types in Python?

Just print to stderr is really a bad design in Paddle. If an Evaluator could return any types of metrics, it seems would fix this bug. For example, if we use beam search evaluator, we could get the result sentences in metrics.

[reyoung]

I also think Evaluator in Paddle is an overused concept. Just tell me why we need print Evaluator when we can get any layer's output in Python?

[lcy-seso]

@reyoung I think the problem is we should disable some evaluators that only to print, and formalize evaluators that do not store their results in Arguments, so that they can work with v2 API better.

[lcy-seso]

@pkuyym will fix this bug.

[pkuyym]

@lcy-seso @reyoung
I find that the v2 api will call getValue in a loop to retrieve all metrics calculated by ChunkEvaluator. So I just implement corresponding virtual functions of ChunkEvaluator to return Precision、Recall and F1-Score. Functions I implemented are getType、getNames and getValue. Please feel free to point out what I've missed.