Allow for model evaluation directly from cdqa

[fmikaelian]

The idea is to implement the evaluate.py script inside the package under /utils

[fmikaelian]

You should now be able to evaluate with:

from cdqa.utils.metrics import evaluate, evaluate_from_files

evaluate(dataset, predictions) # as json objects
evaluate_from_files(dataset_file='data/dev-v1.1.json', prediction_file='logs/bert_qa_squad_v1.1_sklearn/predictions.json') # as json files

[fmikaelian]

@andrelmfarias Are we evaluating the reader only or the full QAPipeline?

[fmikaelian]

The evaluation of the reader can be done with the all_predictions object:

https://github.com/fmikaelian/cdQA/blob/3583256cbf73e8f3674f57182e824a5ca7c4f7be/cdqa/reader/bertqa_sklearn.py#L623-L634

https://github.com/fmikaelian/cdQA/blob/3583256cbf73e8f3674f57182e824a5ca7c4f7be/cdqa/reader/bertqa_sklearn.py#L649-L650

Here is a reproducible example on dev SQuAD:

import os
from ast import literal_eval
import pandas as pd
import joblib
import json

from cdqa.utils.filters import filter_paragraphs
from cdqa.reader.bertqa_sklearn import BertProcessor, BertQA
from cdqa.utils.metrics import evaluate

df = pd.read_csv('../data/bnpp_newsroom_v1.1/bnpp_newsroom-v1.1.csv', converters={'paragraphs': literal_eval})

df['paragraphs'] = df['paragraphs'].apply(filter_paragraphs)
df['content'] = df['paragraphs'].apply(lambda x: ' '.join(x))

reader = joblib.load('../models/bert_qa_squad_v1.1_sklearn/bert_qa_squad_v1.1_sklearn.joblib')
reader.output_dir = '../logs/'

processor = BertProcessor(do_lower_case=True, is_training=False)
examples, features = processor.fit_transform(X='../data/dev-v1.1.json') # replace by custom dataset
preds = reader.predict((examples, features))

with open('../data/dev-v1.1.json') as dataset_file:
    dataset_json = json.load(dataset_file)
    dataset = dataset_json['data']

with open('../logs/predictions.json') as all_predictions:
    all_predictions = json.load(all_predictions)

evaluate(dataset, all_predictions) # as json objects

{'exact_match': 81.2488174077578, 'f1': 88.43242225358777}

To evaluate on a custom dataset we just need to replace the input dataset by dev-v1.1.json.

Evaluating the whole pipeline seems to be a different story though... We will need to use our brains for a bit!

[fmikaelian]

I think if QAPipeline's predict() returns something in the same format of all_predictions that correponds to the custom input dataset labelled in json, we're good.

[andrelmfarias]

Didn't we agree on #135 to create a method prepare_evaluation() in https://github.com/fmikaelian/cdQA/blob/develop/cdqa/utils/metrics.py to handle this instead of doing it directly in QAPipeline.predict()?

[fmikaelian]

Yes, true. It should use qapipelin and returns an object similar to all_predictions, linked to the custom json dataset.

[andrelmfarias]

I think it just should take as input the output of QAPipeline.predict() and we will need to run the code:

predictions = QAPipeline.predict()
pred_for_eval = prepare_evaluation(predictions)

[fmikaelian]

@andrelmfarias

I think in that case pred_for_eval() must convert predictions to the all_predictions format?

That way we could do:

evaluate(dataset, pred_for_eval)

instead of:

evaluate(dataset, all_predictions)

[andrelmfarias]

the predictions output from cdqa does not have the question_id of the question for evaluation. This is needed to convert predictions to all_predictions format in order to evaluate.

We need that BertQA sends the question_id on its prediction.

I will do a pull request with that feature.

[andrelmfarias]

Just realized that we have a problem...

Our annotated dataset (from annotator) does not have a question_id. The evaluate method needs it in order to link the questions and answer.

However, the BertQA class does send a question_id as output, but I don't understand which idi it is... Do you know where does it come from @fmikaelian ?

What I understand is that it is not the question_id we need...

[andrelmfarias]

Just found where does it come from... generate_squad_examples . It is a randomly generated id:

https://github.com/fmikaelian/cdQA/blob/f824db1ad5dab2c2b2527faaa4a6d9f97a308e2a/cdqa/utils/converters.py#L97

We have to discuss it tomorrow, I don't really have an idea on how we can handle it.

[fmikaelian]

You're right. It is something I didn't think of when building the annotator. The question_id is not appended. See https://github.com/fmikaelian/cdQA-annotator/blob/271e005d8ba0bd21277ebc752fa10b1ad8839d37/src/components/AnnotationsPage.vue#L114

What we can do is the following:

1. Manually add a question_id to the current annotated json
2. Provide generate_squad_examples tuples of questions + their question_id
3. In generate_squad_examples set the question_id to the one provided if any
4. Let predict(), probably write_predictions() also output the question_id

We can rediscuss this afternoon

[andrelmfarias]

You're right. It is something I didn't think of when building the annotator. The question_id is not appended. See https://github.com/fmikaelian/cdQA-annotator/blob/271e005d8ba0bd21277ebc752fa10b1ad8839d37/src/components/AnnotationsPage.vue#L114

What we can do is the following:

1. Manually add a question_id to the current annotated json
2. Provide generate_squad_examples tuples of questions + their question_id
3. In generate_squad_examples set the question_id to the one provided if any
4. Let predict(), probably write_predictions() also output the question_id

We can rediscuss this afternoon

1. I agree
2. Yes, it will be a major change in the code. We have to be attentive when doing it
3. Actually will need both ids... because we use the current randomly generated id to sort the paragraphs chosen by the retriever and send it to the reader
4. We could do it as an option on predict send_id = True where its default value is False as we will only need it for evaluation.

[andrelmfarias]

@mamrou will work on 1.

[andrelmfarias]

@fmikaelian

I will work on 2. 3. 4.

[fmikaelian]

Reporting @mamrou's solution for 1. (Manually add a question_id to the current annotated json)

import json
import uuid

with open("/path_to_json") as json_file:  
    data = json.load(json_file) 
	
articles = data["data"]
for article in range(0, len(articles)):
    paragraphs = data["data"][article]['paragraphs']
    
    for paragraph in range(0, len(paragraphs)):
        questions = data["data"][article]['paragraphs'][paragraph]['qas']
        
        for question in range(0, len(questions)):
            unique_id = uuid.uuid4()
            data["data"][article]['paragraphs'][paragraph]['qas'][question]['id'] = str(unique_id)

with open('path_to_output_json', 'w') as outfile:  
    json.dump(data, outfile)

[fmikaelian]

Thanks for your help!