Question: Can I see any evaluation metric?

[RobKnop]

I'm doing this:

# pyhton:
reader.train(data_dir=train_data, train_filename="2020-02-23_answers.json", test_file_name='TEST_answers.json', use_gpu=False, n_epochs=1, dev_split=0.1)
# result:
Preprocessing Dataset: 12 Dicts [00:08,  1.46 Dicts/s]
Preprocessing Dataset: 4 Dicts [00:06,  1.56s/ Dicts]
Train epoch 1/1 (Cur. train loss: 0.0712): 100%|██████████| 87/87 [09:25<00:00,  6.50s/it]
Evaluating: 100%|██████████| 189/189 [06:49<00:00,  2.17s/it]
02/23/2020 16:40:48 - INFO - haystack.reader.farm -   Saving reader model to ../../saved_models/distilbert-base-uncased-distilled-squad

I want to see if my model improves. How I can do?
Or should I switch to FARM or use FARM directly now?

[tholor]

Hey @RobKnop,

Good point! We suppress most of the logging from FARM / Transformers. For the evaluation part this is clearly unwanted. Making it more verbose in #28. If you find other parts, where you miss some logs, please let us know.

Regarding training in FARM vs haystack: The idea is that FARM allows fine-grained control over training (incl. access to all building blocks like LR schedule, optimizers ...). In contrast, haystack has a light-weight & simple training pipeline that covers ~ 80% of most common training jobs, but let's the user do all QA dev end-to-end without switching frameworks.

[RobKnop]

Hey @tholor

Thanks for the fast answer. I already pulled the current master and restarted the training to check if it's helping :)

Also thanks for your overview about FARM vs haystack.
My approach is to push my project as far as I can with haystack and then dig deeper and deeper. So FARM is next and huggingface's Transformers probably after.

[RobKnop]

@tholor Looks good. I do not have enough labelled at the moment, I know. But I still have some questions:

Preprocessing Dataset: 12 Dicts [00:09, 1.33 Dicts/s] --> Training set? I also have just 10 document/paragraphes, not 12
Preprocessing Dataset: 4 Dicts [00:06, 1.60s/ Dicts] --> Test set? I also have 1 document with 12 qna-pairs
Train epoch 1/1 (Cur. train loss: 0.0712): 100%|██████████| 87/87 [08:50<00:00, 6.10s/it]
Evaluating: 100%|██████████| 189/189 [06:22<00:00, 2.02s/it] --> what are these?
02/24/2020 21:33:35 - INFO - farm.eval -

\|// \|// \|// \|// \|//
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

---

***** EVALUATION | TEST SET | AFTER 86 BATCHES ***** . What are batches in this context?

---

\|// \|// \|// \|// \|//
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

02/24/2020 21:33:35 - INFO - farm.eval -
_________ question_answering _________
02/24/2020 21:33:35 - INFO - farm.eval - loss: 0.10436401187962281
02/24/2020 21:33:35 - INFO - farm.eval - task_name: question_answering
02/24/2020 21:33:35 - INFO - farm.eval - EM: 0.0
02/24/2020 21:33:35 - INFO - farm.eval - f1: 0.0
02/24/2020 21:33:35 - INFO - farm.eval - report:
not Implemented

[tholor]

Hi @RobKnop ,

Great, that you are now diving deeper into FARM / the preprocessing steps involved!
One general advise here: It might help you to make the logging more verbose for training. I think this will help you to understand the preprocessing a bit better. You can enable verbose logging via:

import logging
logging.getLogger('farm').setLevel(logging.INFO)
# ... rest of your code

Preprocessing Dataset: 12 Dicts [00:09, 1.33 Dicts/s]

Yes, these dicts are the number of documents in your squad input file (there it's called "context")
From there, we will split them into smaller passages that fit our model's max_seq_len and pair each passage with a question (= "sample"). If you want to understand more of this internals, this blog article might be helpful.

Training set? I also have just 10 document/paragraphes, not 12

Totally right. The tqdm progressbar was showing an incorrect number here. Fixed it in deepset-ai/FARM#267 . With that fix you should see 10 now.

--> Test set? I also have 1 document with 12 qna-pairs

Depends on your input. Could be dev or test dataset. With more verbose logging it should become clearer.

Evaluating: 100%|██████████| 189/189 [06:22<00:00, 2.02s/it] --> what are these?

You are evaluating on your test dataset which got split into 189 batches (each of batch_size x).
It seems that your test dataset contains more samples than the training dataset. This might be just because the document is longer and therefore gets split into more passages.

***** EVALUATION | TEST SET | AFTER 86 BATCHES ***** . What are batches in this context?

You did 86 steps of training and then started your evaluation. If you have a dev set, you can do additional evaluation runs every x steps/batches (configure this via reader.train(...evaluate_every=100)

Hope this helps. Happy to explain in more detail, if needed :)

[RobKnop]

@tholor Thank you so much. I went though all of your hints and advices. deepset-ai/FARM#267 looks good on my side.
I go ahead and close this issue.