Data format for fine-tuning

[e0397123]

Hi, may I know if it is possible to share data samples for finetuning contriever? In the code, there are fields like "positive_ctxs", "hard_negative_ctxs", "negative_ctxs", "question". Could you provide some examples?

[sld]

You should have train.jsonl and eval.jsonl files.
E.g. of usage in finetuning:

python finetuning.py --eval_data eval.jsonl --train_data train.jsonl  ....

Here is the example row in jsonl file:

{"question":"What is the most popular operating system?","positive_ctxs":[{"text": "Windows is the most popular operating system."}],"negative_ctxs":[{"text": "Windows is the most popular programming language."}],"hard_negative_ctxs":[{"text": "Windows is the most popular game console."}],"title":"Windows","text":"Windows is the most popular operating system."}

For fine-tuning I am using only 'question', 'positive_ctxs' and 'negative_ctxs' fields.