Notus end2end example for preference and instruction generation

[ignacioct]

Closes #143

First draft of the tutorial. Few things to take in mind:

• Missing the conclusions, I'll get them ready when we have the workflow ready, in case something changes.
• Unsure of the fine-tuning process, as using the openai framework fine-tunes a gpt-3.5 model, and maybe we want to fine-tune the notus model. In that case, how should I approach it with the Argilla Trainer?

Looking forward to your review! First time I go through distilabel process on my own, so there's probably a lot of little things to tweak :)

[davidberenstein1957]

@ignacioct, It looks good already but I've got some comments about the introduction of different aspects and the structure.

"Setting up an inference endpoint with Notus" also includes another section about "Defining a custom generation task for a distilabel pipeline".
I think it is really useful to showcase the custom task but we might need some context about that it is possible and we just use it for testing the pipeline etc.

Similarly "Download the AI Act PDF document" also cover "Creating a RAG pipeline using Deepset"

Also, for the separate parts we might add some additional introduction about what is happening and why, like the deepest index for example.

I would maybe make the introduction and overview a bit more catchy to focus on the AI act etc. Something like
"Use notus on inference endpoints to create a legal preference dataset based on RAG instructions from the EU AI Act"

You can also emphasize a bit why this is important to do and what the end user might gain from this approach over other ones.

Maybe some of the printed output is a bit long (about the batches etc.)

Also, in the end, I feel that I'm missing some wrap-up about the OpenAI-finetune and why we would need that/how we can use it. Maybe fine-tuning can be skipped and we can redirect people to some of our other tutorials https://docs.argilla.io/en/latest/tutorials_and_integrations/tutorials/tutorials.html

[ignacioct]

@davidberenstein1957 I've implemented your suggestion; just a few doubts

• If the outputs are too long, or do not add value, should I just erase them? Or how shall I deal with 'em.
• When you suggest to add additional context for the deepset index, what part exactly do you mean?
• I'm still unsure if I shall delete the fine-tuning part or not. I'm right now tempted to delete it, point to another tutorial, and focus on creating a conclusion where when emphasize why this was important and has value.

[davidberenstein1957]

• I think erasing them or printing a part of them is fine. e.g. For a list -> show the index. for a text -> show the first x characters.
• Why are we using deepest, what are the steps we are going through? I believe we just use it to parse and process the text, but it is good to communicate that clearly :)
• We can leave it for now and wait for @dvsrepo. I vote for deleting it and redirecting it to other content that focuses on fine-tuning through DPO and preferably using an open-source model.

[davidberenstein1957]

@ignacioct , looking much better already :)
Here the title is nice and problem oriented for example: https://docs.giskard.ai/en/latest/tutorials/llm_tutorials/index.html
some follow-ups,

• I would also rename the title to the one I proposed earlier.
• I would mention the RAG part is not active-based on queries/semantic search but rather a more brute-force approach.
• maybe a step by step 1.2.3. at the introduction would be nice

[ignacioct]

@ignacioct , looking much better already :) Here the title is nice and problem oriented for example: https://docs.giskard.ai/en/latest/tutorials/llm_tutorials/index.html some follow-ups,

• I would also rename the title to the one I proposed earlier.
• I would mention the RAG part is not active-based on queries/semantic search but rather a more brute-force approach.
• maybe a step by step 1.2.3. at the introduction would be nice

Maybe something a little bit shorter like: Use Notus on inference endpoints to create a legal preference dataset ?

[davidberenstein1957]

@ignacioct, great work. It looks very complete.

3 minor remarks:

• Llama2QuestionAnsweringTask, maybe renaming it to QuestionAnsweringTask, might be more generic
• can you rename the filename, to something more explicit aligned with the examples?
• some of the output in the cells might be cropped a bit? I think just showing the keys or text[:x] would suffice in most cases,