🎄 Advent of Haystack - 2024 🎄

[bilgeyucel]

You loved it, we brought it back - December is about to get merrier with the 2nd Edition of the Advent of Haystack! ☃️

Starting December 2, we’re releasing a series of challenges that will take you on an exciting journey: from mastering the basics of Haystack to building cutting-edge pipelines and intelligent AI agents. This year's edition is powered by collaborations with Weaviate, AssemblyAI, NVIDIA, Arize, and MongoDB!

Here’s what you can expect:
✨ Fun coding challenges
📖 Festive stories
🏆 Surprise prizes
💡 Hands-on learning with the best tools in AI app development

Challenges will go live on the Advent of Haystack page. Later, we will use this Discussion page to share the solutions to each challenge. You can also use this page to ask questions or discuss any specific challenge with the rest of the community 🫶

This is your chance to level up your Haystack expertise, connect with a vibrant developer community, and have some holiday fun! Let’s code, create, and celebrate this holiday season! 🎄

[bilgeyucel]

🎉 The wait is over! First Challenge is live: Fetching Holiday Wisdom for Santa 📖

Let's use this thread for questions 👇

[Ashish-Soni08]

Hey Team

I am getting the following error after running the code cell below:
(It was working a day ago, I have not changed anything)

from haystack.components.fetchers import LinkContentFetcher
from haystack.components.converters import HTMLToDocument
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OpenAIGenerator

fetcher = LinkContentFetcher()
converter = HTMLToDocument()
#### INITIALIZE THE MISSING COMPONENTS HERE ####
# comp_one = ...
# comp_two = ...
################################################
template = """Given the information below, answer the query. Only use the provided context to generate the answer and output the used document links
            Context:
            {% for document in documents %}
                {{ document.content }}
                URL: {{ document.meta.url }}
            {% endfor %}

            Question: {{ query }}
            Answer:"""

prompt_builder = PromptBuilder(template = template)
generator = OpenAIGenerator(model="gpt-4o-mini")

[Ashish-Soni08]

!pip install -q --upgrade openai

Fixes it!

[julian-risch]

🎉 And here we go! Second Challenge is live: Help the Elves Find What They've Lost 🧝
If there are any questions, please use this thread 👇

[aazam-gh]

@julian-risch is it possible to use Cohere instead of OpenAI here?

[bilgeyucel]

Hi @aazam-gh, yes. Feel free to change the language models :)

[aazam-gh]

@bilgeyucel I am using the cohere embed model english v2.0 which should produce vectors with the same dimension (4096) as the openai text ada model

from haystack_integrations.components.embedders.cohere.text_embedder import CohereTextEmbedder
from haystack_integrations.components.generators.cohere import CohereGenerator
from haystack.components.builders import PromptBuilder
from haystack_integrations.components.retrievers.weaviate import WeaviateEmbeddingRetriever


template = """
Given the following information, answer the question.

Context:
{% for document in documents %}
    {{ document.content }}
{% endfor %}

Question: {{question}}
Answer:
"""


rag = Pipeline()

rag.add_component("text_embedder", CohereTextEmbedder(model="embed-english-v2.0"))

rag.add_component("retriever", WeaviateEmbeddingRetriever(document_store=document_store))

rag.add_component("prompt_builder", PromptBuilder(template=template))

rag.add_component("llm", CohereGenerator())

rag.connect("text_embedder.embedding", "retriever.query_embedding")
rag.connect("retriever", "prompt_builder.documents")
rag.connect("prompt_builder", "llm")

Yet I am still getting this error.

    status = StatusCode.UNKNOWN
    details = "explorer: get class: concurrentTargetVectorSearch): explorer: get class: vector search: object vector search at index default: shard default_xmqBdn2haNde: vector search: knn search: distance between entrypoint and query node: 1536 vs 4096: vector lengths don't match"
    debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"explorer: get class: concurrentTargetVectorSearch): explorer: get class: vector search: object vector search at index default: shard default_xmqBdn2haNde: vector search: knn search: distance between entrypoint and query node: 1536 vs 4096: vector lengths don\'t match", grpc_status:2, created_time:"2024-12-09T12:56:13.060365121+00:00"}"

[Sprizgola]

Hi @aazam-gh, the model text-embedding-3-small has a embedding dimension of 1536 as state here so I'm pretty sure you should use, at least for the embedding, the same model used to embed the documents, but I'll let @bilgeyucel to have the last word!

[bilgeyucel]

@aazam-gh Sorry for the misinformation here. For most challenges, you can change the models. However, in this challenge, the documents are already indexed using text-embedding-3-small so you need to use the same model for your queries to find the most relevant documents

[julian-risch]

🎄 Today, in the third challenge, we're Making Answers Bright with Multi-Query RAG Magic! 🎄
Let us know if there are any questions in this thread please 👇

[Sprizgola]

Just a quick feedback: it could be useful to share the challenge also on Discord!

[d-kleine]

@bilgeyucel I have already reached out on LinkedIn: It would be great if the HayStack logo could be added to SimpleIcons to highlight skills with HayStack in a developer portfolio. I have noted that the icon is currently missing, but its implementation requires your permission: simple-icons/simple-icons#12288 (comment)

Would that be possible?

[bilgeyucel]

✨ New week, new challenge: Ready to Share the Christmas Magic with Gen Z 🎶 using AssemblyAI?

Let us know if you have any questions in this thread 👇

[d-kleine]

@bilgeyucel I think there is a small issue due to an inconsistency in the provided code:
The OpenAI API key will be stored in a variable:

OPENAI_API_KEY = getpass("Enter OpenAI Api key: ")

but later, it will be read from the environment:

llm = OpenAIGenerator(model="gpt-4o-mini",
                      api_key=Secret.from_token(OPENAI_API_KEY))

This did not work for me as the key has not been saved in the environment before, but I resolved it by saving the key with os.environ["OPENAI_API_KEY"].

[bilgeyucel]

You're right, thank you for letting us know! @d-kleine

[d-kleine]

Alright, thanks!

[bilgeyucel]

Halfway through, and this one’s extra special! 🎉 Discover Elves’ Secret for Faster Development 💨, some of you might already know it! 🤫

Got questions? Drop them in this thread! 👇

[d-kleine]

@bilgeyucel Asking out of curiosity, what is the difference in the Deepset cloud saving the secret key for the API under "Secrets" versus establishing a connection to a service via a PAT under "Connections" in your profile?

[elizaup]

Hey @d-kleine, Connections are best suited for connecting to 3rd party integrations that deepset manages, as we validate the connection. We do not do this for Secrets. Hope this helps!

[d-kleine]

Alright, thanks again!

[TheMimikyu]

@bilgeyucel I signed up for Deepset Studio on December 22nd but still haven't received access. Unfortunately, I'm unable to start the challenge without it. Could you please assist me with this issue?

[bilgeyucel]

@TheMimikyu, you should have the invitation now! Let me know if you need more assistance :)

[bilgeyucel]

🎉 Day 6 is out! 🎉

Will you be able to complete two tasks with NVIDIA NIMs for this challenge and Keep the Christmas Spirit Alive 🛷?

Let's use this thread for any questions 👇

[d-kleine]

I just tried it again, and now it worked! When I was completing the project, also the NIM website response time was really slow, maybe there was a deployment? Anyways, issue resolved ✅

[d-kleine]

@bilgeyucel I would have a question to the day 7 challenge, where can I post it?

[bilgeyucel]

I opened a new thread below! 🙏

[TheMimikyu]

When attempting to access the NvidiaRanker() model with model="nvidia/llama-3.2-nv-rerankqa-1b-v2", it returns ValueError: Model 'nvidia/llama-3.2-nv-rerankqa-1b-v2' is unknown. Please provide an api_url to access it. despite it existing in the Nvidia catalog.
Code:

alternative_ranker = NvidiaRanker(
    model="nvidia/llama-3.2-nv-rerankqa-1b-v2",
    api_key=Secret.from_env_var("NVIDIA_API_KEY")
)

Models nv-rerankqa-mistral-4b-v3 and llama-3.2-nv-rerankqa-1b-v1 are working as expected. But I fail to understand why it won't work for model llama-3.2-nv-rerankqa-1b-v2.

I don't know what the correct api_url should be, but trying api_url = "https://ai.api.nvidia.com/v1/retrieval" gives ValueError: Base URL path is not recognized. Expected format is 'http://host:port/'.

[d-kleine]

I was having this issue too and therefore chose a different reranker model. I just took a closer peak into the source code and found out that the model is missing in this list:
https://github.com/deepset-ai/haystack-core-integrations/blob/16bc80f0d49e3c50b90c17e55b7fb7118a4b8485/integrations/nvidia/src/haystack_integrations/components/rankers/nvidia/ranker.py#L19-L22

So, this must be added:

"nvidia/llama-3_2-nv-rerankqa-1b-v2": "https://ai.api.nvidia.com/v1/retrieval/nvidia/llama-3_2-nv-rerankqa-1b-v2/reranking"

Edit: The issue is that Nvidia seems to host some models at v1/ranking url path (which haystack checks for), but some other models are located at a different url path v1/retrieval with a /reranking suffix:
https://github.com/deepset-ai/haystack-core-integrations/blob/6b8d02d4dd5de710f4e20981b2da525194a5d462/integrations/nvidia/src/haystack_integrations/components/rankers/nvidia/ranker.py#L115

[bilgeyucel]

New challenge is here! For this challenge, we're Judging Toys, Tracing Joy 🧑‍⚖️ with Arize AI

Use the thread for questions 👇

[d-kleine]

@bilgeyucel I have a question to the third task, evaluating the toy correctness, using an LLM as a judge: Our task is submit the number of incorrect toy assignments. Should we here include the information on which list (naughty, nice) the child is on - or not including this information? My question here is whether we should assess whether the toy assignment with respect to list is correct or we should evaluate if a child gets a toy that matches with his/her preferences (without taking the naughty/nice information into account)?

[bilgeyucel]

@d-kleine, LLMs handle assigning toys and evaluating these assignments. So, all you need to do is report the number of "incorrect" labeled matches after LLM evaluation (See Arize Phoenix UI for details). No manual check is needed there and if you think the LLM made a mistake, simply ignore it :)
As a direct answer to your question: if the child is on the naughty list, their preference should be ignored, and they should get "Rabbit R1". The toy match should be evaluated by the LLM accordingly

[d-kleine]

Thanks, but you have not answered my question. I was asking what is meant by being "correct" and "incorrect". I don't understand the task with respect to the evaluation prompt. So, what should we do here?

1. Evaluating whether a child gets the correct toy with respect to his/her list information (nice = toy matching with child's preference; naughty = rabbit toy)
   -> If I do 1., then all assignments are "correct".
2. Evaluating whether the naughty children correctly get the rabbit toy?
   -> If I do 2., then all naughty children will be labeled as "correct" (as they get the rabbit), all others (=nice) "incorrect".

Technically, it also doesn't really make sense to me evaluating a generated output with the same LLM, but that is another story.

[bilgeyucel]

Sorry for the confusion. I wanted to mean the first option. So, naughty ones should get Rabbit R1, and nice ones should get something that suits their preferences. I'm hosting an office hours on discord in 2 mins, feel free to drop by if you'd like to discuss more https://discord.gg/V7vcASxK?event=1316445772970987614

re evaluating with the same model, I agree with you. That's why we wanted to give the option to use other modes for matching and evaluation

[Sprizgola]

Hi all! I was struggling during this challenge because of this bug that was raising this error ClassificationTemplate' object has no attribute 'explanation_template while running the llm_classify function.

I've solved this by instantiating the template in this way:

from phoenix.evals.templates import ClassificationTemplate

rails = ["correct", "incorrect"]
prompt_template = "prompt template ..... "
template = ClassificationTemplate(
    template= prompt_template, rails=rails, 
    explanation_template="Explanation: "
    )

and using it as input for the llm_classify function.

[bilgeyucel]

On Day 8 of Advent of Haystack, join @anakin87 to build the Ultimate Inventory Agent to Control Holiday Chaos 🛠️ 🎄

For any questions, use this thread 👇

[d-kleine]

With the default "api" backend in the DDG Websearch component, I ran quickly in a rate limit, disabling me from retrieving search results from DDG. Switching to "html" (or "lite") worked better for me.

[bilgeyucel]

⏰ The countdown has begun, only two challenges left!

On Day 9, you will create Santa’s Self-Reflecting Gift Agent 🎁 with MongoDB Atlas

Use this thread for questions and comments 👇

[bilgeyucel]

🥳 Hello everyone, it's a wrap. The final challenge is here: Jingle Metrics All the Way 🔔
This time, it's all about evaluating your RAG pipeline with EvaluationHarness.

⭐️ Bonus task included ⭐️

Do you have questions or comments? Drop them in this thread! 👇

[d-kleine]

As the event is over, would it be possible to provide the solution code for the bonus task (RAGEvaluationHarness for Custom Pipelines) for the day 10 challenge?

I think there are 2 possible correct approaches (at least), just want to double-check 🙂

[bilgeyucel]

Hi @d-kleine, actually, we extended the deadline, and now there's time till Jan 6. After that, we'll reveal the solutions and announce the winners! :)

[d-kleine]

Okay, thanks