Shaping Haystack 2.0

[bilgeyucel]

Since Haystack v1.15, we’ve been slowly introducing new components and features to Haystack in the background in preparation for Haystack 2.0 (or v2). After the work we’ve put into the new design of the Haystack API over the last few months, we’re at a point where we would love to start involving the Haystack community in our thought process and slowly gather your input and feedback. In this discussion, we would like to highlight where we are at for the design of the new Haystack API for 2.0, what we want to achieve with the new design, and what our current considerations are.

❓ What does the new 2.0 version mean?

Haystack 2.0 will be a major update to the design of Haystack nodes and pipelines. We believe that the pipeline concept is a fundamental requirement and an optimal fit for building applications with LLMs. Therefore, Pipelines and Nodes will continue to be the foundation of Haystack 2.0. However, the general pipeline structure, Nodes API, and the connection between DocumentStore and Retrievers will change. So, this will be a breaking change for Haystack users.

🏆 Motivation behind Haystack 2.0

At deepset, we put a lot of thought and care into maintaining Haystack as a robust, user-friendly, and production-ready LLM framework. As we have collected feedback from the Haystack community over the years and observed the advancements in the NLP field, such as LLMs and Agents, we see the need to update the pipeline structure with Haystack 2.0 to better align with our users’ needs and state-of-the-art NLP approaches.

When ready, Haystack 2.0 will introduce many improvements, flexibility and, most importantly, it will allow Haystack users to implement customizations and extensions to Haystack much more easily. The new pipeline structure will allow for more flexible, robust, and powerful pipelines. As we change the pipeline structure, we’ll be adapting all components to the new structure, therefore, rewriting many of them. This update gives us the opportunity to enhance the pipeline structure to better make use of LLMs, improve our Agent and Memory implementations, better define the connection between the DocumentStore and Retriever, and so on.

📍 Current status of Haystack 2.0

Haystack 2.0 is still a work in progress. We are defining the requirements for a more powerful and robust LLM framework with continuous feedback from the community, and we’re implementing the new Haystack API so that it’s aligned with the advances in NLP.

Although still in beta, you can find what’s been implemented so far in the preview package of the Haystack repository. To learn how and when components will be migrated, have a look at the Migrate Components to Pipeline v2 roadmap item, where we keep track of issues and PRs about Haystack 2.0. For a detailed overview of the current state of 2.0, check out Sara’s presentation about Haystack 2.0.

Additionally, here is the complete list of proposals so far shaping the design of Haystack 2.0:

• Drop BaseComponent and re-implement Pipeline 🟢
• DocumentStores and Retrievers in Haystack 2.0 🟢
• New EmbeddingRetriever for Haystack 2.0 🟢
• Embedders design for Haystack 2.0 🟢
• LLM support in Haystack 2.0 🟢

🧱 Implemented 2.0 Components and DocumentStores

Using implemented components and document stores, you can already start to:

• Build a Generative QA Pipeline using RAG approach with Generators and Prompt Builder
• Build an Extractive QA Pipeline with Readers
• Build a Web QA Pipeline with Web Search
• Preprocess documents with PreProcessors and File Converters
• Index documents to DocumentStore with Writers and Embedders
• Transcribe audio files with Whisper Transcribers
• ...

Full List of Components

Type	Components
Audio	LocalWhisperTranscriber, RemoteWhisperTranscriber
Builders	AnswerBuilder, PromptBuilder
Caching	UrlCacheChecker
Embedders	OpenAIDocumentEmbedder, OpenAITextEmbedder, SentenceTransformersDocumentEmbedder, SentenceTransformersTextEmbedder
Fetchers	LinkContentFetcher
File Converters	AzureOCRDocumentConverter, HTMLToDocument, PyPDFToDocument, TikaDocumentConverter, TextFileToDocument
Generators	GPTGenerator
PreProcessors	TextDocumentSplitter
Readers	ExtractiveReader
Retrievers	MemoryBM25Retriever, MemoryEmbeddingRetriever
Routers	FileExtensionRouter, MetadataRouter
Web Search	SerperDevWebSearch
Writers	DocumentWriter

Full List of Document Stores

Document Stores
MemoryDocumentStore, ChromaDocumentStore, MarqoDocumentStore

⭐ Highlights of Haystack 2.0

• Pipeline Nodes will be now called Components.

• The new pipeline structure will provide better support for LLMs. The flexible connection between components will introduce new mechanisms, such as parallel branching and looping, that extend the capabilities of pipelines. Components will control the input and output of the pipeline. Thus, components with dynamic input parameters, such as those that use prompts with variables, will easily integrate into the pipeline. Overall, these refinements will not only improve the linear workflows but also ensure that pipelines seamlessly align with the nature of LLMs.

  Here is what a RAG pipeline might look like in Haystack 2.0.👇🏼

  Keep in mind that the components are still work in progress and being discussed in the “LLM Support in Haystack 2.0” proposal.

  

  Representation of a RAG pipeline in Haystack 2.0

• The Components API will change. Components will define the name and the type of all of their inputs and outputs. The new API will reduce complexity and make it easier to create custom components such as Haystack integrations for third-party APIs and databases. The connections between components will be validated before running the pipeline, and Haystack will generate better error messages with instructions on fixing the errors.

• Retrievers will be customized for DocumentStore, not for retrieval methods. Each DocumentStore will have its own Retriever, highly specialized for that specific DocumentStore, handling all its requirements without being bound to a generic interface. Integrating a new DocumentStore will be easier, and the specialized Retriever will be able to adapt more quickly to the new features of the DocumentStore.

• The Embedder will be a separate component instead of being a part of a Retriever. Retrievers won’t be responsible for creating embeddings, the new Embedder component will handle the creation of embeddings. The Retriever class will be simplified, and adding support for new embedding providers and approaches will be more straightforward.

• Pipeline serialization will be more flexible and optimized for humans. JSON, TOML, HCL will be used as serialization formats. Serialization and deserialization of pipelines sharing the same component instance will be possible.

➡️ What’s next?

As we iterate on Haystack 2.0, we’ll update this discussion regularly to reflect the latest changes. We’ll share the design proposals with you in the comments below, update the list above as well and start a conversation about topics where we need your input. As we share more information about Haystack 2.0, please feel free to share your feedback or concerns. If you’d like to get notified when there is an update about Haystack 2.0, subscribe to this entry. You can always contact us using the comments section or the Haystack Discord server to ask questions.

[masci]

It's now available a PyPI package shipping the code in haystack.preview alone. It's built on the main branch so it's highly unstable, but it's useful if you want to try the new features as soon as they are merged, or if you're working on a custom component or document store:

pip install haystack-ai

This is already used by the (work in progress) Chroma Document Store, see its pyproject.toml

[ZanSara]

Sentence Transformers Embedders for Haystack 2.x

In our recent Embedders proposal (https://github.com/deepset-ai/haystack/blob/c38943721fbd702367a8934cb660a72b3143eb86/proposals/text/5390-embedders.md) we defined how embedding text should work in Haystack 2.x. Today, thanks to @anakin87 , we added the first building blocks of this new architecture: SentenceTransformersTextEmbedder and SentenceTransformersDocumentEmbedder, along with their supporting embedding backend.

On their own, Embedders simply take some textual data, be it raw strings or whole Documents, and create an embedding for it. Soon you will also be able to use these Embedders to perform dense retrieval, so stay tuned!

For more information, check out the relevant issue: #5567

[ZanSara]

LLM Support in Haystack 2.0 - Proposal

The proposal for LLM support in Haystack 2.0 has been finally merged 🎉

Here the Proposal's text: https://github.com/deepset-ai/haystack/blob/main/proposals/text/5540-llm-support-2.0.md and here you can find an earlier thread on the same topic: https://discordapp.com/channels/993534733298450452/1141684516709212160/1141684516709212160

The key takeaway from this Proposal is that we're breaking down PromptNode into smaller components.

• PromptBuilder transform prompt templates into the actual text to be sent to the LLM.

• Generators are responsible for generating text given a prompt, specific for each LLM technology (OpenAI, local, TGI, etc).

• AnswerBuilder, DocumentBuilder etc.. are a class of components able to convert the output of LLMs into Haystack objects to be further propagated down a pipeline.

This separation should give you more control over the LLM, how is queried, and adds visibility to what it produces. The tradeoff is a more verbose pipeline definition, but we're collecting user feedback to better understand the impact and potential solutions.

The implementation of these components has already started, so soon you can expect more announcements on this topic and working examples 🚀

[dkbs12]

Hello, bilgeyucel.
It's to happy to hear you're preparing Haystack 2.0 now.
I'm looking forward to see it and it must be a fantastic one.
By the way, I have a question about the Haystack 2.0.
As you know, BM25 is not applicable for Pinecone according to DocumentStore Compatibility now.
Is it possible to use BM25 for Pinecone in Haystack 2.0?

[bilgeyucel]

Hi @dkbs12, in Haystack 2.0, every DocumentStore will have its own specialized Retriever, so you can expect Haystack to cover more features of Pinecone, including BM25 and Hybrid Search.

[dkbs12]

Thank you for your quick reply.
I have another request.
When I run 'print_answers' and 'print_documents', I feel inconvenienced by not having other components such as score, context and name.
Could you add the factors as 'score' and 'context' in the function 'print_documents' as well as add factors as 'score' and 'name' in the function 'print_answers'?
If that could happen, I think it would be a great help to me.

[bilgeyucel]

Hi @dkbs12, thank you the feedback.

For print_documents, you should be getting content instead if context but score is missing.
For print_answers function, I don't know what you refer by 'name' but you can get the 'score' if you use the function with details="medium" 👇

print_answers(prediction, details="medium")

And if these attributes can be found in Answer and Document objects, you can easily print them manually without using print functions. If these attributes is missing in the return objects and you think that these would be a nice addition, feel free to open an issue for them! 😊

[bogdankostic]

AnswerBuilder component now available!

We just merged a PR that introduces a new component for Haystack 2.0 - the AnswerBuilder: #5701

The AnswerBuilder component is typically used in a RAG-Pipeline after an LLM Generator component. It extracts answer strings and referenced documents from the string output of the generator models based on regular expression patterns and packages them into GeneratedAnswer objects.

[silvanocerza]

PromptBuilder is now available

We just merged the PR that introduces the new PromptBuilder component for Haystack 2.0.

This component will be used mostly in Pipelines that comunicate with LLMs, like in Retrieval Augmented Generation. It uses the powerful Jinja template engine, so a template can be a really simple one or more advanced and contain custom logic. See the official Jinja template designer documentation for more information.

A simple standalone usage could look like this.

from haystack.preview.components.generators.prompt_builder import PromptBuilder

template = "Translate the following context to {{ target_language }}. Context: {{ snippet }}; Translation:"
builder = PromptBuilder(template=template)
res = builder.run(target_language="spanish", snippet="I can't speak spanish")

print(res["prompt"])
# Translate the following context to spanish. Context: I can't speak spanish; Translation:

Or a more advanced one with list of documents:

from haystack.preview.components.generators.prompt_builder import PromptBuilder

documents = [
    "This is a really important document",
    "This is not so important",
    "This is completely unrelated"
]
template = """Given the context please answer the question.
Context: 
{% for doc in documents %}
    {{- doc -}};
{% endfor %}
Question: {{ question }};
Answer:
"""

builder = PromptBuilder(template=template)
res = builder.run(documents=documents, question="What is the most important document?")

print(res["prompt"])
# Given the context please answer the question.
# Context: 
# This is a really important document;
# This is not so important;
# This is completely unrelated;

# Question: What is the most important document?;
# Answer:

If you're ready to give it a try just install the preview package with pip install haystack-ai and you're ready to go!

[ZanSara]

SerperDevWebSearch is now available

Thanks to @vblagoje's work on #5712 we now have a SerperDevWebSearch component for Haystack 2.0! 🎉

SerperDev is a Google API based search engine, and SerperDevWebSearch brings web searching capabilities to Haystack. Give it the same question you'd enter on Google and it returns a list of links to the most relevant webpages.

Here's a code snippet demonstrating how to use it:

from typing import List
from haystack.preview import Document
from haystack.preview.components.websearch import SerperDevSearchAPI

ws = SerperDevSearchAPI(api_key="insert_your_key_here")  # Go to https://serper.dev/ to get one
results: List[Document] = ws.run(query="Who is the boyfriend of Olivia Wilde?")
print(results)

SerperDevWebSearch is just the first of a series of Web oriented components coming soon to Haystack. See the roadmap here.

[ZanSara]

GPT35Generator released 🚀

We just merged our first Generator component, GPT35Generator: this is the last building block that enables Haystack 2.0 to use LLMs in pipelines.

For now the support provided by GPT35Generator is very basic: it is capable of querying OpenAI's GPT3.5 models with a single query, and does not support longer conversations yet. However, combined with PromptBuilder and AnswerBuilders, it finally makes possible to build Retrieval-Augmented Generative (RAG) Pipelines with Haystack 2.0! 🎉

Here is how you can use this component in isolation:

import os
from haystack.preview.components.generators.openai.gpt35 import GPT35Generator

component = GPT35Generator(api_key=os.environ.get("OPENAI_API_KEY"), n=1)
results = component.run(prompts=["What's the capital of France?", "What's the capital of Germany?"])
print(reuslts)

Stay tuned for some more demos and example code.

See the PR to learn more: #5714

[ZanSara]

GPT35Generator also supported GPT4, but we've now added an additional component called GPT4Generator too: #5744. The only difference with GPT35Generator is that it defaults to gpt-4 as the model name. More Generators soon to come 🚀

[anakin87]

MemoryEmbeddingRetriever!

We just merged MemoryEmbeddingRetriever, which allows semantic/dense/embedding retrieval in our simple MemoryDocumentStore.
For in-depth information on dense and sparse retrieval, see this page.

A step-by-step example

from haystack.preview.document_stores import MemoryDocumentStore
from haystack.preview.components.embedders.sentence_transformers_document_embedder import SentenceTransformersDocumentEmbedder
from haystack.preview.components.embedders.sentence_transformers_text_embedder import SentenceTransformersTextEmbedder
from haystack.preview.components.retrievers.memory import MemoryEmbeddingRetriever
from haystack.preview.dataclasses import Document

# initialize the Document Store
ds = MemoryDocumentStore(embedding_similarity_function="cosine")

docs = [
    Document(content="fish", metadata={"name": "doc1"}),
    Document(content="some cars", metadata={"name": "doc2"}),
    Document(content="an elephant and a mouse", metadata={"name": "doc3"})
]

# initialize a DocumentEmbedder to embed your Documents
doc_embedder = SentenceTransformersDocumentEmbedder(model_name_or_path="sentence-transformers/all-mpnet-base-v2")
doc_embedder.warm_up()

# embed your Documents and write them in the Document Store
docs = doc_embedder.run(docs)["documents"]
ds.write_documents(docs)

# initialize a TextEmbedder to embed your query
query_embedder = SentenceTransformersTextEmbedder(model_name_or_path="sentence-transformers/all-mpnet-base-v2")
query_embedder.warm_up()

# embed your query
query_embedding = query_embedder.run(["animals"])["embeddings"][0]

# initialize a Retriever to retrieve the relevant documents using semantic similarity
retriever = MemoryEmbeddingRetriever(document_store=ds)
retrieved_docs = retriever.run(queries_embeddings=[query_embedding])['documents'][0]

for doc in retrieved_docs:
    print(doc.content)
    print(doc.score)

# an elephant and a mouse
# 0.7709958700352771
# fish
# 0.7398794287674773
# some cars
# 0.7083275654188841

Other examples

Notebook on using MemoryEmbeddingRetriever in pipelines

Related PRs

• MemoryDocumentStore - Embedding retrieval (2.0) #5715
• MemoryEmbeddingRetriever (2.0) #5726

[bogdankostic]

TikaDocumentConverter is now available!

We just merged a PR (#5847) that allows you to to convert files of various types like txt, pdf, or docx to Document objects using Tika as the content detector.

[ZanSara]

UrlCacheChecker component released!

The UrlCacheChecker can be used to speed up Web Retrieval in Haystack 2.0. With the introduction of WebSearch and LinkContentFetcher components, web search is becoming soon a possibility for the retrieval step of RAG pipelines. UrlCacheChecker moves one more step towards this goal: it allows you to make sure you haven't retrieved data from any of the webpages returned by WebSearch, and if so, returns the already pre-processed documents from the document store instead of downloading them from the web again.

graph TD;

IN{IN} -- query --> WebSearch
IN{IN} -- query --> PromptBuilder
WebSearch --links --> UrlCacheChecker
UrlCacheChecker -- misses --> LinkFetcher
LinkFetcher  -- pages --> HTMLToDocumentConverter
HTMLToDocumentConverter -- docs --> TextSplitter
TextSplitter -- docs --> Join
UrlCacheChecker -- hits --> Join
Join -- docs --> PromptBuilder
PromptBuilder -- prompt --> GPTGenerator
GPTGenerator -- replies --> AnswerBuilder
AnswerBuilder -- answers --> OUT{OUT}

See the UrlCacheChecker PR: #5841 and the Web Retrieval epic: #5614

[ZanSara]

ExtractiveReader is now available

Thanks to the efforts of @MichelBartels we're proud to announce the release of ExtractiveReader for Haystack 2.0! 🎉

ExtractiveReader adds traditional Extractive Question Answering capabilities to 2.0 pipelines. While designed to resemble Haystack 1.x's FARMReader, ExtractiveReader implementation is both smaller and faster, while maintaining a performance that closely matched its predecessor.

See the detailed discussion about the differences between these two components in the PR: #5553

And here is how you can try this new component:

docs = [[Document(content="Angela Merkel is the chancellor of Germany."), Document(content="Olaf Scholz is the chancellor of Germany")], [Document(content="Jerry is the head of the department.")]]
queries = ["Who is the chancellor of Germany?", "What is Jerry's role?"]
reader = ExtractiveReader("deepset/tinyroberta-squad2")
p = Pipeline()
p.add_component("reader", reader)

print(p.run({"reader": {"documents": docs, "queries": queries}}))

[vblagoje]

PyPDFToDocument component is now available

PyPDFToDocument component is designed to convert PDF files into a list of Document objects, which can then be seamlessly used in Haystack 2.0 pipelines.

For more details, see this PR

Here is how you can use this component:

from haystack.preview.components.file_converters.pypdf import PyPDFToDocument

paths = [preview_samples_path / "pdf" / "react_paper.pdf"]
converter = PyPDFToDocument()
output = converter.run(paths=paths)
docs = output["documents"]
assert len(docs) == 1
assert "ReAct" in docs[0].text

[vblagoje]

LinkContentFetcher component released

LinkContentFetcher is responsible for fetching content from a given URL and converting it into a Document object, which can then be used in your Haystack 2.0 pipeline.

For more details, see #5724

Here is how you can use this component:

from haystack.preview import Document
from haystack.preview.components.fetchers import LinkContentFetcher

lcf = LinkContentFetcher()
doc: Document = lcf.fetch(url="example.com")
print(doc)

[TuanaCelik]

A PSA on this component. It was changed to return ByteStream rather than Document type.
This change means the intended use of the component is now:

from haystack.preview.components.fetchers import LinkContentFetcher

lcf = LinkContentFetcher()
streams = lcf.run([url="example.com"])["streams"]

The PR that introduced the change: 6a50123

[ZanSara]

Extractive QA now supported

With the release of ExtractiveReader, now Haystack 2.0 supports traditional Extractive Question Answering pipelines alongside RAG. Here is an example of such pipeline:

from haystack.preview import Pipeline, Document
from haystack.preview.document_stores import MemoryDocumentStore
from haystack.preview.components.retrievers import MemoryBM25Retriever
from haystack.preview.components.readers import ExtractiveReader


document_store = MemoryDocumentStore()
documents = [
    Document(text="My name is Jean and I live in Paris."),
    Document(text="My name is Mark and I live in Berlin."),
    Document(text="My name is Giorgio and I live in Rome."),
]
document_store.write_documents(documents)

qa_pipeline = Pipeline()
qa_pipeline.add_component(instance=MemoryBM25Retriever(document_store=document_store), name="retriever")
qa_pipeline.add_component(instance=ExtractiveReader(model_name_or_path="deepset/tinyroberta-squad2"), name="reader")
qa_pipeline.connect("retriever", "reader")

question = "Who lives in Paris?"

result = qa_pipeline.run({"retriever": {"query": question}, "reader": {"query": question}})

print(result["reader"]["answers"].data)

[dkbs12]

Does Haystack 2.0 include HyDE(Hypothetical Document Embeddings) as a retrieval method?

[bilgeyucel]

Hi @dkbs12, yes, the pipeline structure in 2.0 is flexible enough for HyDE approach. Basically, what you would do is connect the output of the Generator to the Retriever.

[sandangel]

Hi, is there an update on this discussion? I think this has not been updated for a while, so I wonder what is the current status and when should we expect v2 to be ready.

[TuanaCelik]

Hey @sandangel Good shout, this discussion went unnoticed sorry about that! Here are some updates that will probably be useful to others too:

1. On December 4th we released Haystack 2.0-Beta, it's still in beta because development will continue until the end of Q1 this year with support for 1.x continuing as well. BUT the good news is that you can install the 2 versions via different packages so it's quite clean to work with one ot the other. pip install haystack-ai for 2.0-Beta
2. You can read the announcement for this in our article announcing the Beta release
3. Here, you can also find the 'release notes' for the Beta version, with an extensive table showing current status at the bottom. I'll add that table to this discussion separately too.
4. We also hosted an 'Advent of Haystack' starting with the beta release with the intention to have people try out the first official commitment to 2.0 and give us feedback

[vblagoje]

@sandangel the updates stopped when we merged Haystack 2.x preview to main. Simply follow the development on the main branch. The old main branch is now 1.x branch. We have recently released beta3 of Haystack 2.x and expect these beta releases to continue. There is no definite cutoff date for the 2.0 final but it should come soon-ish.

[TuanaCelik]

Status Update 🚀

• Haystack 2.0-Beta was made available on Dec 4th
• 1.x support continues in the meantime so you'll see 2 types of releases: supporting 1.x with bugfixes and continued inprovements
• On top of the table you saw for the 2.0-Beta release notes, it's safe to say that we now also have support for: Pinecone, Qdrant, Jina embedding models and models via Google Vertex AI. Expect to see an updated version of that table for following 2.0 releases.
• If you're interested in all the additional integrations that become available for 2.0, this is a good table to look at too on haystack-core-integrations
• In the coming days, the roadmap for Q1 will be made public

[julian-risch]

@sandangel The roadmap for Q1 is public now: https://github.com/orgs/deepset-ai/projects/3

[sandangel]

@TuanaCelik @vblagoje Thank you so much for the update. I will follow the development on main branch.
In that case, shall we close this discussion and redirect others to follow other sources for the status update?

[bilgeyucel]

Hello everyone, we have just published a new discussion entry: Haystack 2.0-Beta. The new discussion will serve as your ultimate guide until the stable release of Haystack 2.0.
We won't be updating this discussion anymore. Feel free to subscribe to it to stay in the loop on all upcoming updates 🚀

[TuanaCelik]

Closing this discussion in favor of the Haystack 2.0-Beta discussion following the beta release.