Slow search

[kamaljai]

I have this following codeset which takes 30 seconds to return a response. I'm not sure how to make it fast

document_store = ElasticsearchDocumentStore(host="server.lan", username="", password="",
index="insurance", embedding_dim=768,
embedding_field="embedding")

retriever = ElasticsearchRetriever(document_store=document_store)


print(retriever.retrieve(query=query))

if modelname == "roberta":

    reader = FARMReader(
        model_name_or_path="deepset/roberta-base-squad2",
        use_gpu=False,
        num_processes=8

    )

else:
    reader = FARMReader(
        model_name_or_path="my_model" + "/" + modelname,
        use_gpu=False,
        num_processes=8
    )


pipeline = ExtractiveQAPipeline(reader, retriever)

result = pipeline.run(query=query, top_k_retriever=10,top_k_reader=3)

return result

[aantti]

Hi @kamaljai,

We've recently published a series of articles on Haystack's performance here: https://medium.com/deepset-ai/parameter-tweaking-get-faster-answers-from-your-haystack-pipeline-b94f0f528111

In general, without a GPU, inference could be pretty slow.

Hope this helps.