Missing embeddings in MostSimiliarDocumentsPipeline and query_by_embedding

[kaeppen]

Hello

I am trying to use the MostSimiliarDocumentsPipeline and keep getting the following error:
"elasticsearch.exceptions.RequestError: RequestError(400, "search_phase_execution_exception: Likely some of your stored documents don't have embeddings. Run the document store's update_embeddings() method.", 'compile error')"

I also get this error when i try to run query_by_embedding directly on my document store (which is Elasticsearch).

My code is based on the example provided by bogdankostic in this discussion: #1496

I have tried the folllowing:

• Made sure that my documents are not short as described in the link
• Called document_store.update_embeddings
• Pulled out all documents and printed their embeddings - they all have values / are not None (return_embedding=True)
• Printed "document_store.get_embedding_count()" on the correct index - it is 10 (matching my number of documents)
• Set document_store.return_embedding to True
• Extracted the id of the first document i pull out, print its embeddings (not None) and use them for the search

All of these experiments seem to point to the fact that my documents actually have embeddings, but i still get the error.
The exception is thrown around line 1239-1245 in elasticsearch.py because "self.embedding_count" returns 0 but when i make the exact same call myself i get a non-zero value.
Also, if i use a random word as id in MostSimiliarDocumentsPipeline.run, it does not throw an exception but just returns an empty list of results.

I have traced through the haystack source code and can't seem to find anywhere i am missing a parameter or anything like that.
I have also searched through previous discussions to resolve my issue.

Am i trying to use this the wrong way or is something else at fault?
I am sorry if i missed something very obvious.

I have attached a file with the code i am running.
haystack_post.txt

Sebastian

[bogdankostic]

Hi @kaep! Thanks for the detailed description of your problem and for providing code to reproduce this. Unfortunately, your code works just fine on my side. Neither query_by_embedding nor the MostSimilarDocumentsPipeline's run method trigger an error on Elasticsearch side using current Haystack master and Elasticsearch version 7.9.2. What version of Haystack and Elasticsearch are you using?

[kaeppen]

Hello @bogdankostic
Thank you for the fast reply!
I am using the version 7.5.1 docker image of Elasticsearch as well as "farm-haystack" from pip.
I must admit that i don't know why i am using 7.5.1, up until now i was sure that i was using the same version as in the docker-compose provided with your demo... Will try with ersion 7.9.2 asap.

Update:
I just tried again with Elasticsearch 7.9.2 and it seems to be working.
Thank you very much for the help.
I am quite embarrassed that i didn't even consider whether i was using the correct version of Elasticsearch.

[bogdankostic]

Great to hear that changing Elasticsearch versions already solved your problem :)