feat: Support embedding dimensions on DeepsetCloudDocumentStore

[dmigo]

Related Issues

• fixes #issue-number

Proposed Changes:

• Add embedding_dim to dc store
• Remove similarity and return_embedding from query params, it is not used

How did you test it?

Notes for the reviewer

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added tests that demonstrate the correct behavior of the change
• I've used the conventional commit convention for my PR title
• I documented my code
• I ran pre-commit hooks and fixed any issue

[tstadel]

Removing return_embedding from get_document is fine for now. In the future we might support that in all document stores: #3007. Currently it's just confusing as it has no effect.

[tstadel]

Changes for deepsetcloud look good! However I guess something with your schema generation went wrong. There are multiple types missing that should be there. We first have to fix that before merging. Can you please revert all changes to haystack/json-schemas/haystack-pipeline-master.schema.json? The schema should update automatically during CI...

[anakin87]

Sorry for the intrusion.
I encountered the same problem with schema generation in the past. I add some information to be helpful...

• the CI shows an error, requesting to locally update and commit the JSON schema
• you try to generate the schema but the right schema is generated only if:
  • you have a full installation of Haystack (pip install -e '.[all]')
  • every module is working and importable (update_json_schema in generate_json_schema.py somewhere tries to import all the possible nodes and I found out that in my installation audio nodes were not working.)

[dmigo]

@anakin87 Thanks for the hints! pip install -e '.[all]' is what I try to run at the moment, however some packages fail to install due to M1 probably. So, I'm debugging this behaviour right now.

[dmigo]

To generate a valid schma I did:

1. brew install openblas
2. brew upgrade cmake
3. Excluded onnx from the list of dependencies
4. GRPC_PYTHON_BUILD_SYSTEM_ZLIB=true OPENBLAS="$(brew --prefix openblas)" pip install -e '.[all]'
5. python .github/utils/generate_json_schema.py

[tstadel]

LGTM!