Plug & Play download & configuration of HuggingFace sentence transformer models

[eostis]

(As usual, I'll give the point of view of small projects like WooCommerce ...)

The current way of using HF models is quite technical: build scripts to download models, convert to ONNX, deploy
(https://github.com/vespa-engine/sample-apps/blob/master/transformers/src/python/setup-model.py)

I suggest to ease the process, to be able to use any HF sentence transformer model, by configuration rather than scripts.

Some possibilities:

• Add parameters to the docker compose file
• Add parameters to the schema (most flexible I think)
• Add parameters to the services.xml file

[jobergum]

It's a bit technical, plus people want to quantize, not quantize, and so forth.

[eostis]

Closing inactive issue.

[bratseth]

I think we want to stick with a curated set with a guaranteed lifecycle: https://cloud.vespa.ai/en/model-hub

[eostis]

Reopened!

[jobergum]

Context @eostis ?

[eostis]

I think it would be useful to put some models on the shelves. The onnx generation is a real headache. Maybe by making Vespa Hub downloadable?

[jobergum]

We can probably put a few onnx models on HF. Do you think it is still painful to use Optimum to export models?

[eostis]

It can be tricky:

• bge-small-en ONNX component fails to load with "Opset 19 is under development" #28161 (onnxruntime version)
• [documentation] Add Pytorch 2.0 as a prerequisite for onnx model created with optimum-cli #28007 (Pytorch version)

[jobergum]

Sure, I get that, but it's a general problem with the python world of dependencies that I don't think we can fully address :)