Manually deployed vespa with docker compose Failed to establish a new connection: [Errno 111] #663
Comments
ricoms
changed the title
|
Hi, sorry for slow reponse! The (first) problem to solve is the configproxy not reaching the config server. Please try https://github.com/vespa-engine/sample-apps/blob/master/examples/operations/multinode-HA/docker-compose.yaml and validate that works - then you can modify the compose file, adding your stuff. Maybe best place to look is network/hostnames - this looks like a connectivity problem, so maybe add a network and use a fully qualified hostname instead of just |
|
@kkraune Hi, I try to use bge-m3 model to do embedding hybrid search, and I use refer to official tutorial to deploy a local docker containter to use vespa. Since I have over millions data to feed, so I try to use feed_iterable function to feed iterable bulk data, and I encountered same problems as above, like my env: linux, pyvespa version is 0.39, docker image is latest |
|
Hi @Gladiator566 I think you must look in the vespa.log to validate what the problem might be - and if so, follow the advise to try /multinode-HA/docker-compose.yaml to verify this works, before trying your own configuration You can also try https://pyvespa.readthedocs.io/en/latest/getting-started-pyvespa-cloud.html to make it easier, using the free trial, to eliminate other failures. |
|
@kkraune I try to use vespa cloud as tutorial, but i got error like |
|
Thanks for reporting. Can you add the steps you took, so we can reproduce? Or did you follow the steps in https://pyvespa.readthedocs.io/en/latest/getting-started-pyvespa-cloud.html and it failed here? A good hint is also to make sure there are no applications already deployed. @hmusum I assume this is an error from our API, we should document how to fix this |
|
@kkraune yes, I follow the exact steps in https://pyvespa.readthedocs.io/en/latest/getting-started-pyvespa-cloud.html , concretely as bellow:
and it failed at Thanks. |
|
Hi again, I tried https://pyvespa.readthedocs.io/en/latest/getting-started-pyvespa-cloud.html and it worked for me. I run the notebook locally on my laptop. Some ideas the |
|
The "Value of X-Content-Hash header does not match computed content hash" error is due to some misconfiguration or bug on the client side, but it's hard to say what the user should do without knowing the root cause of the error. |
|
The problem seems to be a mismatch with the hash computed in https://github.com/vespa-engine/pyvespa/blob/master/vespa/deployment.py#L644 and validation of this in Vespa Cloud. pyvespa 0.39, which is the latest. We are looking into. |
|
@Gladiator566 : I have created vespa-engine/vespa#30219 for the X-Content-Hash issue you reported - this is most likely a different issue than reported by @ricoms here. Thanks for reporting! |
Hi, I also encounter the same problem. Did you know how to fix it? |
|
Hi @vudangthinh ! I don't think increasing number of connections will help, the error message is probably a symptom of a maxed out instance. The https://docs.vespa.ai/en/vespa-cli.html has better feed flow control, can you please try that and see how the feeding goes and let me know? |
|
I tried to use vespa feed, however the error still persistent:
|
|
OK - can you please check vespa.log inside the Docker Container? Could be a resource problem, the log might say |
|
You are probably sending more requests than the system can handle timely and therefore some of them end up crossing a connection recycling event. These will be retried until timeout so not really an error in itself, but you probably want to increase your resources (maybe run with GPU) or feed slower. Setting a lower timeout (--timeout) should get rid of these messages and lead to less queuing which is probably advantageous if you want to determine faster what actual max throughput you can get. |
hi.
I raised Vespa locally with the following docker compose:
The services
locust-workerandlocust-masterbasically uses pyvespa to create the aApplicationPackagewith a defined schema and then I create a vespa client withVespa(from vespa.application module). When I try to usefeed_iterableI get the following errors on mylocust-workerapplication.And here is the full vespa log:
I noticed 2 things from Vespa logs:
2024-01-18 04:54:22 [2024-01-18 07:54:22.097] INFO configserver Container.com.yahoo.container.jdisc.ConfiguredApplication Switching to the latest deployed set of configurations and components. Application config generation: 0which I saw a video which mentions that that is expected and should be ok.configproxyis retrying to connect to something and never connects.From this I have a comment about pyvespa docs: I did not see any document that is not using the
VespaDockerdeployment approach. Is there a way to apply aApplicationPackageto an existing locally deployed Vespa with pyvespa?Also, I tested out
wait_for_application_up(120)available fromVespaimplementation, but it always throw a timeout error. I also looked around its code to find a "deploy" action of the schema, but I was not able to find anything like that.The text was updated successfully, but these errors were encountered: