New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Quick-start tutorial runs with internal timeouts #3640
Comments
yaps |
Thank you for your detailed ticket. We'll look into this. |
The docker container quick start http://docs.vespa.ai/documentation/vespa-quick-start.html uses https://github.com/vespa-engine/docker-image/blob/master/include/start-container.sh as ENTRYPOINT and will start both vespa-configserver and vespa-services in parallel if no arguments are present (It should be given arguments in a multi-node installation). Since the configuration server is slower to start then vespa-services you get these could not connect errors. They are transient though. Here we see configuration server is initialising bundles while configproxy (which is a process started by vespa-services) tries to talk to it over RPC on port 19070.
The remaining log entries will be printed until you have deployed a application package (http://docs.vespa.ai/documentation/cloudconfig/application-packages.html) which would describe which services should be started by the sentinel process. See also #3588 as it has a few pointers.
|
Sorry, I think I got confused by the many different ports mentioned. While the errors about 19070 happen, the ones that keep coming over and over are about port 19090:
They also keep coming after preparing a sample application with Re Kubernetes: I'm running this in a single container locally in Minikube. I'm not yet trying out a multi-node setup. (Protip: I recommend using triple backtics when pasting chunks of raw text, otherwise your lines will be wrapped and become unreadable.) |
http://localhost:8080/ApplicationStatus does return |
Edit: Ignore that last comment. That is mapped to the port of the config server. Port 8080 is not operational. |
Thanks for the update. How much memory do you have available in the Minikube? From a few searches it seems like 1G is the default and that is going to be insufficient. Vespa is a beast with a lot of java jvm's with rather high max heap settings and I'm suspecting that your trouble is from insufficient memory available inside the Minikube. Does it work if you enable more memory? 4G is great, 2G sufficient for getting through sample quick start on docker. Thanks for the pro tip on quotes. These are what I found from the log snippets provided in the gist.
Which is great, then finally sentinel gets configuration and tries to start the set of services
So no connectivity issues, but then sentinel tries to start these services which again will subscribe to new configuration and this is when it starts to to go down badly where all of them complain about not being able to subscribe to configuration. I suspect that configuration server has been killed due to oom due to potentially memory constraints when starting all these processes. |
. |
@atombender were you able to make any progress on this issue? Thanks! |
@jobergum Sorry, I've been taking a few days off from work. I'll get back to you. But I did try starting with 4GB of RAM, with no difference in behaviour. Shouldn't the logs say if something is being forcibly killed due to memory usage? |
Thanks for the update, Yes, I was wrong. We can see the configuration server is alive and fine from the log. The configurations server fails to produce configuration due to this
Inside your docker container, what is the output of
Your manifest https://gist.githubusercontent.com/atombender/2c24a899ce8d051396ba0e97bba6822f/raw/8523e9c09834882d857267c1f78b1981c2750f99/statefulset.yml states
Data in Vespa is stored in /opt/vespa/var/db/vespa/search/ and I wonder if any of these volume settings have overwritten the config def installation files but not familiar with the syntax above. |
The database volume is initially empty when Vespa starts for the first time. I'm not populating that volume with anything — all that data is exclusively being written to by Vespa. Unfortunately: $ kubectl exec -it vespa-0 -- cat /opt/vespa/var/db/vespa/config_server/serverdb/serverdefs/search.config.qr-start.def
cat: /opt/vespa/var/db/vespa/config_server/serverdb/serverdefs/search.config.qr-start.def: No such file or directory Here's a tarball of the whole |
So that is the problem then.
Without these def files configuration server is not able to serve configuration. Please try change the mountPath to /opt/vespa/var/db/vespa/search. |
D'oh. I just assumed that this would be the data folder and that Vespa would behave correctly if it was initially empty. I've never actually encountered an application before that puts non-volatile data under a folder named Does this mean that data in Started up again, looks much better now. |
Great, finally progress. I see that http://docs.vespa.ai/documentation/reference/files-processes-and-ports.html could need some work. We do run the exact same docker images in production so we should be able to provide some guidance on volumes etc on this in our documentation but little can be found on this subject today unfortunately. Please have patience while we build out the documentation on this. $VESPA_HOME/var/db/vespa/search has all the index & storage data so you want to persist that, no state for search or storage should be stored outside of that directory. |
Thanks, this was helpful. Closing this since it seems to get me further in the tutorial. |
Would you mind sending the steps needed for a Minikube version of the quickstart? |
@bratseth: Sure, given these manifests, something like this: # Start Minikube
$ minikube start --vm-driver=xhyve --memory 4096
# Mount sample apps, assuming they've been checking out in current dir
$ minikube mount $PWD/sample-apps:/data/vespa-sample-apps &
# Install statefulset and its service
$ kubectl apply -f service.yml statefulset.yml
# Wait for success by monitoring until pod shows it's running:
$ kubectl get pod --watch
# Deploy sample app
$ kubectl exec -it vespa-0 -- bash -c '/opt/vespa/bin/vespa-deploy prepare /vespa-sample-apps/basic-search/src/main/application/ && /opt/vespa/bin/vespa-deploy activate' |
@atombender or anyone else It seems that i need to have multiple (at least two ports 80(80) and 19071 on the same domain open) and ingress does not support that POSTING to and specify Anyone has a suggestion how i can create the ingress exposing needed ports ? Service:
Ingress:
|
Gist with log files,
ps
output,netstat
output, Kubernetes manifest.I don't know if this is an actual error or not, but I get lots of log entries about not being able to connect to the config server at port 19070, lines like this:
I'm starting Vespa under Kubernetes, using the exact commands here. One difference is that Kubernetes is setting the host name, but I don't see why that matters. Using curl, I can confirm that connections to port 19070 hang and eventually time out.
From what I can tell, the config server has been started. The config sentinel (PID 833 in the
ps
output), however, keeps restarting.Using
netstat
, I see that the config server process is indeed listening on port 19070.Port 19070 is operational.
The text was updated successfully, but these errors were encountered: