Load main page error

[tigermask1978]

Details

All docker servers are up, when I load the web ui, something is wrong as the image shows:

[Weves]

Hey @tigermask1978, could you try pulling up the logs from the api_server container? I suspect something is going wrong there

[tigermask1978]

Hi @Weves, Yes,the problem is the api_server cannot download the Embedding model "thenlper/gte-small" from the huggingface because of my network envrionment. So how to configure Danswer to use the local model? Thanks.

[redcathode]

I've been getting the same issue: the danswer-stack-api_server-1 container keeps restarting. Figure I can just paste the logs here:

danswer-stack-api_server-1     | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.                                                                                                                                                                                                                                                                
danswer-stack-api_server-1     | INFO  [alembic.runtime.migration] Will assume transactional DDL.                                                                                                                                                                                                                                                              
danswer-stack-api_server-1     | 2024-01-05 23:01:21.030285: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered                                                                                                           
danswer-stack-api_server-1     | 2024-01-05 23:01:21.030339: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered                                                                                                            
danswer-stack-api_server-1     | 2024-01-05 23:01:21.030366: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered                                                                                                        
danswer-stack-api_server-1     | 2024-01-05 23:01:22.101697: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT                                                                                                                                                                                                   
danswer-stack-api_server-1     | /usr/local/lib/python3.11/site-packages/transformers/utils/hub.py:128: FutureWarning: Using `DISABLE_TELEMETRY` is deprecated and will be removed in v5 of Transformers. Use `HF_HUB_DISABLE_TELEMETRY` instead.                                                                                                              
danswer-stack-api_server-1     |   warnings.warn(                                                                                                                                                                                                                                                                                                              
danswer-stack-api_server-1     | INFO:     Started server process [10]                                                                                                                                                                                                                                                                                         
danswer-stack-api_server-1     | INFO:     Waiting for application startup.                                                                                                                                                                                                                                                                                    
danswer-stack-api_server-1     | 01/05/2024 11:01:25 PM             users.py  68 : Using Auth Type: disabled                                                                                                                                                                                                                                                   
danswer-stack-api_server-1     | 01/05/2024 11:01:25 PM              main.py 217 : Danswer API Key: dn_fBgW2_BUa2cqOBm9JWcwkPaP41Sg5hmh-bMz-djM                                                                                                                                                                                                                
danswer-stack-api_server-1     | 01/05/2024 11:01:25 PM              main.py 225 : Using LLM Provider: openai                                                                                                                                                                                                                                                  
danswer-stack-api_server-1     | 01/05/2024 11:01:25 PM              main.py 226 : Using LLM Model Version: gpt-3.5-turbo                                                                                                                                                                                                                                      
danswer-stack-api_server-1     | 01/05/2024 11:01:25 PM          chat_llm.py  48 : LLM Model Class: ChatLiteLLM, Model Config: {'model': 'openai/gpt-3.5-turbo', 'request_timeout': 60.0, 'model_kwargs': {'frequency_penalty': 0, 'presence_penalty': 0}, 'n': 1, 'max_tokens': 1024}                                                                         
danswer-stack-api_server-1     | 01/05/2024 11:01:25 PM              main.py 245 : Using Embedding model: "thenlper/gte-small"                                                                                                                                                                                                                                 
danswer-stack-api_server-1     | 01/05/2024 11:01:25 PM              main.py 255 : Warming up local NLP models.                                                                                                                                                                                                                                                
danswer-stack-api_server-1     | 01/05/2024 11:01:25 PM search_nlp_models.py  58 : Loading thenlper/gte-small                                                                                                                                                                                                                                                  
danswer-stack-api_server-1     | 2024-01-05 23:01:26.218976: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 93763584 exceeds 10% of free system memory.                                                                                                                                                                                    
danswer-stack-api_server-1     | 2024-01-05 23:01:26.364838: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 93763584 exceeds 10% of free system memory.                                                                                                                                                                                    
danswer-stack-api_server-1     | 2024-01-05 23:01:26.402567: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 93763584 exceeds 10% of free system memory.                                                                                                                                                                                    
danswer-stack-api_server-1     | 2024-01-05 23:01:27.276982: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 93763584 exceeds 10% of free system memory.                                                                                                                                                                                    
danswer-stack-api_server-1     | 2024-01-05 23:01:27.315012: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 93763584 exceeds 10% of free system memory.                                                                                                                                                                                    
danswer-stack-api_server-1     | 01/05/2024 11:01:27 PM              main.py 261 : GPU is not available                                                                                                                                                                                                                                                        
danswer-stack-api_server-1     | 01/05/2024 11:01:27 PM              main.py 262 : Torch Threads: 4                                                                                                                                                                                                                                                            
danswer-stack-api_server-1     | 01/05/2024 11:01:27 PM              main.py 264 : Verifying query preprocessing (NLTK) data is downloaded                                                                                                                                                                                                                     
danswer-stack-api_server-1     | 01/05/2024 11:01:27 PM              main.py 269 : Verifying default connector/credential exist.                                                                                                                                                                                                                               
danswer-stack-api_server-1     | 01/05/2024 11:01:27 PM              main.py 275 : Loading default Prompts and Personas                                                                                                                                                                                                                                        
danswer-stack-api_server-1     | 01/05/2024 11:01:27 PM              main.py 278 : Verifying Document Index(s) is/are available.                                                                                                                                                                                                                               
danswer-stack-api_server-1     | ERROR:    Traceback (most recent call last):                                                                                                                                                                                                                                                                                  
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn                                                                                                                                                                                                                                
danswer-stack-api_server-1     |     conn = connection.create_connection(                                                                                                                                                                                                                                                                                      
danswer-stack-api_server-1     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                      
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 72, in create_connection                                                                                                                                                                                                                    
danswer-stack-api_server-1     |     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):                                                                                                                                                                                                                                                    
danswer-stack-api_server-1     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                     
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/socket.py", line 962, in getaddrinfo                                                                                                                                                                                                                                                        
danswer-stack-api_server-1     |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):                                                                                                                                                                                                                                                   
danswer-stack-api_server-1     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                    
danswer-stack-api_server-1     | socket.gaierror: [Errno -2] Name or service not known                                                                                                                                                                                                                                                                         
danswer-stack-api_server-1     |                                                                                                                                                                                                                                                                                                                               
danswer-stack-api_server-1     | During handling of the above exception, another exception occurred:                                                                                                                                                                                                                                                           
danswer-stack-api_server-1     |                                                                                                                                                                                                                                                                                                                               
danswer-stack-api_server-1     | Traceback (most recent call last):                                                                                                                                                                                                                                                                                            
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen                                                                                                                                                                                                                              
danswer-stack-api_server-1     |     httplib_response = self._make_request(                                                                                                                                                                                                                                                                                    
danswer-stack-api_server-1     |                        ^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                    
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request                                                                                                                                                                                                                        
danswer-stack-api_server-1     |     conn.request(method, url, **httplib_request_kw)                                                                                                                                                                                                                                                                           
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request                                                                                                                                                                                                                                  
danswer-stack-api_server-1     |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)                                                                                                                                                                                                                                              
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/http/client.py", line 1294, in request                                                                                                                                                                                                                                                      
danswer-stack-api_server-1     |     self._send_request(method, url, body, headers, encode_chunked)                                                                                                                                                                                                                                                            
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/http/client.py", line 1340, in _send_request                                                                                                                                                                                                                                                
danswer-stack-api_server-1     |     self.endheaders(body, encode_chunked=encode_chunked)                                                                                                                                                                                                                                                                      
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/http/client.py", line 1289, in endheaders                                                                                                                                                                                                                                                   
danswer-stack-api_server-1     |     self._send_output(message_body, encode_chunked=encode_chunked)                                                                                                                                                                                                                                                            
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/http/client.py", line 1048, in _send_output                                                                                                                                                                                                                                                 
danswer-stack-api_server-1     |     self.send(msg)                                                                                                                                                                                                                                                                                                            
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/http/client.py", line 986, in send                                                                                                                                                                                                                                                          
danswer-stack-api_server-1     |     self.connect()                                                                                                                                                                                                                                                                                                            
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect                                                                                                                                                                                                                                  
danswer-stack-api_server-1     |     conn = self._new_conn()                                                         
danswer-stack-api_server-1     |            ^^^^^^^^^^^^^^^^                                                         
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
danswer-stack-api_server-1     |     raise NewConnectionError(                                                       
danswer-stack-api_server-1     | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f77ebb89290>: Failed to establish a new connection: [Errno -2] Name or service not known
danswer-stack-api_server-1     |                                                                                     
danswer-stack-api_server-1     | During handling of the above exception, another exception occurred:                 
danswer-stack-api_server-1     |                                                                                     
danswer-stack-api_server-1     | Traceback (most recent call last):                                                  
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send                                                                                                                                              
danswer-stack-api_server-1     |     resp = conn.urlopen(                                                                                             
danswer-stack-api_server-1     |            ^^^^^^^^^^^^^                                                            
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 799, in urlopen                                                                                                                                      
danswer-stack-api_server-1     |     retries = retries.increment(                                                                                     
danswer-stack-api_server-1     |               ^^^^^^^^^^^^^^^^^^                                                    
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment                                                                                                                                        
danswer-stack-api_server-1     |     raise MaxRetryError(_pool, url, error or ResponseError(cause))                                                   
danswer-stack-api_server-1     | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='index', port=19071): Max retries exceeded with url: /application/v2/tenant/default/prepareandactivate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77ebb89290>: Failed to establish a new connection: [Errno -2] Name or service
 not known'))                                                                                                                                                                  
danswer-stack-api_server-1     |                                                                                                                                               
danswer-stack-api_server-1     | During handling of the above exception, another exception occurred:                 
danswer-stack-api_server-1     |                                                                                     
danswer-stack-api_server-1     | Traceback (most recent call last):                                                  
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 677, in lifespan                                                                                                                                          
danswer-stack-api_server-1     |     async with self.lifespan_context(app) as maybe_state:                           
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 566, in __aenter__
danswer-stack-api_server-1     |     await self._router.startup()                                                    
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 656, in startup   
danswer-stack-api_server-1     |     handler()                                                                       
danswer-stack-api_server-1     |   File "/app/danswer/main.py", line 279, in startup_event                                                            
danswer-stack-api_server-1     |     get_default_document_index().ensure_indices_exist()                             
danswer-stack-api_server-1     |   File "/app/danswer/document_index/vespa/index.py", line 606, in ensure_indices_exist                                                                                                                                                
danswer-stack-api_server-1     |     response = requests.post(deploy_url, headers=headers, data=f)                   
danswer-stack-api_server-1     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                    
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 115, in post 
danswer-stack-api_server-1     |     return request("post", url, data=data, json=json, **kwargs)                                                                               
danswer-stack-api_server-1     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                               
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request                                                         
danswer-stack-api_server-1     |     return session.request(method=method, url=url, **kwargs)                                                                                  
danswer-stack-api_server-1     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                  
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request                                                   
danswer-stack-api_server-1     |     resp = self.send(prep, **send_kwargs)                                                                                                     
danswer-stack-api_server-1     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                     
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send                                                      
danswer-stack-api_server-1     |     r = adapter.send(request, **kwargs)                                                                                                       
danswer-stack-api_server-1     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                       
danswer-stack-api_server-1     |   File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 519, in send                                                      
danswer-stack-api_server-1     |     raise ConnectionError(e, request=request)                                                                                                 
danswer-stack-api_server-1     | requests.exceptions.ConnectionError: HTTPConnectionPool(host='index', port=19071): Max retries exceeded with url: /application/v2/tenant/default/prepareandactivate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77ebb89290>: Failed to establish a new connection: [Errno -2] Name or serv
ice not known'))                                                                       
danswer-stack-api_server-1     |                                                       
danswer-stack-api_server-1     | ERROR:    Application startup failed. Exiting.  

[yuhongsun96]

@tigermask1978, is there a reason you don't want to allow the container to download models? There are a couple models required to run Danswer. If you really want, you can manually copy over those models into the persistent volume:

This is however more complicated than necessary and I would avoid doing this unless absolutely necessary

[yuhongsun96]

@b0vik your issue is due to the document index container not being accessible. It's called index-1 if you're using docker compose. Can you check if that one is failing to start up (you can check the "Last Started" time to see if it has been restarting)

[redcathode]

@yuhongsun96 Yeah, didn't notice that. Here are the logs for the index container:

maximus@node05:~$ docker logs danswer-stack-index-1
/usr/local/bin/start-container.sh: line 67:    11 Illegal instruction     /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67:    10 Illegal instruction     /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67:    10 Illegal instruction     /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67:    10 Illegal instruction     /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67:     9 Illegal instruction     /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67:    10 Illegal instruction     /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67:     9 Illegal instruction     /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67:    10 Illegal instruction     /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67:    10 Illegal instruction     /opt/vespa/bin/vespa-start-configserver

[yuhongsun96]

@b0vik you're probably running on a CPU without AVX2 instruction support.
Vespa requires AVX2 by default but they do provide a container that does not require it, see here:
https://hub.docker.com/r/vespaengine/vespa-generic-intel-x86_64

If either of you want a faster channel of communications with us we invite you to join our Slack or Discord! Alternatively if you want to have a quick chat with the maintainer (Chris and myself), you can just select a time on my Cal

We'd love to hear more about your use cases and help you get set up for success

[github-actions]

This issue is stale because it has been open 75 days with no activity. Remove stale label or comment or this will be closed in 15 days.