core.model_runtime.errors.invoke.InvokeError: [huggingface_tei] Error: timed out

[gree2]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

I use nginx to proxy 8 bge embedding model.
Those 8 bge models running inside 8 ghcr.io/huggingface/text-embeddings-inference:cpu-1.5 containers.
When uploading files to Knowledge, some text file may run into status Error with message [huggingface_tei] Error: timed out.

dify_070_worker logs output core.model_runtime.errors.invoke.InvokeError: [huggingface_tei] Error: timed out

bge_emb_6100 logs output 192.168.1.63 - - [20/Aug/2024:15:12:03 +0800] "POST /v1/embeddings HTTP/1.1" 499 0 "-" "python-httpx/0.27.0"

2. Additional context or comments

docker logs -f --tail 20 dify_070_worker
... omit some lines

[2024-08-20 07:11:46,756: INFO/MainProcess] Start retry document: d54613d9-514c-4eb3-8221-09b00681d802
[2024-08-20 07:11:47,268: INFO/MainProcess] HTTP Request: POST http://192.168.1.62:6100/tokenize "HTTP/1.1 200 OK"
**... omit 509 lines**
[2024-08-20 07:11:58,221: INFO/MainProcess] HTTP Request: POST http://192.168.1.62:6100/tokenize "HTTP/1.1 200 OK"
[2024-08-20 07:12:03,080: WARNING/MainProcess] --- Logging error ---
[2024-08-20 07:12:03,080: WARNING/MainProcess] Traceback (most recent call last):
[2024-08-20 07:12:03,080: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
[2024-08-20 07:12:03,080: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 233, in handle_request
    resp = self._pool.handle_request(req)
[2024-08-20 07:12:03,080: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    return self._connection.handle_request(request)
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 143, in handle_request
    raise exc
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 113, in handle_request
    ) = self._receive_response_headers(**kwargs)
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 186, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 224, in _receive_event
    data = self._network_stream.read(
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 124, in read
    with map_exceptions(exc_map):
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
[2024-08-20 07:12:03,081: WARNING/MainProcess] httpcore.ReadTimeout: timed out
[2024-08-20 07:12:03,081: WARNING/MainProcess]
The above exception was the direct cause of the following exception:
[2024-08-20 07:12:03,081: WARNING/MainProcess] Traceback (most recent call last):
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 36, in invoke
    return self._invoke(model, credentials, texts, user)
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/huggingface_tei/text_embedding/text_embedding.py", line 94, in _invoke
    results = TeiHelper.invoke_embeddings(server_url, iter_texts)
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/huggingface_tei/tei_helper.py", line 151, in invoke_embeddings
    resp = httpx.post(
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_api.py", line 319, in post
    return request(
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_api.py", line 106, in request
    return client.request(
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
[2024-08-20 07:12:03,081: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
[2024-08-20 07:12:03,082: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
[2024-08-20 07:12:03,082: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
[2024-08-20 07:12:03,082: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 232, in handle_request
    with map_httpcore_exceptions():
[2024-08-20 07:12:03,082: WARNING/MainProcess]   File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
[2024-08-20 07:12:03,082: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
[2024-08-20 07:12:03,082: WARNING/MainProcess] httpx.ReadTimeout: timed out
[2024-08-20 07:12:03,082: WARNING/MainProcess]
During handling of the above exception, another exception occurred:
[2024-08-20 07:12:03,082: WARNING/MainProcess] Traceback (most recent call last):
[2024-08-20 07:12:03,082: WARNING/MainProcess]   File "/app/api/core/embedding/cached_embedding.py", line 51, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
[2024-08-20 07:12:03,082: WARNING/MainProcess]   File "/app/api/core/model_manager.py", line 170, in invoke_text_embedding
    return self._round_robin_invoke(
[2024-08-20 07:12:03,082: WARNING/MainProcess]   File "/app/api/core/model_manager.py", line 302, in _round_robin_invoke
    return function(*args, **kwargs)
[2024-08-20 07:12:03,082: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 38, in invoke
    raise self._transform_invoke_error(e)
[2024-08-20 07:12:03,082: WARNING/MainProcess] core.model_runtime.errors.invoke.InvokeError: [huggingface_tei] Error: timed out
[2024-08-20 07:12:03,082: WARNING/MainProcess]
During handling of the above exception, another exception occurred:
[2024-08-20 07:12:03,082: WARNING/MainProcess] Traceback (most recent call last):
[2024-08-20 07:12:03,082: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 1100, in emit
    msg = self.format(record)
[2024-08-20 07:12:03,082: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 943, in format
    return fmt.format(record)
[2024-08-20 07:12:03,082: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/celery/utils/log.py", line 146, in format
    msg = super().format(record)
[2024-08-20 07:12:03,083: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 678, in format
    record.message = record.getMessage()
[2024-08-20 07:12:03,083: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 368, in getMessage
    msg = msg % self.args
[2024-08-20 07:12:03,083: WARNING/MainProcess] TypeError: not all arguments converted during string formatting
[2024-08-20 07:12:03,083: WARNING/MainProcess] Call stack:
[2024-08-20 07:12:03,083: WARNING/MainProcess]   File "/usr/local/lib/python3.10/threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
[2024-08-20 07:12:03,083: WARNING/MainProcess]   File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
[2024-08-20 07:12:03,083: WARNING/MainProcess]   File "/usr/local/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
[2024-08-20 07:12:03,083: WARNING/MainProcess]   File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker
    work_item.run()
[2024-08-20 07:12:03,083: WARNING/MainProcess]   File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
[2024-08-20 07:12:03,083: WARNING/MainProcess]   File "/app/api/core/indexing_runner.py", line 743, in _process_chunk
    index_processor.load(dataset, chunk_documents, with_keywords=False)
[2024-08-20 07:12:03,083: WARNING/MainProcess]   File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 60, in load
    vector.create(documents)
[2024-08-20 07:12:03,083: WARNING/MainProcess]   File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 100, in create
    embeddings = self._embeddings.embed_documents([document.page_content for document in texts])
[2024-08-20 07:12:03,083: WARNING/MainProcess]   File "/app/api/core/embedding/cached_embedding.py", line 81, in embed_documents
    logger.error('Failed to embed documents: ', ex)
[2024-08-20 07:12:03,083: WARNING/MainProcess] Message: 'Failed to embed documents: '
Arguments: (InvokeError(),)
[2024-08-20 07:12:03,274: WARNING/MainProcess] --- Logging error ---
[2024-08-20 07:12:03,275: WARNING/MainProcess] Traceback (most recent call last):
[2024-08-20 07:12:03,275: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
[2024-08-20 07:12:03,276: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 233, in handle_request
    resp = self._pool.handle_request(req)
[2024-08-20 07:12:03,276: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
[2024-08-20 07:12:03,276: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
[2024-08-20 07:12:03,276: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    return self._connection.handle_request(request)
[2024-08-20 07:12:03,276: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 143, in handle_request
    raise exc
[2024-08-20 07:12:03,276: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 113, in handle_request
    ) = self._receive_response_headers(**kwargs)
[2024-08-20 07:12:03,276: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 186, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
[2024-08-20 07:12:03,277: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 224, in _receive_event
    data = self._network_stream.read(
[2024-08-20 07:12:03,277: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 124, in read
    with map_exceptions(exc_map):
[2024-08-20 07:12:03,277: WARNING/MainProcess]   File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
[2024-08-20 07:12:03,277: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
[2024-08-20 07:12:03,277: WARNING/MainProcess] httpcore.ReadTimeout: timed out
[2024-08-20 07:12:03,277: WARNING/MainProcess]
The above exception was the direct cause of the following exception:
[2024-08-20 07:12:03,277: WARNING/MainProcess] Traceback (most recent call last):
[2024-08-20 07:12:03,277: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 36, in invoke
    return self._invoke(model, credentials, texts, user)
[2024-08-20 07:12:03,277: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/huggingface_tei/text_embedding/text_embedding.py", line 94, in _invoke
    results = TeiHelper.invoke_embeddings(server_url, iter_texts)
[2024-08-20 07:12:03,277: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/huggingface_tei/tei_helper.py", line 151, in invoke_embeddings
    resp = httpx.post(
[2024-08-20 07:12:03,277: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_api.py", line 319, in post
    return request(
[2024-08-20 07:12:03,277: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_api.py", line 106, in request
    return client.request(
[2024-08-20 07:12:03,278: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
[2024-08-20 07:12:03,278: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
[2024-08-20 07:12:03,278: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
[2024-08-20 07:12:03,278: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
[2024-08-20 07:12:03,278: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
[2024-08-20 07:12:03,278: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 232, in handle_request
    with map_httpcore_exceptions():
[2024-08-20 07:12:03,278: WARNING/MainProcess]   File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
[2024-08-20 07:12:03,278: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
[2024-08-20 07:12:03,278: WARNING/MainProcess] httpx.ReadTimeout: timed out
[2024-08-20 07:12:03,278: WARNING/MainProcess]
During handling of the above exception, another exception occurred:
[2024-08-20 07:12:03,278: WARNING/MainProcess] Traceback (most recent call last):
[2024-08-20 07:12:03,278: WARNING/MainProcess]   File "/app/api/core/embedding/cached_embedding.py", line 51, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
[2024-08-20 07:12:03,278: WARNING/MainProcess]   File "/app/api/core/model_manager.py", line 170, in invoke_text_embedding
    return self._round_robin_invoke(
[2024-08-20 07:12:03,278: WARNING/MainProcess]   File "/app/api/core/model_manager.py", line 302, in _round_robin_invoke
    return function(*args, **kwargs)
[2024-08-20 07:12:03,279: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 38, in invoke
    raise self._transform_invoke_error(e)
[2024-08-20 07:12:03,279: WARNING/MainProcess] core.model_runtime.errors.invoke.InvokeError: [huggingface_tei] Error: timed out
[2024-08-20 07:12:03,279: WARNING/MainProcess]
During handling of the above exception, another exception occurred:
[2024-08-20 07:12:03,279: WARNING/MainProcess] Traceback (most recent call last):
[2024-08-20 07:12:03,279: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 1100, in emit
    msg = self.format(record)
[2024-08-20 07:12:03,279: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 943, in format
    return fmt.format(record)
[2024-08-20 07:12:03,279: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/celery/utils/log.py", line 146, in format
    msg = super().format(record)
[2024-08-20 07:12:03,279: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 678, in format
    record.message = record.getMessage()
[2024-08-20 07:12:03,279: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 368, in getMessage
    msg = msg % self.args
[2024-08-20 07:12:03,279: WARNING/MainProcess] TypeError: not all arguments converted during string formatting
[2024-08-20 07:12:03,279: WARNING/MainProcess] Call stack:
[2024-08-20 07:12:03,280: WARNING/MainProcess]   File "/usr/local/lib/python3.10/threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
[2024-08-20 07:12:03,280: WARNING/MainProcess]   File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
[2024-08-20 07:12:03,280: WARNING/MainProcess]   File "/usr/local/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
[2024-08-20 07:12:03,280: WARNING/MainProcess]   File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker
    work_item.run()
[2024-08-20 07:12:03,280: WARNING/MainProcess]   File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
[2024-08-20 07:12:03,280: WARNING/MainProcess]   File "/app/api/core/indexing_runner.py", line 743, in _process_chunk
    index_processor.load(dataset, chunk_documents, with_keywords=False)
[2024-08-20 07:12:03,280: WARNING/MainProcess]   File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 60, in load
    vector.create(documents)
[2024-08-20 07:12:03,280: WARNING/MainProcess]   File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 100, in create
    embeddings = self._embeddings.embed_documents([document.page_content for document in texts])
[2024-08-20 07:12:03,280: WARNING/MainProcess]   File "/app/api/core/embedding/cached_embedding.py", line 81, in embed_documents
    logger.error('Failed to embed documents: ', ex)
[2024-08-20 07:12:03,280: WARNING/MainProcess] Message: 'Failed to embed documents: '
Arguments: (InvokeError(),)
[2024-08-20 07:12:03,285: WARNING/MainProcess] --- Logging error ---
[2024-08-20 07:12:03,286: WARNING/MainProcess] Traceback (most recent call last):
[2024-08-20 07:12:03,286: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
[2024-08-20 07:12:03,286: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 233, in handle_request
    resp = self._pool.handle_request(req)
[2024-08-20 07:12:03,286: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
[2024-08-20 07:12:03,286: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
[2024-08-20 07:12:03,286: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    return self._connection.handle_request(request)
[2024-08-20 07:12:03,286: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 143, in handle_request
    raise exc
[2024-08-20 07:12:03,286: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 113, in handle_request
    ) = self._receive_response_headers(**kwargs)
[2024-08-20 07:12:03,286: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 186, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
[2024-08-20 07:12:03,287: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 224, in _receive_event
    data = self._network_stream.read(
[2024-08-20 07:12:03,287: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 124, in read
    with map_exceptions(exc_map):
[2024-08-20 07:12:03,287: WARNING/MainProcess]   File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
[2024-08-20 07:12:03,287: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
[2024-08-20 07:12:03,287: WARNING/MainProcess] httpcore.ReadTimeout: timed out
[2024-08-20 07:12:03,287: WARNING/MainProcess]
The above exception was the direct cause of the following exception:
[2024-08-20 07:12:03,287: WARNING/MainProcess] Traceback (most recent call last):
[2024-08-20 07:12:03,287: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 36, in invoke
    return self._invoke(model, credentials, texts, user)
[2024-08-20 07:12:03,287: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/huggingface_tei/text_embedding/text_embedding.py", line 94, in _invoke
    results = TeiHelper.invoke_embeddings(server_url, iter_texts)
[2024-08-20 07:12:03,287: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/huggingface_tei/tei_helper.py", line 151, in invoke_embeddings
    resp = httpx.post(
[2024-08-20 07:12:03,287: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_api.py", line 319, in post
    return request(
[2024-08-20 07:12:03,287: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_api.py", line 106, in request
    return client.request(
[2024-08-20 07:12:03,287: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
[2024-08-20 07:12:03,287: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
[2024-08-20 07:12:03,288: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
[2024-08-20 07:12:03,288: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
[2024-08-20 07:12:03,288: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
[2024-08-20 07:12:03,288: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 232, in handle_request
    with map_httpcore_exceptions():
[2024-08-20 07:12:03,288: WARNING/MainProcess]   File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
[2024-08-20 07:12:03,288: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
[2024-08-20 07:12:03,288: WARNING/MainProcess] httpx.ReadTimeout: timed out
[2024-08-20 07:12:03,288: WARNING/MainProcess]
During handling of the above exception, another exception occurred:
[2024-08-20 07:12:03,288: WARNING/MainProcess] Traceback (most recent call last):
[2024-08-20 07:12:03,288: WARNING/MainProcess]   File "/app/api/core/embedding/cached_embedding.py", line 51, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
[2024-08-20 07:12:03,288: WARNING/MainProcess]   File "/app/api/core/model_manager.py", line 170, in invoke_text_embedding
    return self._round_robin_invoke(
[2024-08-20 07:12:03,288: WARNING/MainProcess]   File "/app/api/core/model_manager.py", line 302, in _round_robin_invoke
    return function(*args, **kwargs)
[2024-08-20 07:12:03,288: WARNING/MainProcess]   File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 38, in invoke
    raise self._transform_invoke_error(e)
[2024-08-20 07:12:03,289: WARNING/MainProcess] core.model_runtime.errors.invoke.InvokeError: [huggingface_tei] Error: timed out
[2024-08-20 07:12:03,289: WARNING/MainProcess]
During handling of the above exception, another exception occurred:
[2024-08-20 07:12:03,289: WARNING/MainProcess] Traceback (most recent call last):
[2024-08-20 07:12:03,289: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 1100, in emit
    msg = self.format(record)
[2024-08-20 07:12:03,289: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 943, in format
    return fmt.format(record)
[2024-08-20 07:12:03,289: WARNING/MainProcess]   File "/app/api/.venv/lib/python3.10/site-packages/celery/utils/log.py", line 146, in format
    msg = super().format(record)
[2024-08-20 07:12:03,289: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 678, in format
    record.message = record.getMessage()
[2024-08-20 07:12:03,289: WARNING/MainProcess]   File "/usr/local/lib/python3.10/logging/__init__.py", line 368, in getMessage
    msg = msg % self.args
[2024-08-20 07:12:03,289: WARNING/MainProcess] TypeError: not all arguments converted during string formatting
[2024-08-20 07:12:03,289: WARNING/MainProcess] Call stack:
[2024-08-20 07:12:03,290: WARNING/MainProcess]   File "/usr/local/lib/python3.10/threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
[2024-08-20 07:12:03,290: WARNING/MainProcess]   File "/usr/local/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
[2024-08-20 07:12:03,290: WARNING/MainProcess]   File "/usr/local/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
[2024-08-20 07:12:03,290: WARNING/MainProcess]   File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker
    work_item.run()
[2024-08-20 07:12:03,290: WARNING/MainProcess]   File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
[2024-08-20 07:12:03,290: WARNING/MainProcess]   File "/app/api/core/indexing_runner.py", line 743, in _process_chunk
    index_processor.load(dataset, chunk_documents, with_keywords=False)
[2024-08-20 07:12:03,290: WARNING/MainProcess]   File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 60, in load
    vector.create(documents)
[2024-08-20 07:12:03,290: WARNING/MainProcess]   File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 100, in create
    embeddings = self._embeddings.embed_documents([document.page_content for document in texts])
[2024-08-20 07:12:03,290: WARNING/MainProcess]   File "/app/api/core/embedding/cached_embedding.py", line 81, in embed_documents
    logger.error('Failed to embed documents: ', ex)
[2024-08-20 07:12:03,290: WARNING/MainProcess] Message: 'Failed to embed documents: '
Arguments: (InvokeError(),)
[2024-08-20 07:12:03,291: ERROR/MainProcess] consume document failed
Traceback (most recent call last):
  File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
  File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 233, in handle_request
    resp = self._pool.handle_request(req)
  File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
    raise exc from None
  File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
  File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    return self._connection.handle_request(request)
  File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 143, in handle_request
    raise exc
  File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 113, in handle_request
    ) = self._receive_response_headers(**kwargs)
  File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 186, in _receive_response_headers
    event = self._receive_event(timeout=timeout)
  File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 224, in _receive_event
    data = self._network_stream.read(
  File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 124, in read
    with map_exceptions(exc_map):
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/app/api/.venv/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 36, in invoke
    return self._invoke(model, credentials, texts, user)
  File "/app/api/core/model_runtime/model_providers/huggingface_tei/text_embedding/text_embedding.py", line 94, in _invoke
    results = TeiHelper.invoke_embeddings(server_url, iter_texts)
  File "/app/api/core/model_runtime/model_providers/huggingface_tei/tei_helper.py", line 151, in invoke_embeddings
    resp = httpx.post(
  File "/app/api/.venv/lib/python3.10/site-packages/httpx/_api.py", line 319, in post
    return request(
  File "/app/api/.venv/lib/python3.10/site-packages/httpx/_api.py", line 106, in request
    return client.request(
  File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 827, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
  File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/app/api/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1015, in _send_single_request
    response = transport.handle_request(request)
  File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 232, in handle_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/app/api/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/core/indexing_runner.py", line 77, in run
    self._load(
  File "/app/api/core/indexing_runner.py", line 690, in _load
    tokens += future.result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/app/api/core/indexing_runner.py", line 743, in _process_chunk
    index_processor.load(dataset, chunk_documents, with_keywords=False)
  File "/app/api/core/rag/index_processor/processor/paragraph_index_processor.py", line 60, in load
    vector.create(documents)
  File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 100, in create
    embeddings = self._embeddings.embed_documents([document.page_content for document in texts])
  File "/app/api/core/embedding/cached_embedding.py", line 82, in embed_documents
    raise ex
  File "/app/api/core/embedding/cached_embedding.py", line 51, in embed_documents
    embedding_result = self._model_instance.invoke_text_embedding(
  File "/app/api/core/model_manager.py", line 170, in invoke_text_embedding
    return self._round_robin_invoke(
  File "/app/api/core/model_manager.py", line 302, in _round_robin_invoke
    return function(*args, **kwargs)
  File "/app/api/core/model_runtime/model_providers/__base/text_embedding_model.py", line 38, in invoke
    raise self._transform_invoke_error(e)
core.model_runtime.errors.invoke.InvokeError: [huggingface_tei] Error: timed out

docker logs -f --tail 20 bge_emb_6100
... omit some lines

192.168.1.63 - - [20/Aug/2024:15:11:57 +0800] "POST /tokenize HTTP/1.1" 200 31162 "-" "python-httpx/0.27.0"
192.168.1.63 - - [20/Aug/2024:15:11:57 +0800] "POST /tokenize HTTP/1.1" 200 30117 "-" "python-httpx/0.27.0"
2024/08/20 15:11:58 [warn] 33#33: *86085 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000468, client: 192.168.1.63, server: , request: "POST /tokenize HTTP/1.1", host: "192.168.1.62:6100"
192.168.1.63 - - [20/Aug/2024:15:11:58 +0800] "POST /tokenize HTTP/1.1" 200 23532 "-" "python-httpx/0.27.0"
192.168.1.63 - - [20/Aug/2024:15:11:58 +0800] "POST /tokenize HTTP/1.1" 200 30687 "-" "python-httpx/0.27.0"
192.168.1.63 - - [20/Aug/2024:15:11:58 +0800] "POST /tokenize HTTP/1.1" 200 135214 "-" "python-httpx/0.27.0"
2024/08/20 15:11:58 [warn] 33#33: *86095 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000469, client: 192.168.1.63, server: , request: "POST /v1/embeddings HTTP/1.1", host: "192.168.1.62:6100"
192.168.1.63 - - [20/Aug/2024:15:11:58 +0800] "POST /tokenize HTTP/1.1" 200 26153 "-" "python-httpx/0.27.0"
192.168.1.63 - - [20/Aug/2024:15:11:58 +0800] "POST /tokenize HTTP/1.1" 200 26550 "-" "python-httpx/0.27.0"
192.168.1.63 - - [20/Aug/2024:15:11:58 +0800] "POST /tokenize HTTP/1.1" 200 28731 "-" "python-httpx/0.27.0"
192.168.1.63 - - [20/Aug/2024:15:11:58 +0800] "POST /tokenize HTTP/1.1" 200 28842 "-" "python-httpx/0.27.0"
2024/08/20 15:11:58 [warn] 31#31: *86102 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000470, client: 192.168.1.63, server: , request: "POST /tokenize HTTP/1.1", host: "192.168.1.62:6100"
2024/08/20 15:11:58 [warn] 33#33: *86101 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000471, client: 192.168.1.63, server: , request: "POST /tokenize HTTP/1.1", host: "192.168.1.62:6100"
192.168.1.63 - - [20/Aug/2024:15:11:58 +0800] "POST /tokenize HTTP/1.1" 200 241946 "-" "python-httpx/0.27.0"
192.168.1.63 - - [20/Aug/2024:15:11:58 +0800] "POST /tokenize HTTP/1.1" 200 280758 "-" "python-httpx/0.27.0"
2024/08/20 15:11:58 [warn] 31#31: *86106 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000472, client: 192.168.1.63, server: , request: "POST /v1/embeddings HTTP/1.1", host: "192.168.1.62:6100"
2024/08/20 15:11:58 [warn] 33#33: *86105 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000473, client: 192.168.1.63, server: , request: "POST /v1/embeddings HTTP/1.1", host: "192.168.1.62:6100"
192.168.1.63 - - [20/Aug/2024:15:12:03 +0800] "POST /v1/embeddings HTTP/1.1" 499 0 "-" "python-httpx/0.27.0"
192.168.1.63 - - [20/Aug/2024:15:12:03 +0800] "POST /v1/embeddings HTTP/1.1" 499 0 "-" "python-httpx/0.27.0"
192.168.1.63 - - [20/Aug/2024:15:12:03 +0800] "POST /v1/embeddings HTTP/1.1" 499 0 "-" "python-httpx/0.27.0"

nginx config

user  nginx;
worker_processes  4;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {

    upstream apps {
        # least_conn;
        # random;
        server 192.168.1.62:6101;
        server 192.168.1.62:6102;
        server 192.168.1.62:6103;
        server 192.168.1.62:6104;
        server 192.168.1.62:6105;
        server 192.168.1.62:6106;
        server 192.168.1.62:6107;
        server 192.168.1.62:6108;
    }

    # include       /etc/nginx/mime.types;
    # default_type  application/octet-stream;

    # log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                   '$status $body_bytes_sent "$http_referer" '
    #                   '"$http_user_agent" "$http_x_forwarded_for"';

    # access_log  /var/log/nginx/access.log  main;

    # sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout 120;
    proxy_buffering on;
    proxy_buffer_size  128k;
    proxy_buffers 100  128k;
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;

    server {
        listen 80;

        location / {
            proxy_pass http://apps/;

        }
    }

}

[crazywoola]

Please check you can access this embedding endpoint in the container. :)

[crazywoola]

#10929 (comment)