[Bug]connection error of telemetry tracing server in Chatqna backend #1552
Description
Priority
P2-High
OS type
Ubuntu
Hardware type
Gaudi2
Installation method
- Pull docker images from hub.docker.com
- Build docker images from source
- Other
Deploy method
- Docker
- Docker Compose
- Kubernetes Helm Charts
- Kubernetes GMC
- Other
Running nodes
Single Node
What's the version?
chatqna-chatqna-ui-695995789c-sz67q opea/chatqna-ui:1.2
chatqna-data-prep-67f484b58f-xwvct opea/dataprep:1.2
chatqna-db8987c4c-slm6z opea/chatqna-without-rerank:1.2
chatqna-nginx-6d9df4b75b-swts2 opea/nginx:latest
chatqna-redis-vector-db-66c94f7fc5-csfx2 redis/redis-stack:7.2.0-v9
chatqna-retriever-usvc-5b64ff97c8-4fkd9 opea/retriever:1.2
chatqna-tei-7fc4845868-lr2wx ghcr.io/huggingface/text-embeddings-inference:cpu-1.5
chatqna-tgi-f5fc79849-bhrsk ghcr.io/huggingface/tgi-gaudi:2.3.1
chatqna-tgi-f5fc79849-fv48f ghcr.io/huggingface/tgi-gaudi:2.3.1
chatqna-tgi-f5fc79849-jdmwj ghcr.io/huggingface/tgi-gaudi:2.3.1
chatqna-tgi-f5fc79849-jwsxb ghcr.io/huggingface/tgi-gaudi:2.3.1
chatqna-tgi-f5fc79849-nhkdj ghcr.io/huggingface/tgi-gaudi:2.3.1
chatqna-tgi-f5fc79849-q5glp ghcr.io/huggingface/tgi-gaudi:2.3.1
chatqna-tgi-f5fc79849-td5lk ghcr.io/huggingface/tgi-gaudi:2.3.1
chatqna-tgi-f5fc79849-zxtr9 ghcr.io/huggingface/tgi-gaudi:2.3.1
Description
- using helm-chart deploy Chatqna(no-rerank) v1.2 . images from docker hub.
- got error message in chatqna backend(see below).
Seems like it's trying to send the telemetry tracing message to jaeger server
don't know how to disable this feature.
chatqna-5cd9d489f6-h6ktr chatqna INFO: 100.83.122.244:12195 - "POST /v1/chatqna HTTP/1.1" 500 Internal Server Error chatqna-5cd9d489f6-h6ktr chatqna ERROR: Exception in ASGI application chatqna-5cd9d489f6-h6ktr chatqna Traceback (most recent call last): chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn chatqna-5cd9d489f6-h6ktr chatqna sock = connection.create_connection( chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection chatqna-5cd9d489f6-h6ktr chatqna raise err chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection chatqna-5cd9d489f6-h6ktr chatqna sock.connect(sa) chatqna-5cd9d489f6-h6ktr chatqna ConnectionRefusedError: [Errno 111] Connection refused chatqna-5cd9d489f6-h6ktr chatqna chatqna-5cd9d489f6-h6ktr chatqna The above exception was the direct cause of the following exception: chatqna-5cd9d489f6-h6ktr chatqna chatqna-5cd9d489f6-h6ktr chatqna Traceback (most recent call last): chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen chatqna-5cd9d489f6-h6ktr chatqna response = self._make_request( chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 493, in _make_request chatqna-5cd9d489f6-h6ktr chatqna conn.request( chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 445, in request chatqna-5cd9d489f6-h6ktr chatqna self.endheaders() chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/http/client.py", line 1298, in endheaders chatqna-5cd9d489f6-h6ktr chatqna self._send_output(message_body, encode_chunked=encode_chunked) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/http/client.py", line 1058, in _send_output chatqna-5cd9d489f6-h6ktr chatqna self.send(msg) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/http/client.py", line 996, in send chatqna-5cd9d489f6-h6ktr chatqna self.connect() chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 276, in connect chatqna-5cd9d489f6-h6ktr chatqna self.sock = self._new_conn() chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 213, in _new_conn chatqna-5cd9d489f6-h6ktr chatqna raise NewConnectionError( chatqna-5cd9d489f6-h6ktr chatqna urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff4c86b9710>: Failed to establish a new connection: [Errno 111] Connection refused chatqna-5cd9d489f6-h6ktr chatqna chatqna-5cd9d489f6-h6ktr chatqna The above exception was the direct cause of the following exception: chatqna-5cd9d489f6-h6ktr chatqna chatqna-5cd9d489f6-h6ktr chatqna Traceback (most recent call last): chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 667, in send chatqna-5cd9d489f6-h6ktr chatqna resp = conn.urlopen( chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 841, in urlopen chatqna-5cd9d489f6-h6ktr chatqna retries = retries.increment( chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment chatqna-5cd9d489f6-h6ktr chatqna raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='chatqna-tgi', port=80): Max retries exceeded with url: /v1/chat/completions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff4c86b9710>: Failed to establish a new connection: [Errno 111] Connection refused')) chatqna-5cd9d489f6-h6ktr chatqna chatqna-5cd9d489f6-h6ktr chatqna During handling of the above exception, another exception occurred: chatqna-5cd9d489f6-h6ktr chatqna chatqna-5cd9d489f6-h6ktr chatqna Traceback (most recent call last): chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi chatqna-5cd9d489f6-h6ktr chatqna result = await app( # type: ignore[func-returns-value] chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ chatqna-5cd9d489f6-h6ktr chatqna return await self.app(scope, receive, send) chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ chatqna-5cd9d489f6-h6ktr chatqna await super().__call__(scope, receive, send) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__ chatqna-5cd9d489f6-h6ktr chatqna await self.middleware_stack(scope, receive, send) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__ chatqna-5cd9d489f6-h6ktr chatqna raise exc chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__ chatqna-5cd9d489f6-h6ktr chatqna await self.app(scope, receive, _send) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 174, in __call__ chatqna-5cd9d489f6-h6ktr chatqna raise exc chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 172, in __call__ chatqna-5cd9d489f6-h6ktr chatqna await self.app(scope, receive, send_wrapper) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__ chatqna-5cd9d489f6-h6ktr chatqna await self.app(scope, receive, send) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ chatqna-5cd9d489f6-h6ktr chatqna await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app chatqna-5cd9d489f6-h6ktr chatqna raise exc chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app chatqna-5cd9d489f6-h6ktr chatqna await app(scope, receive, sender) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__ chatqna-5cd9d489f6-h6ktr chatqna await self.middleware_stack(scope, receive, send) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 735, in app chatqna-5cd9d489f6-h6ktr chatqna await route.handle(scope, receive, send) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle chatqna-5cd9d489f6-h6ktr chatqna await self.app(scope, receive, send) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 76, in app chatqna-5cd9d489f6-h6ktr chatqna await wrap_app_handling_exceptions(app, request)(scope, receive, send) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app chatqna-5cd9d489f6-h6ktr chatqna raise exc chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app chatqna-5cd9d489f6-h6ktr chatqna await app(scope, receive, sender) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 73, in app chatqna-5cd9d489f6-h6ktr chatqna response = await f(request) chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app chatqna-5cd9d489f6-h6ktr chatqna raw_response = await run_endpoint_function( chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function chatqna-5cd9d489f6-h6ktr chatqna return await dependant.call(**values) chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/home/user/chatqna.py", line 359, in handle_request chatqna-5cd9d489f6-h6ktr chatqna result_dict, runtime_graph = await self.megaservice.schedule( chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/home/user/GenAIComps/comps/cores/telemetry/opea_telemetry.py", line 51, in wrapper chatqna-5cd9d489f6-h6ktr chatqna res = await func(*args, **kwargs) chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 109, in schedule chatqna-5cd9d489f6-h6ktr chatqna response, node = await done_task chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/home/user/GenAIComps/comps/cores/telemetry/opea_telemetry.py", line 51, in wrapper chatqna-5cd9d489f6-h6ktr chatqna res = await func(*args, **kwargs) chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 220, in execute chatqna-5cd9d489f6-h6ktr chatqna response = requests.post( chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 115, in post chatqna-5cd9d489f6-h6ktr chatqna return request("post", url, data=data, json=json, **kwargs) chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request chatqna-5cd9d489f6-h6ktr chatqna return session.request(method=method, url=url, **kwargs) chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request chatqna-5cd9d489f6-h6ktr chatqna resp = self.send(prep, **send_kwargs) chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send chatqna-5cd9d489f6-h6ktr chatqna r = adapter.send(request, **kwargs) chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 700, in send chatqna-5cd9d489f6-h6ktr chatqna raise ConnectionError(e, request=request) chatqna-5cd9d489f6-h6ktr chatqna requests.exceptions.ConnectionError: HTTPConnectionPool(host='chatqna-tgi', port=80): Max retries exceeded with url: /v1/chat/completions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff4c86b9710>: Failed to establish a new connection: [Errno 111] Connection refused')) chatqna-5cd9d489f6-h6ktr chatqna Exception while exporting Span batch. chatqna-5cd9d489f6-h6ktr chatqna Traceback (most recent call last): chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn chatqna-5cd9d489f6-h6ktr chatqna sock = connection.create_connection( chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection chatqna-5cd9d489f6-h6ktr chatqna raise err chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection chatqna-5cd9d489f6-h6ktr chatqna sock.connect(sa) chatqna-5cd9d489f6-h6ktr chatqna ConnectionRefusedError: [Errno 111] Connection refused chatqna-5cd9d489f6-h6ktr chatqna chatqna-5cd9d489f6-h6ktr chatqna The above exception was the direct cause of the following exception: chatqna-5cd9d489f6-h6ktr chatqna chatqna-5cd9d489f6-h6ktr chatqna Traceback (most recent call last): chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen chatqna-5cd9d489f6-h6ktr chatqna response = self._make_request( chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 493, in _make_request chatqna-5cd9d489f6-h6ktr chatqna conn.request( chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 445, in request chatqna-5cd9d489f6-h6ktr chatqna self.endheaders() chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/http/client.py", line 1298, in endheaders chatqna-5cd9d489f6-h6ktr chatqna self._send_output(message_body, encode_chunked=encode_chunked) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/http/client.py", line 1058, in _send_output chatqna-5cd9d489f6-h6ktr chatqna self.send(msg) chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/http/client.py", line 996, in send chatqna-5cd9d489f6-h6ktr chatqna self.connect() chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 276, in connect chatqna-5cd9d489f6-h6ktr chatqna self.sock = self._new_conn() chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 213, in _new_conn chatqna-5cd9d489f6-h6ktr chatqna raise NewConnectionError( chatqna-5cd9d489f6-h6ktr chatqna urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff4ca08e390>: Failed to establish a new connection: [Errno 111] Connection refused chatqna-5cd9d489f6-h6ktr chatqna chatqna-5cd9d489f6-h6ktr chatqna The above exception was the direct cause of the following exception: chatqna-5cd9d489f6-h6ktr chatqna chatqna-5cd9d489f6-h6ktr chatqna Traceback (most recent call last): chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 667, in send chatqna-5cd9d489f6-h6ktr chatqna resp = conn.urlopen( chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 841, in urlopen chatqna-5cd9d489f6-h6ktr chatqna retries = retries.increment( chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment chatqna-5cd9d489f6-h6ktr chatqna raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff4ca08e390>: Failed to establish a new connection: [Errno 111] Connection refused')) chatqna-5cd9d489f6-h6ktr chatqna chatqna-5cd9d489f6-h6ktr chatqna During handling of the above exception, another exception occurred: chatqna-5cd9d489f6-h6ktr chatqna chatqna-5cd9d489f6-h6ktr chatqna Traceback (most recent call last): chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/opentelemetry/sdk/trace/export/__init__.py", line 360, in _export_batch chatqna-5cd9d489f6-h6ktr chatqna self.span_exporter.export(self.spans_list[:idx]) # type: ignore chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 189, in export chatqna-5cd9d489f6-h6ktr chatqna return self._export_serialized_spans(serialized_data) chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 159, in _export_serialized_spans chatqna-5cd9d489f6-h6ktr chatqna resp = self._export(serialized_data) chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 133, in _export chatqna-5cd9d489f6-h6ktr chatqna return self._session.post( chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 637, in post chatqna-5cd9d489f6-h6ktr chatqna return self.request("POST", url, data=data, json=json, **kwargs) chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request chatqna-5cd9d489f6-h6ktr chatqna resp = self.send(prep, **send_kwargs) chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send chatqna-5cd9d489f6-h6ktr chatqna r = adapter.send(request, **kwargs) chatqna-5cd9d489f6-h6ktr chatqna ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ chatqna-5cd9d489f6-h6ktr chatqna File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 700, in send chatqna-5cd9d489f6-h6ktr chatqna raise ConnectionError(e, request=request) chatqna-5cd9d489f6-h6ktr chatqna requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff4ca08e390>: Failed to establish a new connection: [Errno 111] Connection refused'))
Reproduce steps
- deploy the chatqna v1.2 by using helm-chart
- send request by using benchmarking scripts
- see the log in chatqna backend
Raw log
Attachments
No response