Failed request (500): Embedding Request Error

[pmarini-nc]

Hello,

I get the following error when asking a question in Context Chat:

docker logs -f nc_app_context_chat_backend

2025-10-02T14:17:41+0000: [ERROR|network_em]: Unexpected error while getting embeddings
Traceback (most recent call last):
  File "/app/context_chat_backend/network_em.py", line 51, in _get_embedding
    response = client.post(
               ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1144, in post
    return self.request(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 812, in request
    request = self.build_request(
              ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 378, in build_request
    return Request(
           ^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_models.py", line 408, in __init__
    headers, stream = encode_request(
                      ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_content.py", line 216, in encode_request
    return encode_json(json)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_content.py", line 179, in encode_json
    ).encode("utf-8")
      ^^^^^^^^^^^^^^^
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 9130-9131: surrogates not allowed
2025-10-02T14:17:41+0000: [ERROR|pgvector]: Error adding documents to vectordb
Traceback (most recent call last):
  File "/app/context_chat_backend/network_em.py", line 51, in _get_embedding
    response = client.post(
               ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1144, in post
    return self.request(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 812, in request
    request = self.build_request(
              ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 378, in build_request
    return Request(
           ^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_models.py", line 408, in __init__
    headers, stream = encode_request(
                      ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_content.py", line 216, in encode_request
    return encode_json(json)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_content.py", line 179, in encode_json
    ).encode("utf-8")
      ^^^^^^^^^^^^^^^
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 9130-9131: surrogates not allowed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/context_chat_backend/vectordb/pgvector.py", line 149, in add_indocuments
    chunk_ids.extend(self.client.add_documents(indoc.documents[i:i+batch_size]))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/langchain_core/vectorstores/base.py", line 288, in add_documents
    return self.add_texts(texts, metadatas, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/langchain_postgres/vectorstores.py", line 885, in add_texts
    embeddings = self.embedding_function.embed_documents(texts_)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/network_em.py", line 99, in embed_documents
    return self._get_embedding(texts)  # pyright: ignore[reportReturnType]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/network_em.py", line 88, in _get_embedding
    raise EmbeddingException('Error: unexpected error while getting embeddings') from e
context_chat_backend.types.EmbeddingException: Error: unexpected error while getting embeddings
2025-10-02T14:17:41+0000: [ERROR|utils]: original traceback: Traceback (most recent call last):
  File "/app/context_chat_backend/network_em.py", line 51, in _get_embedding
    response = client.post(
               ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1144, in post
    return self.request(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 812, in request
    request = self.build_request(
              ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 378, in build_request
    return Request(
           ^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_models.py", line 408, in __init__
    headers, stream = encode_request(
                      ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_content.py", line 216, in encode_request
    return encode_json(json)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_content.py", line 179, in encode_json
    ).encode("utf-8")
      ^^^^^^^^^^^^^^^
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 9130-9131: surrogates not allowed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/context_chat_backend/utils.py", line 74, in exception_wrap
    resconn.send({ 'value': fun(*args, **kwargs), 'error': None })
                            ^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/chain/ingest/injest.py", line 192, in embed_sources
    return _process_sources(vectordb, config, sources_filtered)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/chain/ingest/injest.py", line 157, in _process_sources
    added_source_ids, retry_source_ids = vectordb.add_indocuments(indocuments)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/vectordb/pgvector.py", line 149, in add_indocuments
    chunk_ids.extend(self.client.add_documents(indoc.documents[i:i+batch_size]))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/langchain_core/vectorstores/base.py", line 288, in add_documents
    return self.add_texts(texts, metadatas, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/langchain_postgres/vectorstores.py", line 885, in add_texts
    embeddings = self.embedding_function.embed_documents(texts_)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/network_em.py", line 99, in embed_documents
    return self._get_embedding(texts)  # pyright: ignore[reportReturnType]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/network_em.py", line 88, in _get_embedding
    raise EmbeddingException('Error: unexpected error while getting embeddings') from e
context_chat_backend.types.EmbeddingException: Error: unexpected error while getting embeddings

2025-10-02T14:17:41+0000: [ERROR|controller]: Error occurred in an embedding request: /loadSources:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/dist-packages/starlette/routing.py", line 75, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 302, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/fastapi/routing.py", line 215, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/starlette/concurrency.py", line 38, in run_in_threadpool
    return await anyio.to_thread.run_sync(func)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/anyio/_backends/_asyncio.py", line 2476, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/anyio/_backends/_asyncio.py", line 967, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/controller.py", line 191, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/context_chat_backend/controller.py", line 384, in _
    loaded_sources, not_added_sources = exec_in_proc(
                                        ^^^^^^^^^^^^^
  File "/app/context_chat_backend/utils.py", line 97, in exec_in_proc
    raise result['error']
context_chat_backend.types.EmbeddingException: Error: unexpected error while getting embeddings
2025-10-02T14:17:41+0000: [ERROR|utils]: Failed request (500): Embedding Request Error: Error: unexpected error while getting embeddings
INFO:     10.0.0.241:0 - "PUT /loadSources HTTP/1.1" 500 Internal Server Error

cat nextcloud.log

{
  "reqId": "qOg6IWbpvn6KUkeYrmb8",
  "level": 3,
  "time": "2025-10-02T16:21:32+02:00",
  "remoteAddr": "",
  "user": "--",
  "app": "no app in context",
  "method": "",
  "url": "--",
  "message": "Unknown error while processing TaskProcessing task",
  "userAgent": "--",
  "version": "32.0.0.13",
  "exception": {
    "Exception": "RuntimeException",
    "Message": "Error received from Context Chat Backend (ExApp) with status code 504: unknown error",
    "Code": 0,
    "Trace": [
      {
        "file": "/var/www/nextcloud/apps/context_chat/lib/Service/LangRopeService.php",
        "line": 335,
        "function": "requestToExApp",
        "class": "OCA\\ContextChat\\Service\\LangRopeService",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/apps/context_chat/lib/TaskProcessing/ContextChatProvider.php",
        "line": 121,
        "function": "query",
        "class": "OCA\\ContextChat\\Service\\LangRopeService",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/TaskProcessing/Manager.php",
        "line": 1016,
        "function": "process",
        "class": "OCA\\ContextChat\\TaskProcessing\\ContextChatProvider",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/private/TaskProcessing/SynchronousBackgroundJob.php",
        "line": 54,
        "function": "processTask",
        "class": "OC\\TaskProcessing\\Manager",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/public/BackgroundJob/Job.php",
        "line": 61,
        "function": "run",
        "class": "OC\\TaskProcessing\\SynchronousBackgroundJob",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php",
        "line": 43,
        "function": "start",
        "class": "OCP\\BackgroundJob\\Job",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php",
        "line": 29,
        "function": "start",
        "class": "OCP\\BackgroundJob\\QueuedJob",
        "type": "->"
      },
      {
        "file": "/var/www/nextcloud/cron.php",
        "line": 175,
        "function": "execute",
        "class": "OCP\\BackgroundJob\\QueuedJob",
        "type": "->"
      }
    ],
    "File": "/var/www/nextcloud/apps/context_chat/lib/Service/LangRopeService.php",
    "Line": 168,
    "message": "Unknown error while processing TaskProcessing task",
    "exception": {},
    "CustomMessage": "Unknown error while processing TaskProcessing task"
  }
}
{
  "reqId": "qOg6IWbpvn6KUkeYrmb8",
  "level": 2,
  "time": "2025-10-02T16:21:32+02:00",
  "remoteAddr": "",
  "user": "--",
  "app": "no app in context",
  "method": "",
  "url": "--",
  "message": "A TaskProcessing context_chat:context_chat task with id 11 failed with the following message: Error received from Context Chat Backend (ExApp) with status code 504: unknown error",
  "userAgent": "--",
  "version": "32.0.0.13",
  "data": []
}

# occ context_chat:stats

ContextChat statistics:
Installed time: 2025-10-01 08:38 UTC
Index complete time: 2025-10-02 09:44 UTC
Total time taken for complete index: 1 days 01:05 (hh:mm)
Total eligible files: 9262
Files in indexing queue: 101
New files in indexing queue (without updates): 100
Queued documents (without files):array (
)
Files successfully sent to backend: 8701
Indexed documents: array (
  'files__default' => 8716,
)
Actions in queue: 0

Using occ context_chat:search works.

[DaphneMuller]

@pmarini-nc hey Pietro! I hope you're doing well and great to see you back in the Nextcloud community 💙 I noticed that there was some confusion among new employees whether you are working at the company or not. You might want to consider changing your GitHub handle, as your format with -nc is almost a standard format that is very often used by employees.

[kyteinsky]

hello,
there should be a download button for the ex-app logs in the "admin settings -> assistant/artificial intelligence -> context chat" section. Can you find the error log there and maybe if the file is not private, share it with us so we may be able to test the fix?

[pmarini-nc]

@pmarini-nc hey Pietro! I hope you're doing well and great to see you back in the Nextcloud community 💙 I noticed that there was some confusion among new employees whether you are working at the company or not. You might want to consider changing your GitHub handle, as your format with -nc is almost a standard format that is very often used by employees.

Hello @DaphneMuller , thank you for contacting me! Changing the Github handle has some consequences on one's Github identity, see https://docs.github.com/en/account-and-profile/concepts/username-changes. The System Address Book is a good source of information of the company employee directory. Moreover the Github profile indicates clearly that I'm not part of nextcloud-gmbh anymore (organizations) and points to my company website and social networks. Do get in contact with me, if you think you need additional clarifications, you'll find all the info in RCA Systems website.

[pmarini-nc]

hello, there should be a download button for the ex-app logs in the "admin settings -> assistant/artificial intelligence -> context chat" section. Can you find the error log there and maybe if the file is not private, share it with us so we may be able to test the fix?

Hello @kyteinsky , sorry for the late reply. In the next few weeks I'll not be able to provide the information you have been asking me, unfortunately. I'm currently trying LocalAI as backend, mainly because of HaRP not supporting multi-tenancy at the moment (see nextcloud/HaRP#49) and I may need to reinstall Context Chat and Context Chat Backend.

Thanks again for your great support! In November / December I should have more time and send you the logs.

[kyteinsky]

thanks for the info. I'll merge the linked PR in the meantime and close this issue. You can verify if the fix works whenever you have time and if it doesn't, we can always reopen this issue.