new: chat with file - backend side by nquang29 · Pull Request #1703 · BasedHardware/omi

nquang29 · 2025-01-16T09:31:41Z

No description provided.

beastoin · 2025-01-19T00:06:20Z

1/ so, the user can not ask the follow up question on the previous files. right ? i see the condition that check the last message only

@nquang29

nquang29 · 2025-01-19T03:42:15Z

1/ with the same thread and request body contain file_id that user want to ask, they can ask follow up question @beastoin
e.g. i asked about an image

mdmohsin7 · 2025-01-23T14:50:06Z

@nquang29 can you pls return the links for the thumbnails in /v2/messages if the message contains any? Not just the file ids, the thumbnails as well if they are images. I'll have to show them in the app like the screenshot below

nquang29 · 2025-01-24T04:28:14Z

@mdmohsin7 it's done, pls help me check it

mdmohsin7 · 2025-01-24T16:29:57Z

Hi @nquang29, the GET /v2/messages endpoint does not seem to return the files objects. It returns the file ids but not the files. See the below output from that endpoint call for reference. It has the file id but not the file

beastoin · 2025-01-25T02:25:49Z

#1617

nquang29 · 2025-01-25T05:17:20Z

Hi @mdmohsin7, Sorry for not noting the APIs clearly, I think we have had a misunderstanding:
after call api post /files, we received response below, we have :

id : id of file use for omi's system
openai_file_id: id of file we got from openai
We should use id for api send messages

and this is files object in the output:

mdmohsin7 · 2025-01-25T18:40:57Z

Thanks for the clarification @nquang29. Everything works perfectly, checkout the video in #1629 😄

@beastoin review time 🌚

beastoin · 2025-02-02T04:32:36Z

1/ i understand that if the app includes file_id, the backend will switch to chat-with-files, right ? if so, when will the app include it? e.g. Can i ask as a question about the file by sending the 2nd plain text message, with the first message is attached file ?

you know that the chat system can determine a user's memories by extracting topics, time from the question, which is parsed by the last 10 messages. from that we can help LLM understand about the current context for a better answer.

how does the chat-with-file work in term of context?

@nquang29

nquang29 · 2025-02-03T05:00:34Z

1/ you cannot ask the 2nd question with plain text message. When the first message includes attached files, the following messages have to attached files to be able to chat with files until the chat is clear - end thread of chat-with-files
@beastoin

beastoin · 2025-02-04T01:06:41Z

1/ did the app handle that logic, right ? > the following messages have to attached files

@nquang29

nquang29 · 2025-02-04T06:49:18Z

hi @mdmohsin7 , regarding logic thread - end thread:
when starting a thread to chat with file, the following messages need include attached files, and we will use 'clear chat' to force end thread
please help me to verify if this logic has been implemented in the app
or if you have any suggestions, feel free to share with me

mdmohsin7 · 2025-02-04T07:45:43Z

Hi @nquang29, the way currently it is implemented:

User attaches files, the file ids are sent with that message when the user presses the send button
After sending a message with files, when the user sends another message, this new message does not include the file ids
The clear chat functionality is already implemented and when the user presses on clear chat, it makes a call to DELETE v1/messages which also ends the thread

Pls let me know if any changes are required in this

nquang29 · 2025-02-05T09:22:13Z

Flow in BE that I implemented:

A thread for chat-with-file is started when the first message with file_ids is sent. At this point, the chat feature switches to chat-with-file mode.
Subsequent messages in the thread must also include file_ids to maintain the chat-with-file functionality. If a message is plain text (without file_ids), the logic for sending the message will follow the current flow, meaning the user cannot ask questions about the files attached earlier.
The chat-with-file thread will end if the user presses 'Clear Chat', and the chat feature will revert to the current chat flow.

so these messages need include file_ids until the chat is cleared @mdmohsin7
After sending a message with files, when the user sends another message, this new message does not include the file ids

nquang29 · 2025-02-05T09:25:42Z

1/ The app doesn't handle this logic yet. I’ve just discussed with @mdmohsin7 about this case, and I've asked him to add it @beastoin

mdmohsin7 · 2025-02-05T12:02:50Z

Got it, @nquang29. I can include the file IDs in subsequent messages. However, it seemed to work fine even without them, as the chat was still responding to content related to the file to some extent when I was testing it.

Also, could you please add an indicator or a field to specify which message first included the file? Otherwise, it would be difficult for me to determine which messages should display the file thumbnails, as I rely on the files and files_id field in the response for this and all the subsequent human messages will have those fields as non empty.

nquang29 · 2025-02-06T06:30:49Z

I have a situation:
The user attached 2 files in the first message, and in the second message, the user attached 1 additional file. How can you determine which images have already been displayed and decide whether to show the newly attached image from the second message? @mdmohsin7

beastoin · 2025-02-06T07:13:21Z

2/ have you use the chat session or the similar term yet?, e.g. chat_session{type, file_ids, } . for example #1758

@nquang29 @mdmohsin7

mdmohsin7 · 2025-02-06T08:42:16Z

I have a situation: The user attached 2 files in the first message, and in the second message, the user attached 1 additional file. How can you determine which images have already been displayed and decide whether to show the newly attached image from the second message? @mdmohsin7

I think if the backend only returns the files when the user has actually attached them to that message, it will cover all cases. You can return the file IDs for all related messages, but return the actual file objects only for the messages to which they were attached. The file IDs will help identify which files the messages are related to, while the file objects will only be used to display the files. What do you think? @nquang29

mdmohsin7 · 2025-02-06T08:46:47Z

2/ have you use the chat session or the similar term yet?, e.g. chat_session{type, file_ids, } . for example #1758

Nope on my side @beastoin

nquang29 · 2025-02-06T18:41:02Z

@mdmohsin7 i made some changes:
only messages with attached files by user will include files object in response, the subsequent messages that included file_ids will not return files object in response unless new files are attached by user
please help me to verify and update the app

nquang29 · 2025-02-06T18:45:18Z

2/ No, sir. Is it necessary to use a session in this case? I'm not sure what its purpose would be in the context you're asking about. @beastoin

mdmohsin7 · 2025-02-07T18:48:39Z

Hi @nquang29 yes it works

nquang29 · 2025-02-08T10:20:10Z

1/ The app has handled the logic of adding file_id to the subsequent messages @beastoin

beastoin · 2025-02-12T06:35:10Z

1/ ✅
2/ if you have a good structure(e.g. data structure) you done 70% of the job. so, think a bit about using the session to manage the user's chat. basically, file_ids just a property of a specific chat session. but non block, you could create a new ticket to impl it.
3/ what is this loc for ? https://github.com/BasedHardware/omi/pull/1703/files#diff-c310bf3c570d1fab00eb842de5e8edd7dcdca5a5f1751ba0e223ab224fe1ef24R126
4/ does chat with file includes user's memories, user's fact yet ?

@nquang29

nquang29 · 2025-02-13T08:28:05Z

2/ I have already implemented a chat session:

A session starts when the initial message is sent (i.e., clear chat).
The current flow allows sending a message without exist session, and will links the message to the session if session exists
When retrieving messages, if no session exists, all messages are returned. If a session exists, only messages related to that session are returned.
The session ends when the chat is cleared.

3/ I use this code to determine whether a message is related to a chat with a file or not. I have updated this logic to recognize file-related questions using a prompt.

4/ Based on /3, the system can identify whether a question related to a file or not, so users can ask about their memory as well.
@beastoin

beastoin · 2025-02-14T11:14:01Z

2/ ✅
3/ any case where it could conflict with the chat session ? e.g. the app requests to /messages with a chat session id but the reply is not based on the files (from that chat session).
4/ so for now, chat with file will not include the users memories, facts, right ?
5/ did the app include the chat session when calling /messages or not ? or, do we need it ?

@nquang29

nquang29 · 2025-02-14T19:04:19Z

3/ i just changed the logic for getting the newly attached file to prioritize retrieving it from the session

4/ that's right. I haven't quite figured out the connection between chat-with-file and the users memories or facts. Could you help me understand it clearly?

5/ the app will need to include chat session in case there is a list of chat session history similar to ChatGPT. In that case, the app needs to request a specific chat session id to distinguish and manage sessions or continue chatting in this session
However, for now, since the user only chats and clears the chat within a single session, I think this is not necessary
@beastoin

beastoin · 2025-02-15T01:44:39Z

3/ chat session must have a plugin_id (or app_id) field to determine if the user is chatting with omi or another app (e.g. Naval). e.g. i can have 10 sessions with omi but 3 sessions with Naval, the last session chatting with Omi is not the same with the last session chatting with Naval.
4/ ✅ ok, will tell you more later haha
5/ do we need Moshin to do anything more in the app ?

the last commit pls sir, then drop the demo video + deploy plan.

@nquang29 @mdmohsin7

nquang29 · 2025-02-15T08:23:56Z

3/ i have updated it. thanks for the explanation, i now have a better understanding of the plugin on the app.
5/ no sir, app will not need any futher updates
@beastoin

nquang29 · 2025-02-15T08:25:42Z

hi @mdmohsin7
i made some changes on BE, pls help me to check if the app sill working as expected
and record a video demo for both cases: chat with omi and chat with another app

mdmohsin7 · 2025-02-15T09:15:47Z

When I change from an app back to omi, I get this error. WIll have to create an index in firestore

Edit: I've created this index in dev env
cc: @beastoin need to create the same index in prod

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/fastapi/routing.py", line 193, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mohsin/Documents/work/based-hardware/Friend/backend/routers/chat.py", line 352, in get_messages
    messages = chat_db.get_messages(uid, limit=100, include_memories=True, plugin_id=plugin_id, chat_session_id=chat_session_id)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mohsin/Documents/work/based-hardware/Friend/backend/utils/other/endpoints.py", line 83, in measure_time
    result = func(*args, **kw)
             ^^^^^^^^^^^^^^^^^
  File "/Users/mohsin/Documents/work/based-hardware/Friend/backend/database/chat.py", line 119, in get_messages
    for doc in messages_ref.stream():
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/cloud/firestore_v1/query.py", line 357, in _make_stream
    response_iterator, expected_prefix = self._get_stream_iterator(
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/cloud/firestore_v1/query.py", line 229, in _get_stream_iterator
    response_iterator = self._client._firestore_api.run_query(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/cloud/firestore_v1/services/firestore/client.py", line 1562, in run_query
    response = rpc(
               ^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target
    _retry_error_helper(
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper
    raise final_exc from source_exc
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target
    result = target()
             ^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 174, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.FailedPrecondition: 400 The query requires an index. You can create it here: https://console.firebase.google.com/v1/r/project/based-hardware-dev/firestore/indexes?create_composite=ClNwcm9qZWN0cy9iYXNlZC1oYXJkd2FyZS1kZXYvZGF0YWJhc2VzLyhkZWZhdWx0KS9jb2xsZWN0aW9uR3JvdXBzL21lc3NhZ2VzL2luZGV4ZXMvXxABGhMKD2NoYXRfc2Vzc2lvbl9pZBABGgsKB2RlbGV0ZWQQARoNCglwbHVnaW5faWQQARoOCgpjcmVhdGVkX2F0EAIaDAoIX19uYW1lX18QAg

nquang29 · 2025-02-15T09:19:02Z

@mdmohsin7 yes, pls help me to create index on your local env firebase
i will note it in deploy plan

mdmohsin7 · 2025-02-15T09:25:52Z

@nquang29 @beastoin works fine on the app side

ScreenRecording_02-15-2025.14-37-10_1.1.mp4

beastoin · 2025-02-15T10:00:29Z

resolve conflicts pls sir @nquang29

nquang29 · 2025-02-15T10:16:53Z

Deploy plan:
1/ add config: BUCKET_CHAT_FILES="chat_files" (can replace with another name)

2/ create bucket in gg cloud storage with name in 1/

3/ create index in firestore db:

collectionID: messages
fields: chat_session_id Ascending, deleted Ascending, plugin_id Ascending, created_at Descending, name Descending

4/ deploy BE service

nquang29 · 2025-02-15T10:17:26Z

resolved conflict sir @beastoin

beastoin · 2025-02-15T10:33:07Z

lgtm, congratulation @nquang29 🚀

beastoin · 2025-02-16T02:35:03Z

hi @nquang29, man, regrading to the first step of the deploy plans, do we really need to make the bucket public ?

nquang29 · 2025-02-16T02:46:11Z

No need, sir @beastoin . It still works for not-public. This is the test environment, right? I am not the one who created it in the test environment.

beastoin · 2025-02-16T04:57:46Z

deploy plan drop #1617 (comment)

nquang29 force-pushed the 1617_chat_file_be branch from 2115bb1 to 9756686 Compare January 16, 2025 09:32

nquang29 changed the title ~~new: chat with file~~ new: chat with file - backend side Jan 16, 2025

nquang29 force-pushed the 1617_chat_file_be branch from 9756686 to f2f80e4 Compare January 16, 2025 09:38

mdmohsin7 mentioned this pull request Jan 17, 2025

Ability to attach files in chat - app side #1629

Merged

nquang29 added 7 commits February 15, 2025 17:05

resolve conflict

19bb19b

add files in api send message

9df5c99

chore: only return file of message attached

6068bb4

add promt file & session p1

0265e0f

session p2

b012dea

prioritize get new file from session

449a1f2

add plugin to session

ae1199c

nquang29 force-pushed the 1617_chat_file_be branch from 4aafd6e to ae1199c Compare February 15, 2025 10:05

beastoin merged commit 588f2f8 into BasedHardware:main Feb 15, 2025

Conversation

nquang29 commented Jan 16, 2025

Uh oh!

beastoin commented Jan 19, 2025

Uh oh!

nquang29 commented Jan 19, 2025

Uh oh!

mdmohsin7 commented Jan 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nquang29 commented Jan 24, 2025

Uh oh!

mdmohsin7 commented Jan 24, 2025

Uh oh!

beastoin commented Jan 25, 2025

Uh oh!

nquang29 commented Jan 25, 2025

Uh oh!

mdmohsin7 commented Jan 25, 2025

Uh oh!

beastoin commented Feb 2, 2025

Uh oh!

nquang29 commented Feb 3, 2025

Uh oh!

beastoin commented Feb 4, 2025

Uh oh!

nquang29 commented Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mdmohsin7 commented Feb 4, 2025

Uh oh!

nquang29 commented Feb 5, 2025

Uh oh!

nquang29 commented Feb 5, 2025

Uh oh!

mdmohsin7 commented Feb 5, 2025

Uh oh!

nquang29 commented Feb 6, 2025

Uh oh!

beastoin commented Feb 6, 2025

Uh oh!

mdmohsin7 commented Feb 6, 2025

Uh oh!

mdmohsin7 commented Feb 6, 2025

Uh oh!

nquang29 commented Feb 6, 2025

Uh oh!

nquang29 commented Feb 6, 2025

Uh oh!

mdmohsin7 commented Feb 7, 2025

Uh oh!

nquang29 commented Feb 8, 2025

Uh oh!

beastoin commented Feb 12, 2025

Uh oh!

nquang29 commented Feb 13, 2025

Uh oh!

beastoin commented Feb 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nquang29 commented Feb 14, 2025

Uh oh!

beastoin commented Feb 15, 2025

Uh oh!

nquang29 commented Feb 15, 2025

Uh oh!

nquang29 commented Feb 15, 2025

Uh oh!

mdmohsin7 commented Feb 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nquang29 commented Feb 15, 2025

Uh oh!

mdmohsin7 commented Feb 15, 2025

Uh oh!

beastoin commented Feb 15, 2025

Uh oh!

nquang29 commented Feb 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mdmohsin7 commented Jan 23, 2025 •

edited

Loading

nquang29 commented Feb 4, 2025 •

edited

Loading

beastoin commented Feb 14, 2025 •

edited

Loading

mdmohsin7 commented Feb 15, 2025 •

edited

Loading

nquang29 commented Feb 15, 2025 •

edited

Loading