Add Ability to Summarize Documents #800

MythicalCow · 2024-06-06T18:06:57Z

Uses entire file text and summarizer model to generate document summary.
Uses the contents of the user's query to create a tailored summary.
Integrates with File Filters New Feature: Adding File Filtering to Conversations #788 for a better UX.

…be completed

… filtering to be added

…nd clean up required

…m/MythicalCow/khoj into features/conversation-file-filter

…nt conversation's file filter

Integrations Merge

…or catching.

merging API issue fix related to duplicate file management

…orous error catching

…tures/document-summarization

src/khoj/database/migrations/0044_conversation_file_filters.py

…w one

src/khoj/routers/api_chat.py

…and to tool selection

…nt-summarization

src/khoj/routers/api_chat.py

tests/test_openai_chat_director.py

…chat configuration

sabaimran

🚀

sabaimran · 2024-06-18T14:01:57Z

Closes #509
Closes #787

MythicalCow and others added 30 commits May 28, 2024 18:09

UI update for file filtered conversations

7c6be74

linter fixes

599917e

linter fixes

cf9f4dd

UI and API endpoints integrations complete. file querying changes to …

8569092

…be completed

UI streamlining by adding a plus button instead of a static search bar

8640f24

backend and UI conversation ID based file filtering implemented. file…

0d87a8f

… filtering to be added

file filtering based on selected files implemented. further testing a…

d2a369f

…nd clean up required

final touches and unnecessary comments removed.

75d251d

adding authentication requirement to file-filters endpoint

fcc99db

fileobjectadapter created and tested

e515f04

removed temp file

8de18ac

Apply file filters with correct syntax

b727a52

file object adapter notes for later

71184d5

made PR changes for a better user experience

13d71a1

Merge branch 'khoj-ai:master' into features/conversation-file-filter

0c129ca

padding fix

9714b63

Merge branch 'features/conversation-file-filter' of https://github.co…

649edd0

…m/MythicalCow/khoj into features/conversation-file-filter

small UI improvements

cd204c7

embeddings update to handle FileObjects

4a2233e

embeddings update to handle FileObjects

e5c8576

Merge branch 'khoj-ai:master' into features/document-summarization

ec0b0a5

save

312b8c4

added support for new command: /summarize

0bf3d57

addressed security risk highlighted in PR.

60de2b4

added handling for automatically adding an uploaded file to the curre…

f80654b

…nt conversation's file filter

Merge pull request #1 from MythicalCow/features/conversation-file-filter

27e3771

Integrations Merge

added integrations between file filter and summarize command with err…

c7ea0d9

…or catching.

fixed API issue related to duplicate file management

1f52e9a

Merge pull request #2 from MythicalCow/features/conversation-file-filter

464576f

merging API issue fix related to duplicate file management

added query based context to summarization command and added more rig…

55d191b

…orous error catching

MythicalCow added 7 commits June 12, 2024 23:22

database migration fix

dc291b2

added support for orgmode to /summarize

7c05b51

return type specifier

7d99d1d

merge conflict fix?

6209acf

adding logging

1851c0e

Merge branch 'master' of https://github.com/MythicalCow/khoj into fea…

435d8e6

…tures/document-summarization

bug fix for update_raw_text adapter

a538485

MythicalCow marked this pull request as ready for review June 14, 2024 05:02

MythicalCow requested a review from sabaimran June 14, 2024 05:02

sabaimran reviewed Jun 14, 2024

View reviewed changes

src/khoj/database/migrations/0044_conversation_file_filters.py Show resolved Hide resolved

resolve merge issues with migration files. keep the old one, add a ne…

070bde2

…w one

sabaimran reviewed Jun 14, 2024

View reviewed changes

src/khoj/routers/api_chat.py Outdated Show resolved Hide resolved

sabaimran reviewed Jun 14, 2024

View reviewed changes

src/khoj/routers/api_chat.py Show resolved Hide resolved

sabaimran added 3 commits June 14, 2024 11:58

Add error handling for when file object not found, add summarize comm…

0b2e391

…and to tool selection

Merge branch 'master' of github.com:khoj-ai/khoj into features/docume…

1da9bc4

…nt-summarization

Add summarize support for normal streamed chat, non socket

d19ec47

sabaimran reviewed Jun 14, 2024

View reviewed changes

src/khoj/routers/api_chat.py Show resolved Hide resolved

sabaimran and others added 3 commits June 14, 2024 12:33

Remove print line for file list

2919c16

added some tests for /summarize command.

bba083b

additional tests added in

3344a96

sabaimran reviewed Jun 16, 2024

View reviewed changes

tests/test_openai_chat_director.py Show resolved Hide resolved

tests/test_openai_chat_director.py Show resolved Hide resolved

tests/test_openai_chat_director.py Show resolved Hide resolved

MythicalCow and others added 4 commits June 16, 2024 23:26

resolving merge conflict

d7ef0d8

added additional tests

5933042

added tests to offline chat director

10e2af2

Fix normal api for chat to retrieve notes, update method for offline …

8e6c1c2

…chat configuration

sabaimran approved these changes Jun 17, 2024

View reviewed changes

Skip offline chat director tests

de112ed

debanjum changed the title ~~New Feature: Document Summarization Command~~ Add Ability to Summarize Documents Jun 18, 2024

sabaimran merged commit d4e5c95 into khoj-ai:master Jun 18, 2024
6 of 7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Ability to Summarize Documents #800

Add Ability to Summarize Documents #800

MythicalCow commented Jun 6, 2024

sabaimran left a comment

sabaimran commented Jun 18, 2024

Add Ability to Summarize Documents #800

Add Ability to Summarize Documents #800

Conversation

MythicalCow commented Jun 6, 2024

sabaimran left a comment

Choose a reason for hiding this comment

sabaimran commented Jun 18, 2024