localdocs: implement .docx support #2986
Conversation
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
If we don't copy DocumentInfo, we can store uncopyable objects inside of it. Signed-off-by: Jared Van Bortel <jared@nomic.ai>
We already use 'doc' to refer to a parsed representation of the document, not the file info. Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
cebtenzzre
force-pushed
the
docx-support
branch
from
65c97dd
to
448da04
Compare
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
Apple Clang still does not support __cpp_aggregate_paren_init (P0960R3). Signed-off-by: Jared Van Bortel <jared@nomic.ai>
| if (m_chunk.length() < maxChunkSize + 1) { | ||
| word = m_reader->word(); | ||
| if (m_chunk.isEmpty()) | ||
| m_page = m_reader->page(); // page number of first word |
There was a problem hiding this comment.
okay, so chunkstreamer is a class now? what is the lifetime of this object?
There was a problem hiding this comment.
It's matched to the lifetime of Database. I thought it would make sense to separate the state from the main Database class, since they have separate responsibilities, even though they work together.
That change isn't part of this PR anymore after the merge, since it was also done on main. I wanted to make sure the version number was no longer |
|
No, that's okay. Thanks for the explanation. |
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
|
The build passed for Mac and Linux, and on Windows after adjusting the CI timeouts (since build-and-test-gpt4all-chat is missing the timeout adjustment, even though the installer jobs have it). |
Using DuckX to parse .docx files similar to the way we parse PDF.
Leaving as draft until we can resolve the fact that we are chunking paragraphs and not pages. The best way to fix that is to stream the document instead of grabbing large discrete chunks of it, but this is blocked on merge of #2969 because that also touches chunkStream.edit: With the recent changes, this PR now streams the docx statefully so we don't need to worry about seeking to a particular run/paragraph every time we enter the database code, and we slice by time and not some arbitrary unit such as pages to allow the reader code to be more generic.