Repository navigation
v1.1.0
PageSift 1.1.0 Release Notes
Release Version:
v1.1.0
License: AGPL-3.0-only (Commercial licensing available)
Changes since 1.0.0. Measurements are in the README and benchmarks.
New input formats
| Group | Formats | Notes |
|---|---|---|
| Legacy Office | .doc, .xls, .ppt, .xlsb |
Word keeps headings, lists, tables, footnotes; Word 6/95 is text only |
| OpenDocument | .odt, .ods, .odp, flat XML .fodt, .fods, .fodp, .fodg |
|
| Word processors | .rtf, .wpd, .wp, .hwp, .hwpx |
WordPerfect for Mac 1.x and 3.x is not read |
| E-books | .mobi, .azw, .azw3, .prc, .fb2 |
DRM reports document is encrypted |
.eml, .mbox, .msg |
Attachments are converted when their type is supported; a compressed RTF body is read | |
| Archives | .zip, .tar, .tgz, .gz, .bz2, .xz, .7z |
Members are converted recursively, up to 3 levels and 1,000 members |
| Images | .png, .jpg, .gif, .bmp, .tif, .webp |
Requires OCR to be on; animated WebP is not read |
| Databases | .sqlite, .db, .gpkg, .dbf |
Read from the file format; no SQL is run |
| XML documents | DocBook, JATS, SVG (text only), OPML | |
| Text and notebooks | Markdown family, reStructuredText, AsciiDoc, Org, LaTeX, Typst, BibTeX, RIS, subtitles, Jupyter | |
| Source code | 373 languages by extension or file name | One fenced code block with the language tag. C, C++, C#, Go, Java, JavaScript, TypeScript, Lua, PHP, Python and Rust are split into one section per top-level function or class, with function, class and import names in the metadata. Other languages, and files that are over 2 MB or do not parse cleanly, stay one block |
Fetching a URL uses the same readers for binary formats. Text, data and source code fetched from the web stay on the web
page path, as before.
New integrations
- LangChain:
integrations/langchain-pagesift, a document loader. - LlamaIndex:
integrations/llama-index-readers-pagesift, a reader. - C ABI:
crates/pagesift-ffi, with a header incrates/pagesift-ffi/include/pagesift.h. - Language bindings in
bindings/, each with its own tests: C, Go, Java, Kotlin, C#, Ruby, PHP, Lua (LuaJIT), Perl, Dart, Swift, Zig. Each release haspagesift-ffi-<version>-<target>.tar.gzwith the library and header for Linux x86_64, Linux aarch64 and macOS arm64; on other platforms build it withcargo build --release -p pagesift-ffi. - Binding packages:
- With the library inside (Linux x86_64, Linux aarch64, macOS arm64): Maven Central
io.github.yolow92:pagesift(Java, Kotlin), NuGetPagesift, RubyGemspagesift, pub.devpagesift. - Using a library you install: Packagist
yolow92/pagesift, LuaRockspagesift(LuaJIT only), and Go, Swift and Zig from Git tags. - Perl and C stay in the repository and are not published to a registry. No binding package has a Windows library.
- With the library inside (Linux x86_64, Linux aarch64, macOS arm64): Maven Central
Source-code parsing
crates/pagesift-syntax is a parsing runtime written in Rust that reads tree-sitter grammars. Its parse tables are converted from the upstream grammars; their sources and licenses are in crates/pagesift-syntax/THIRD-PARTY.md. Each included grammar is checked against that grammar's own test corpus.
Fixes
- PDF: Type 1C fonts whose glyph names only repeat the character code (
C65) no longer produce empty output.
Known limits
Support for more file types, and function and class sections for more source-code languages, will be added over time. If you need one, please open an issue.
- Source code outside the languages named above is not parsed: functions and classes are not extracted.
- Function and class sections split at the top level of a file; code inside a namespace or module stays in one section.
.doc: page headers, footers and comments are left out. List numbers are not kept; numbered lists come out as bullets.- dBASE memo files (
.dbt,.fpt) are not read; memo columns stay empty. - Audio, video, HEIC and JPEG 2000 files,
.pst, iWork (.pages,.numbers,.key) and RAR are not supported.