Skip to content

v1.1.0

Choose a tag to compare

@github-actions github-actions released this 04 Nov 08:44
· 107 commits to main since this release

PageSift 1.1.0 Release Notes

Release Version: v1.1.0
License: AGPL-3.0-only (Commercial licensing available)

Changes since 1.0.0. Measurements are in the README and benchmarks.


New input formats

Group Formats Notes
Legacy Office .doc, .xls, .ppt, .xlsb Word keeps headings, lists, tables, footnotes; Word 6/95 is text only
OpenDocument .odt, .ods, .odp, flat XML .fodt, .fods, .fodp, .fodg
Word processors .rtf, .wpd, .wp, .hwp, .hwpx WordPerfect for Mac 1.x and 3.x is not read
E-books .mobi, .azw, .azw3, .prc, .fb2 DRM reports document is encrypted
Email .eml, .mbox, .msg Attachments are converted when their type is supported; a compressed RTF body is read
Archives .zip, .tar, .tgz, .gz, .bz2, .xz, .7z Members are converted recursively, up to 3 levels and 1,000 members
Images .png, .jpg, .gif, .bmp, .tif, .webp Requires OCR to be on; animated WebP is not read
Databases .sqlite, .db, .gpkg, .dbf Read from the file format; no SQL is run
XML documents DocBook, JATS, SVG (text only), OPML
Text and notebooks Markdown family, reStructuredText, AsciiDoc, Org, LaTeX, Typst, BibTeX, RIS, subtitles, Jupyter
Source code 373 languages by extension or file name One fenced code block with the language tag. C, C++, C#, Go, Java, JavaScript, TypeScript, Lua, PHP, Python and Rust are split into one section per top-level function or class, with function, class and import names in the metadata. Other languages, and files that are over 2 MB or do not parse cleanly, stay one block

Fetching a URL uses the same readers for binary formats. Text, data and source code fetched from the web stay on the web
page path, as before.

New integrations

  • LangChain: integrations/langchain-pagesift, a document loader.
  • LlamaIndex: integrations/llama-index-readers-pagesift, a reader.
  • C ABI: crates/pagesift-ffi, with a header in crates/pagesift-ffi/include/pagesift.h.
  • Language bindings in bindings/, each with its own tests: C, Go, Java, Kotlin, C#, Ruby, PHP, Lua (LuaJIT), Perl, Dart, Swift, Zig. Each release has pagesift-ffi-<version>-<target>.tar.gz with the library and header for Linux x86_64, Linux aarch64 and macOS arm64; on other platforms build it with cargo build --release -p pagesift-ffi.
  • Binding packages:
    • With the library inside (Linux x86_64, Linux aarch64, macOS arm64): Maven Central io.github.yolow92:pagesift (Java, Kotlin), NuGet Pagesift, RubyGems pagesift, pub.dev pagesift.
    • Using a library you install: Packagist yolow92/pagesift, LuaRocks pagesift (LuaJIT only), and Go, Swift and Zig from Git tags.
    • Perl and C stay in the repository and are not published to a registry. No binding package has a Windows library.

Source-code parsing

crates/pagesift-syntax is a parsing runtime written in Rust that reads tree-sitter grammars. Its parse tables are converted from the upstream grammars; their sources and licenses are in crates/pagesift-syntax/THIRD-PARTY.md. Each included grammar is checked against that grammar's own test corpus.

Fixes

  • PDF: Type 1C fonts whose glyph names only repeat the character code (C65) no longer produce empty output.

Known limits

Support for more file types, and function and class sections for more source-code languages, will be added over time. If you need one, please open an issue.

  • Source code outside the languages named above is not parsed: functions and classes are not extracted.
  • Function and class sections split at the top level of a file; code inside a namespace or module stays in one section.
  • .doc: page headers, footers and comments are left out. List numbers are not kept; numbered lists come out as bullets.
  • dBASE memo files (.dbt, .fpt) are not read; memo columns stay empty.
  • Audio, video, HEIC and JPEG 2000 files, .pst, iWork (.pages, .numbers, .key) and RAR are not supported.