2026.07.07

This release introduces a major overhaul of the local AI subsystem, adding support for Llama.cpp and the IBM Granite model, alongside extensive naming standardization and build system improvements. A new abstract AI runner interface unifies inference backends, while rigorous snake_case normalization aligns the entire codebase with the project's coding guidelines.

The llama hums, a granite sound,
With abstract runners all around.
The names in snakes now coil and sleep,
While docs and tests more vigil keep.
A modular heart, a cleaner choice,
DocWire finds its stronger voice.
🦙🧠🔧📚

• Features

  • Llama.cpp Integration: Added an optional Llama.cpp backend (docwire_ai_llama) for running GGUF-based models. A new llama_runner class implements the abstract ai_runner interface, providing process() and embed() methods with thread-safe backend management.
  • IBM Granite Model Support: The new local-ai-model-granite feature installs the IBM Granite 4.0 1B Q8_0 GGUF model, making a compact yet capable LLM available out of the box.
  • Abstract AI Runner Interface: Introduced docwire::ai::ai_runner – a pure virtual base that defines process(), embed(), and unload() for all local AI backends. Backends guarantee thread safety and deterministic teardown.
  • Modular AI Backends: Split the local AI runtime into separate optional libraries: docwire_ai_ct2 (CTranslate2) and docwire_ai_llama (Llama.cpp). A common docwire_ai library now contains shared components such as ai::task, ai::summarize, ai::translate, and ai::embed.
  • High-Level Local AI API: New convenience classes in the ai::local namespace (summarize, translate, task, passage::embedder, query::embedder) automatically select a default backend model, simplifying the most common use cases.
  • Documentation: Added the DocWire SDK Manifesto and Coding Guidelines to the repository documentation (doc/manifesto.md, doc/coding_guidelines.md), outlining the project’s philosophy and architectural rules.

• Improvements

  • Naming Normalization: All class/struct/enum names have been converted to snake_case according to the new coding guidelines. Publicly visible types such as ChainElement, PlainTextExporter, HtmlWriter, ParsingChain, TransformerFunc, FileStream, ZipReader, etc., are now chain_element, plain_text_exporter, html_writer, parsing_chain, transformer_func, file_stream, zip_reader, and so on. Document element tags (document::Text → document::text, mail::Mail → mail::mail, …) have been similarly renamed.
  • Enum Names Standardized: Enumerations such as Model, ImageDetail, Language, output_type etc., inside various namespaces now follow snake_case (e.g., openai::model, openai::image_detail, etc.).
  • Upgraded libbfio to version 20260623 for improved stability and compatibility.
  • ASan Fix for Recent MSVC: Added _DISABLE_OPTIONAL_ANNOTATION to prevent link errors when building with AddressSanitizer on MSVC.
  • CTranslate2 Build Fix: The custom spdlog submodule has been replaced by the standard vcpkg spdlog port, resolving spdlog‑related build errors.
  • Vcpkg Conflict Resolution: Fixed a file installation conflict between the ctranslate2 and nlohmann-json ports.
  • Mailio Crash Prevention: The EML parser’s BoundaryTracker now handles prematurely closed multipart sections and injects empty parts when necessary, preventing a crash in the underlying mailio library.

• Refactor

  • Backend Separation: The former monolithic local_ai library has been split into docwire_ai_ct2 (CTranslate2 runner, tokenizer) and docwire_ai_llama (Llama.cpp runner). The docwire_local_ai library becomes a convenience layer that links the chosen backends.
  • Namespace & Module Reorganization:
    • CTranslate2‑specific classes moved into docwire::ai::ct2 (ct2_runner, tokenizer).
    • Llama.cpp‑specific code lives in docwire::ai::llama.
    • Shared AI chain elements reside directly in docwire::ai.
    • Local default wrappers moved to docwire::ai::local.
  • Configuration Strong Types: Introduced strong_type and several dedicated types (context_size, thread_count, token_limit, temperature, min_p, batch_size) in model_inference_config.h for safer Llama parameter passing.
  • Build System Restructuring: New CMake options DOCWIRE_CT2 and DOCWIRE_LLAMA control optional compilation of the respective backends. Feature flags local-ai-ct2, local-ai-llama, and local-ai-model-granite are exposed in the vcpkg port.

• Tests

  • Backend Integration Tests: Added local_ai_ct2_integration and local_ai_llama_integration tests that execute a full pipeline (document parsing → text extraction → local AI task) to validate the new backends.
  • More Forgiving AI Tests: Local AI summarization tests now use fuzzy matching instead of exact string comparison, reducing flakiness due to model output variation.
  • EML Robustness Tests: New test cases (endboundary_first.eml, missing_inner_closing_boundary.eml, nested_multiparts_missing.eml, valid_format.eml) ensure correct handling of malformed or non‑standard MIME boundaries.
  • Test Infrastructure Fixes: Added a GoogleTest PrintTo stringification for the ContentTypeTestCase struct, resolving formatting issues in parameterized content‑type tests.
  • Build Guard: Example tests that require local AI libraries are conditionally compiled only when the corresponding targets are available.

2026.05.25

This release introduces a significant shift in the open-source licensing to AGPLv3 and brings major enhancements to content type detection. We've implemented a multi-stage pipeline with heuristic fallbacks to robustly identify ZIP-based formats and images, even on non-seekable network streams. The test infrastructure has been modularized and consolidated for better developer efficiency, and the HTTP server's SSL configuration has been modernized for improved stability.

The license shifts, a wider net,
To AGPLv3, our standards set.
With content found in streams so deep,
Heuristics wake, while others sleep.
The server's light, in SSL glows,
Refactored tests, where wisdom flows.
A modular heart, a consolidated sight,
DocWire moves forward, into the light.
⚖️🔍🛡️🚀

• Features

  • AGPLv3 Licensing: Transitioned the open-source license from GPLv2 to GNU Affero General Public License version 3 (AGPLv3). This shift not only closes the "SaaS loophole" (ensuring network users receive source code), but also provides key modern advantages: explicit patent grants that protect users from litigation by contributors, compatibility with the Apache License 2.0 (facilitating integration with a wider ecosystem), and improved international legal language for global enforceability.
  • Heuristic Content Type Detection: Introduced specialized heuristic detectors for Images (BMP, WEBP) and ZIP-based containers (OOXML, ODF). These detectors prioritize performance by checking local file headers in the first 4KB before falling back to deep inspection, specifically addressing detection failures on non-seekable streams with libmagic 5.47+.

• Improvements

  • MIME Type Normalization: Signature detection now automatically normalizes legacy or non-standard MIME types returned by libmagic to modern IANA standards (e.g., text/xml to application/xml), providing a consistent target for downstream components.
  • Performance Optimization in Detectors: Updated heuristic detectors (ASP, HTML) to use std::string_view for content searching, reducing memory overhead and improving detection speed.
  • Deterministic MIME Tie-Breaking: Enhanced the data_source logic to use alphabetical tie-breaking when multiple MIME types have identical confidence levels, ensuring consistent results across platforms.
  • Modernized SSL Server Setup: Refactored httplib::SSLServer initialization to use the recommended setup callback for certificate and key configuration, aligning with the latest library API changes.

• Refactor

  • Modular Test Suite: Substantially refactored the test infrastructure by splitting the monolithic api_tests.cpp into specialized source files (core_tests.cpp, error_tests.cpp, log_tests.cpp, etc.) for improved maintainability.
  • Consolidated Test Executable: Consolidated all unit tests into a single docwire_tests binary to drastically reduce process creation overhead during comprehensive test cycles under Valgrind.

• Fixes

  • ZIP-based Format Detection: Fixed regressions in the detection of OOXML (DOCX, XLSX, PPTX) and ODF formats when processed via non-seekable streams.
  • General Stability: Resolved minor issues in the automated test suite and added missing include directives identified during code review.

2026.03.26

This release focuses on developer experience and parsing stability. A new resource management framework allows application developers to bundle and install essential data files (like AI models and OCR dictionaries) with a single line in their CMake configuration. The EML parser has also been substantially strengthened to handle complex nested MIME structures and malformed boundaries with deterministic content selection, ensuring significantly more reliable text extraction from emails.

One line to find, one line to stay,
The resources follow the SDK’s way.
The email stream, a clearer sight,
With boundaries tracked and sorted right.
A solid base, a robust frame,
DocWire answers every claim.
🐇✨📦📧🛡️

• Features

  • One-Command Resource Management: Introduced a robust CMake framework (docwire_deploy_resources, docwire_install_resources) that automates data file discovery and deployment. Developers can now bundle all necessary AI models and OCR dictionaries into their own applications with a single line in their build scripts.
  • Relocatable Package Support: Resource paths are now exported in the package configuration, enabling relocatable SDK installations and full support for custom CMAKE_INSTALL_DATADIR layouts.
  • Robust Resource Validation: CMake configuration now explicitly fails if required resource paths (like models or dictionaries) are missing, providing clear diagnostic errors early in the build process.
  • Custom Archive Definitions: Integrated an updated libmagic database (from upstream Nov 2024) to resolve regressions in ZIP and container detection, ensuring reliable file type identification without external patches.
  • Tiered Valgrind Suppression System: Implemented a granular suppression architecture with tool-specific files and a debug/release split, allowing for precise silencing of third-party false positives during deep thread-safety analysis.

• Improvements

  • EML Parsing Robustness: The email parser now features deterministic selection for multipart/alternative parts—prioritizing non-empty HTML content—and utilizes a new BoundaryTracker utility to correctly process complex nested structures and malformed boundaries.
  • Unnamed Attachment Support: The EML parser now handles attachments without filenames via std::optional, ensuring content is extracted and labeled correctly rather than being skipped.
  • Shared Library Resource Discovery: Resource resolution now employs a prioritized multi-candidate search strategy, favoring the library module path over the executable path. This ensures correct data file discovery when the SDK is used as a plugin in generic hosts like Python.
  • Modernized HTTP Integration: The HTTP module now uses httplib in CONFIG mode with native OpenSSL support, simplifying dependency management and improving build stability.
  • Sanitizer Stability: Introduced a fallback mechanism to disable asynchronous DNS resolution under ThreadSanitizer and Helgrind, bypassing known system-level stack-unwinding bugs on modern Linux distributions.
  • Plain Text Attachment Formatting: Refined the PlainTextWriter to provide cleaner spacing and clearer labeling for attachments in both PST and EML exports.

• Fixes

  • macOS Installation Pathing: Fixed installation errors on macOS by explicitly specifying LIBRARY and FRAMEWORK destinations for runtime dependencies.
  • Windows Runtime Dependency Filtering: Improved installation rules to robustly exclude OS-internal API sets and CI-specific system DLLs from the distribution package.

• Build / CI

  • SDK Installation Verification: Introduced a standalone test suite to verify package integrity independently of the build tree, ensuring both Debug and Release installations are fully functional for downstream users.

• Tests

  • Email Parsing Fixtures: Added a comprehensive set of EML test cases covering nested multiparts, folded boundaries, and unclosed inner boundaries to ensure long-term parsing stability.

2026.01.20

This release introduces a major overhaul of the SDK's core, focusing on modern C++20 features for XML parsing, robust type-safe data conversions, and configurable safety policies. Significant architectural changes have been made to enhance resilience against partial failures in parsers, improve memory management, and standardize date/time handling using std::chrono. The update also includes new utilities for named parameters and non-null or in-range assertions, alongside numerous fixes and documentation improvements.

The XML stream, a new path it takes,
With C++20, a modern API awakes.
Safety policies, strict or relaxed,
Data conversions, precisely taxed.
Partial failures, now gracefully met,
std::chrono dates, no more regret.
A robust foundation, strong and so keen,
DocWire's future, brightly foreseen.
✨🚀🛡️

• Features

  • Modern C++20 XML Parsing API: Introduced a new, expressive, and safe API for parsing XML documents, featuring a single-pass, forward-only reader and utilizing C++20 ranges and views for efficient node filtering and manipulation. This replaces the legacy XmlStream with xml::reader, xml::node_ref, xml::children, and xml::attributes.
  • Configurable Safety Policies: Implemented a general concept of safety policies (strict vs. relaxed) allowing developers to choose between robust error-checking (throwing exceptions on violations) and maximum performance (omitting checks). This is applied through checked and not_null wrappers, and the enforce utility.
  • Type-Safe Data Conversion Framework: Introduced a new convert namespace with try_to and to functions for robust, type-safe data conversions, replacing manual string-to-numeric/date conversions. This framework supports custom format tags like with::date_format and with::partial_match.
  • C++20 std::chrono for Date/Time Handling: Replaced all struct tm usage with std::chrono::sys_seconds for date and time attributes and conversions, enhancing type safety and robustness across the SDK.
  • Partial Failure Resilience in Parsers: Implemented message_counters and make_counted_message_callbacks in archives_parser, EMLParser, HTMLParser, PDFParser, and PSTParser to allow processing to continue even if some sub-items fail, and to detect total failures.
  • Base64 Decoding: Added a base64::decode function and support for parsing data: URLs in HTML image sources.
  • Named Parameters and Structured Bindings: Introduced named::value and operator""_v literal for creating named parameters, with full support for C++ structured bindings.
  • Non-Null Assertion Utility: Added not_null wrapper for pointer-like types, enforcing non-null invariants based on the configured safety_policy. Includes assume_not_null for unchecked construction when non-null is guaranteed.
  • Ranged Numeric Types: Introduced ranged template for numeric types with compile-time or runtime-checked bounds, along with fluent aliases like at_least, at_most, exactly, and non_negative.
  • Debug-Only Assertions: Added debug_assert for assertions that are only active in debug builds, providing a mechanism for internal contract checking without runtime overhead in release builds.

• Improvements

  • HTML Parser Memory Management: Improved HTML parser by using std::unique_ptr for lxb_html_document_t and validating head text before CSS parsing to prevent potential crashes.
  • PST Parser Robustness: Enhanced PST parser with unique_handle RAII wrappers for libpff and libbfio resources, ensuring exception-safe cleanup. Fixed destruction order of handles and used LIBBFIO_OPEN_READ to prevent accidental file truncation.
  • Logging Enhancements: Refactored log_scope macro and scope class for better control and zero-cost in release builds. Added to_log_value for docwire::named::value to improve structured logging.
  • Error Reporting: make_error_from_tuple and make_error_ptr_from_tuple functions were added for more flexible error creation.
  • Efficient String View Access: Added a string_view() method to data_source for efficient, allocation-free access to data as std::string_view where possible.

• Refactor

  • XML Parsing Integration: All XML-based parsers (CommonXMLDocumentParser, ODFOOXMLParser, ODFXMLParser, XMLParser) have been refactored to utilize the new C++20 XML parsing API.
  • Date/Time Serialization: Removed serialization_time.h as std::chrono types are now directly convertible to/from strings via the new convert framework.
  • rapidfuzz Dependency: Updated the rapidfuzz dependency to rapidfuzz-cpp to align with the upstream vcpkg port rename.
  • XmlStream Removal: The legacy XmlStream class and its associated files (xml_stream.h, xml_stream.cpp) have been removed, replaced by the new xml::reader API.

• Fixes

  • CommonXMLDocumentParser::onODFOOXMLText: Removed a redundant check for the "#text" node name.
  • ODFOOXMLParser::onOOXMLFldData: Added a handler to skip fldData content in OOXML.

• Documentation

  • Updated README.md: The README.md file has been updated to highlight the new "Modern C++20 XML Parsing" and "Configurable Safety Policies" features, including a new example demonstrating the XML parsing API.
  • Doxygen Documentation: Comprehensive Doxygen documentation has been added for the new XML parsing utilities and helper classes.

• Tests

  • New XML API Tests: Added xml_tests.cpp and xml_parsing_example.cpp to thoroughly test the new C++20 XML parsing API.
  • api_tests.cpp Enhancements: Updated api_tests.cpp with new tests for base64::decode, convert from and to std::chrono, named structured bindings, not_null assertions, and PartialAndTotalFailures scenarios for ZIP, EML, PST, HTML, and PDF parsers.
  • New Test Files: Added embedded_images.html and embedded_images.html.out for testing HTML image processing.

2025.10.31

This release introduces a comprehensive overhaul of the SDK's diagnostic and testing infrastructure, centered around a new zero-cost logging framework, a powerful serialization system, and a fluent assertion utility. The error handling mechanism has been significantly enhanced to provide richer, more structured diagnostic messages. This release also modernizes all README.md examples, refactors core components for improved performance and thread-safety, and enhances the CI/CD pipeline.

The old logs fade, a silent, costly past,
New macros rise, expressive and fast.
With sinks and filters, a structured, clear design,
And fluent assertions, that truly align.
A core reforged, robust and so keen,
The finest framework, the best we have seen.
✨🚀🔧

• Features

  • Modern Logging Framework: Replaced the legacy logger with a new zero-cost, structured logging system. It features a sink/filter architecture, JSON output, and expressive macros (log_entry, log_scope, log_forward) that are compiled out in release builds for zero performance overhead.
  • Serialization Framework: Introduced a powerful, concept-based serialization framework to convert C++ types into a structured representation, used by the new logging and error systems.
  • Fluent Assertion Utility: Added docwire::ensure, a zero-cost, exception-throwing assertion utility for writing expressive and robust release-build validations (e.g., ensure(value) > 5;).
  • Safe Environment Variable Access: Introduced docwire::environment::get, a safe and convenient wrapper for accessing environment variables.
  • Endian-Aware Binary Reader: Added binary::reader, a new utility for safely parsing little-endian binary data from any stream-like source.

• Improvements

  • Refactored charset_converter: Overhauled the charset_converter for improved performance, correctness, and thread-safety by serializing iconv_open calls with a global mutex to resolve a data race in glibc.
  • Refactored XLSBParser: The XLSBParser now uses the new binary::reader, resulting in cleaner, more robust, and endian-aware parsing logic.
  • Optimized Test Suite: All API tests now run in a single process, significantly improving test suite performance, especially under sanitizers.
  • CI Enhancements:
    • The CI build matrix has been simplified and made more robust.
    • Conditional execution for OpenAI tests is now supported via the run-tests:openai pull request label, manual dispatch options, and repository-specific schedule triggers.

• Refactor

  • Enhanced Error Handling: The error handling framework has been refactored to use the new serialization system and support multiple context items, providing more detailed and structured diagnostic messages.
  • Type Name Normalization: The type_name utility has been refined to more robustly handle C++ standard library implementation artifacts, improving cross-platform consistency.

• Fixes

  • Fixed a bug where unique_identifier counters could be reset by moving the static member definition to a .cpp file, ensuring a single instance across shared libraries.
  • Corrected a lambda capture issue in pix_cache to capture data_source by reference, improving correctness and performance.
  • Prevented Leptonica error message accumulation by clearing the stderr buffer on each use.

• Documentation

  • Modernized README: Replaced all full code examples in README.md with concise snippets that use the new ensure API for assertions. Each snippet now links to its corresponding full, verifiable source file in the tests/ directory.
  • Updated Logging Section: The logging documentation has been completely rewritten to reflect the new framework's features and usage.

• Tests

  • Log Sanitization: Automated tests now sanitize expected log output to robustly handle differences in function name formatting across compilers and platforms.
  • README Snippet Verification: Added a new automated test to ensure all code snippets in README.md are consistent with their full source examples.

2025.09.25

This release introduces a major architectural overhaul, replacing the std::variant-based Tag system with a type-erased, polymorphic message_ptr. This fundamental change refactors the entire data pipeline—including parsers, exporters, filters, and core components—to emit rich, structured document events, simplifying the data flow and making it easier to add new data types. Alongside this, a new high-performance http::server is introduced, allowing developers to expose any processing pipeline as a standalone HTTP/HTTPS microservice with TLS support.

The pipeline's heart, a message new and bright,
Where docs and mail now dance in polymorphic light.
A server hums, where pipelines run as one,
This grand refactor shines, a new day has begun.
🚀🔧🏗️

• Features

  • HTTP Server: Added a high-performance http::server class with HTTPS/TLS support. This allows any processing pipeline to be easily exposed as a standalone microservice via a path-to-pipeline mapping, enabling powerful and scalable data processing endpoints.

• Improvements

  • HTTP Post Content-Type: The http::Post component now automatically sets the Content-Type header for multipart form data based on the MIME type of the input data_source, simplifying API integrations.
  • Enhanced vcpkg Feature Parsing: The build system's vcpkg feature parsing has been improved to handle more complex, object-based, platform-specific feature definitions.

• Refactor

  • Polymorphic Message System: Replaced the core std::variant-based Tag system with a type-erased, polymorphic message_ptr across the entire data pipeline. This major architectural change affects all parsers, exporters, filters, and core components, enabling them to emit rich, structured document events and simplifying the addition of new data types.
  • Reorganized Element Types: Decoupled and reorganized all document, mail, and AI element types into separate, more logical header files (document_elements.h, mail_elements.h, ai_elements.h, attributes.h) for better code organization.
  • Abstracted XML Parsing Options: Refactored the XML parser to use a dedicated XmlStream::no_blanks struct for configuration, improving clarity over raw integer flags.
  • Unified Transformer API: Updated TransformerFunc and StandardFilter to use the new message_transform_func, aligning custom pipeline elements with the new message-based API.

• Fixes

  • MSVC Compilation: Resolved a compilation failure on MSVC by parenthesizing std::min to avoid conflicts with the Windows min macro and added the /bigobj compiler option to handle heavily templated libraries.
  • RTTI Issues on macOS: Added visibility attributes to core document, mail, and AI elements to fix RTTI-related issues on macOS.

• Build

  • Dependency Change: Replaced curlpp with cpp-httplib and OpenSSL to support the http::Post and http::server components.
  • CI Update: Removed the deprecated macos-13 runner from the GitHub Actions workflow.

• Documentation

  • Updated README.md and all examples to use the new message_ptr API.

• Tests

  • End-to-end HTTP Server Tests: Added new integration tests to validate client-server pipeline processing for both HTTP and HTTPS using the new http::server.
  • API Migration: Migrated the entire test suite to use the new message_ptr API, including the addition of custom GTest matchers for easier and more readable assertions.

2025.08.13

This release marks a major milestone by integrating full support for OpenAI's newly released gpt-5 family of models. To reflect this significant advancement in AI, gpt_5 is now the default model for all OpenAI operations, ensuring users have immediate access to state-of-the-art capabilities. The release also includes minor code quality improvements.

The future's here, no longer a dream,
GPT-5 joins the DocWire stream.
A new default, intelligent and vast,
On foundations of quality, built to last.
✨🤖🚀

• Features

  • New OpenAI Model Support: Added support for the next-generation gpt-5 family of models (gpt-5, gpt_5_mini, gpt_5_nano, gpt_5_chat_latest) and new research-focused models (o3_deep_research, o4_mini_deep_research).

• Improvements

  • Updated Default AI Model: The default model for all OpenAI operations has been updated to gpt_5, leveraging the new state-of-the-art capabilities.
  • Updated XML parser initialization: Remove call to deprecated xmlGetGlobalState() for modern libxml2 compatibility.
  • Robust Model Handling: The OpenAI model selection logic now throws an exception for unknown or unsupported models, improving error reporting.
  • Code Quality: Several single-argument constructors in AI-related chain elements (AnalyzeData, ExtractEntities, Summarize) have been marked explicit to prevent unintended implicit conversions.

• Documentation

  • Updated README: The README.md file has been updated to reflect the new default model and the expanded list of available OpenAI models for both chat and CLI usage. Examples were updated to use non-default models to showcase flexibility.

2025.08.05

This release introduces a major new feature: local, offline AI-powered text embeddings. By integrating the multilingual-e5-small model, DocWire can now generate high-quality vector representations for text in over 100 languages, enabling advanced NLP tasks like semantic search and RAG. This update also includes a significant dependency modernization, replacing OpenNMT-Tokenizer with Google's SentencePiece, and numerous build and stability fixes, particularly for MSVC and Valgrind.

A new dimension, in vectors bright,
Where words find place, and meanings take flight.
With models sharp and logic so keen,
A deeper understanding, clearly seen.
From text to numbers, a seamless art,
DocWire now reads the document's heart.
✨🧠🔢

• Features

  • Local AI Embeddings: Introduced a powerful new local_ai::embed chain element to generate high-quality, multilingual text embeddings using the multilingual-e5-small model. This enables advanced NLP tasks like semantic search, RAG, and text clustering to be performed entirely offline.
  • Cosine Similarity Function: Added a cosine_similarity utility function to calculate the similarity between two embedding vectors, making it easy to compare documents and queries.
  • Public Tokenizer API: The local_ai::tokenizer class, now powered by Google's SentencePiece, is exposed as a public API. It includes an encode() method to convert text into token IDs, supporting different tokenizer models like T5Tokenizer and XLMRobertaTokenizer.

• Improvements

  • Unified Model Runner: The local_ai::model_runner has been enhanced to dynamically load and manage both sequence-to-sequence (Translator) and encoder-only (Encoder) models, enabling it to handle both text generation and embedding tasks within a single class.
  • Advanced Pooling and Normalization: Implemented mean pooling over token outputs and L2 normalization for the embedding model to ensure high-quality, standardized vectors as required by models like E5.
  • Simplified AI Chain Element: Added a new convenience constructor to local_ai::model_chain_element that uses a default model, simplifying its usage in common scenarios.
  • CLI Enhancements: The command-line interface now supports embedding generation via the --local-ai-embed option.

• Refactor

  • Dependency Modernization: Replaced the OpenNMT-Tokenizer dependency with a direct integration of Google's SentencePiece library, reducing complexity and aligning with modern NLP tooling.

• Fixes

  • Build (MSVC): Resolved Address Sanitizer (ASan) linking errors on MSVC by adding _DISABLE_STRING_ANNOTATION and _DISABLE_VECTOR_ANNOTATION definitions.
  • Build (CI): Increased CI timeouts for Valgrind-based sanitizers (memcheck, helgrind, callgrind) to prevent premature job termination.
  • Build (CI): Disabled resource-intensive local AI tests when running under Callgrind to ensure CI stability.
  • Build (Valgrind): Added suppressions for known memory leaks in the Abseil library to clean up Valgrind reports.

• Documentation

  • New Embedding Example: Added a comprehensive example to README.md demonstrating how to generate embeddings for a document and multiple queries, and then calculate their cosine similarity.

• Tests

  • Added unit tests for the new local_ai::tokenizer to validate its behavior with both flan-t5 and multilingual-e5 models.
  • The new local AI embedding example from README.md is now compiled and executed as part of the automated test suite.

2025.07.14

This release introduces a significant new capability by adding support for OpenAI text embeddings. It also features a major refactoring that modularizes the HTTP client into its own library, and enhances the robustness of AI components by switching from file extension checks to MIME type-based content identification.

A whisper of meaning, in numbers cast,
New dimensions open, built to last.
For search and reason, a powerful key,
Unlocking knowledge for all to see.
✨🔢🧠🔍

• Features

  • OpenAI Embeddings Support: Added a new openai::embed chain element to generate text embeddings using OpenAI's models (text-embedding-3-small, text-embedding-3-large, text-embedding-ada-002). This includes corresponding CLI options (--openai-embed, --openai-embed-model) and extensive documentation with examples.
  • MIME Type to File Extension Conversion: Introduced a new utility function, content_type::by_file_extension::to_extension, to convert MIME types back to their common file extensions, improving interoperability.

• Improvements

  • Robust Content Handling in AI Components: AI components like openai::Chat and local_ai::model_chain_element now rely on MIME types (text/plain, image/*) instead of file extensions for more accurate content identification.
  • Enhanced http::Post Robustness: The http::Post component can now automatically determine the file extension from the MIME type if it's not explicitly provided in the data_source.

• Refactor

  • Modular HTTP Client: The http::Post class has been moved from docwire_core to a new, dedicated docwire_http library to improve modularity.

• Fixes

  • Build (Windows): Fixed a build failure on Windows by explicitly adding libxml2 as a direct dependency in the vcpkg port.
  • Documentation: Corrected a minor paragraph formatting issue in README.md.

2025.06.29

This release introduces a major update to the integrated OpenAI functionalities, adding support for the latest models like GPT-4o and retiring older ones. It also enhances dependency management by replacing a custom library port with a standard one, and improves the intelligence of several AI-powered features through better prompting.

The old models fade, a starlit past,
New constellations, brighter and vast.
With prompts refined and logic so keen,
A cleaner build, a powerful scene.
✨🤖🔧

• Features

  • Expanded OpenAI Model Support: Integrated support for the latest generation of OpenAI models, including gpt-4o, gpt-4o-mini, gpt-4.1, and the o3 family. This provides access to state-of-the-art AI capabilities for all relevant features.
  • Granular Model Selection for Transcription and TTS:
    • Users can now select specific models for audio transcription, with new options like gpt-4o-transcribe and gpt-4o-mini-transcribe, in addition to whisper-1.
    • Added new Text-to-Speech models, including gpt-4o-mini-tts, and set it as the new default for higher quality voice generation.

• Improvements

  • Modernized Dependency Management: Replaced the custom unzip vcpkg port with the standard minizip port, simplifying the build process and improving maintainability.
  • Enhanced AI Prompting: Significantly improved the system prompts for the Classify and Find features to yield more precise and consistently formatted results from the language models.
  • Updated Default Model: The default model for most OpenAI operations has been updated to gpt-4o to leverage its improved intelligence, speed, and cost-effectiveness.
  • Robust Example Tests: Updated examples in the documentation to use fuzzy string matching for assertions, making tests more resilient to minor variations in AI-generated text.

• Refactor

  • API Modernization: Removed support for deprecated and legacy OpenAI models (e.g., gpt-3.5-turbo, gpt-4-turbo-preview) to streamline the API and focus on current-generation models.
  • Transcription Component: Refactored the openai::Transcribe component to allow model selection, making it more flexible and future-proof.