v0.4.2

Fixed

Text files are decoded by their actual encoding instead of being read as UTF-8.

Every text reader opened files with errors="ignore", which never fails on a non-UTF-8 file — it silently corrupts it. This mattered most on Windows, because it is what Notepad writes:

Notepad "Save as"	Encoding	What RAGMill stored
Unicode	UTF-16 LE	T\x00h\x00e\x00 \x00c\x00o\x00m\x00p…
Unicode big endian	UTF-16 BE	same
ANSI	cp1252	Café — naïve “quoted” résumé costs £50 → Caf nave quoted rsum costs 50
UTF-8 with BOM	UTF-8 + BOM	stray  leading the first chunk

The UTF-16 cases ingested without any error, but the text embedded as noise and never matched a query — a .txt file that reported success and then returned nothing. The ANSI case dropped characters outright: costs £50 became costs 50, a change of meaning with no warning.

Encoding is now detected: byte-order marks are honoured (UTF-8/16/32, with the BOM stripped rather than left in the text), BOM-less UTF-16 is caught via interior NULs, and anything else falls back to cp1252 then latin-1 with a warning naming the encoding used. This covers every text reader — plain text (.txt, .md, .log, .rst), .csv/.tsv, .html, and .rtf — each of which carried its own copy of the flaw.

If a .txt file appeared to ingest but never turned up in search results, this was why.

Upgrading

pip install --upgrade "ragmill[all]"

Documents ingested before this release were stored with corrupted text if they were not UTF-8. Re-ingest them to pick up the correct content.

No API changes.