feat: add lossy UTF-8 reading variants and deprecate strict APIs

[digizeph]

Summary

Fixes #72 — read_lines and read_to_string fail on real-world data containing non-UTF-8 bytes (e.g., Latin-1 in RIPE IRR databases). Adds lossy reading variants that replace invalid sequences with U+FFFD and continue processing instead of aborting mid-stream.

Changes

New APIs

• read_lines_lossy / OneIo::read_lines_lossy — line iterator with lossy UTF-8
• read_to_string_lossy / OneIo::read_to_string_lossy — full-file lossy read
• read_to_bytes / OneIo::read_to_bytes — byte-perfect Vec<u8> read
• read_to_string_lossy_async / read_to_bytes_async — async variants
• OneIo::to_lines_lossy(reader) — convert any reader to lossy line iterator

Deprecated (still functional)

• read_lines → use read_lines_lossy
• read_to_string → use read_to_string_lossy or read_to_bytes
• read_to_string_async → use read_to_string_lossy_async or read_to_bytes_async

CLI

• Defaults to lossy UTF-8 reading (no longer exits on invalid bytes)
• New --strict-utf8 flag for the old strict behavior

Library compatibility

No breaking API changes. Existing functions are deprecated with #[deprecated] but retain identical signatures and behavior. Callers can migrate at their own pace.

Testing

24 new tests covering:

• Latin-1 lossy replacement (0xf3 → U+FFFD)
• CRLF, bare CR, no-trailing-newline handling
• Continuation past bad bytes (the original bug)
• Byte-perfect round-trip via read_to_bytes
• Legacy strict APIs still fail as expected
• Send trait for cross-thread usage

Related

• Spec: specs/02-lossy-read/README.md
• Issue: read_lines fails on non-UTF-8 input — consider a lossy variant #72

[Copilot]

Pull request overview

Adds lossy UTF-8 read APIs to OneIO so callers can keep processing real-world inputs containing invalid UTF-8 bytes, while keeping the existing strict APIs available (but deprecated). Updates documentation, examples, CLI behavior, and tests to prefer the new lossy/default-safe paths.

Changes:

• Added new lossy text APIs (read_lines_lossy, read_to_string_lossy) plus byte-perfect helpers (read_to_bytes) with async counterparts.
• Deprecated strict UTF-8 APIs (read_lines, read_to_string, read_to_string_async) and migrated docs/examples/tests to the new defaults.
• Updated the CLI to default to lossy UTF-8 and added a --strict-utf8 flag for the legacy strict behavior.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
tests/lossy_read_tests.rs	Adds a broad test suite for lossy line iteration and byte-perfect reads.
tests/basic_integration.rs	Switches baseline integration coverage to lossy APIs.
src/lib.rs	Adds crate-level lossy/bytes APIs and deprecates strict wrappers; updates docs usage.
src/client.rs	Implements lossy line iterator + new OneIo lossy/bytes methods; deprecates strict methods.
src/bin/oneio.rs	CLI defaults to lossy line reading and introduces --strict-utf8.
src/async_reader.rs	Adds async lossy string and byte-perfect read helpers; deprecates strict async string read.
specs/README.md	Adds the lossy-read spec to the spec index.
specs/02-lossy-read/README.md	New design spec documenting rationale, API plan, and testing strategy.
README.md	Updates README examples to use lossy APIs.
examples/test_crypto_provider.rs	Updates example to use lossy string read.
examples/async_read.rs	Updates async example to use lossy async string read.
CHANGELOG.md	Records new APIs, CLI behavior change, and deprecations under Unreleased.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

tests/async_integration.rs:106

• This test also uses a fixed filename (tests/_tmp_async_bytes.txt), which can collide under parallel test execution and lead to flakiness. Use a per-test unique temp path (or a shared helper that creates unique temp files) instead of a constant path under tests/.

    let tmp_path = "tests/_tmp_async_bytes.txt";
    let _ = std::fs::remove_file(tmp_path);
    let expected = b"valid\nbad: \xf3\nnext\n";
    std::fs::write(tmp_path, expected).unwrap();

    let bytes = oneio::read_to_bytes_async(tmp_path).await.unwrap();
    assert_eq!(bytes, expected);