Skip to content

v1.7.9

Choose a tag to compare

@stwhwing stwhwing released this 15 Sep 07:44
· 1 commit to main since this release

markitdown-skill v1.7.9

Fixed — silent mojibake on compressed responses

url_fetch.fetch_html() never looked at Content-Encoding, and urllib does not
decompress transparently
(unlike requests). A server answering with gzip therefore had
its compressed bytes decoded as UTF-8 with errors="ignore" — the conversion produced
garbage while still exiting 0.

Reproduced live on both a Windows host and a Linux deployment:

target before after
fetch_html(https://www.python.org/) garbage, 6,259 chars 52,649 chars of clean HTML
url_to_markdown(...) output 4,724 B of junk 14,554 B of readable Markdown

Content-Encoding is now undone first (gzip / deflate, including raw deflate / brotli /
zstd where the optional module is present), then the bytes are decoded using the charset
declared in Content-Type. The response is closed properly via with. Sites that do not
compress are unaffected.

Tests

scripts/tests/test_url_fetch.py grows to 5 cases: gzip, deflate (incl. raw), plain
pass-through, declared charset, and the PinnedHandler regression added in 1.7.8.
The new cases fail against the pre-fix code.