Version 0.9.6.1 - Security fix: SSRF protection for outbound requests

Highlights

• Security fix (CWE-918, SSRF): Article.parse() fetches images by default, and it took image URLs straight from the page (<img src>, og:image). The only validation was a URL scheme check. A page an attacker controls or influences could therefore make the server request internal addresses such as 127.0.0.1, private ranges, or cloud metadata endpoints like 169.254.169.254. Versions up to and including 0.9.6 are affected.
• Outbound requests are now checked: every request made through do_request is rejected with PrivateHostError unless the host resolves to a public IP. Redirect targets are checked at every hop, so a public URL can't bounce to an internal host. This covers read_more_link targets, meta-refresh targets, and Source category, feed and article URLs.
• Image fetching: image downloads, which don't go through do_request, have their own check and follow redirects by hand with a check on each hop.
• New option allow_private_hosts (default False): set it to True to fetch private or internal hosts again.
• NLP requires nltk >=3.10: the Punkt tokenizer in nltk 3.10 splits sentences after closing quotes, which changes the generated summaries.

Breaking changes

• Requests to localhost, private IP ranges, or internal hostnames now fail unless allow_private_hosts=True. This includes article URLs you pass in yourself.
• The nlp extra now needs nltk >=3.10. Summaries can differ from earlier versions because of the sentence-splitting change.
• nltk 3.10 rejects an nltk_data directory that is writable by another account. Point NLTK_DATA at a directory you own if you hit a LookupError for punkt_tab.

Upgrade advice

Anyone using newspaper4k to parse pages from untrusted or user-supplied sources should upgrade. If you must stay on an older version, set fetch_images=False to disable image fetching and validate user-supplied URLs yourself.

Known limitations

DNS rebinding between the host check and the connection is not prevented. For hard guarantees, also restrict outbound traffic at the network level.

What's Changed

• Block SSRF in image extraction, meta refresh, read_more_link and Source URL handling, with allow_private_hosts to opt out
• Add PrivateHostError and network.is_public_url()
• Update NLP test fixtures for nltk 3.10 Punkt and require nltk >=3.10

Full Changelog: 0.9.6...0.9.7