v0.5.1

🌍 Support for 150 Publishers & New Language-Based Search and Corpus Controls 🚀

With this release, Fundus now supports 150 publishers across 30 countries, thanks to the addition of 14 new regions and 24 new publishers!

✨ New Features

As our coverage grows, so does the need for better language and data management—so we’ve introduced two powerful new features:

🔎 Language-Based Publisher Search

You can now filter publishers based on the languages they support. This makes it easier to target specific linguistic corpora or build multilingual datasets.

from fundus import Crawler, PublisherCollection

# Find publishers that support Japanese
filtered_publishers = PublisherCollection.search(languages=["ja"])

# US-based publishers that also offer Spanish content
filtered_publishers = PublisherCollection.us.search(languages=["es"])

crawler = Crawler(*filtered_publishers)
for article in crawler.crawl():
    print(article)

• Add search by language functionality by @addie9800 in #667

🧮 Balanced Article Crawling

You can now cap the number of articles per publisher during crawling using the new max_articles_per_publisher parameter—ideal for creating balanced datasets.

from fundus import Crawler, PublisherCollection

crawler = Crawler(PublisherCollection.us)

for article in crawler.crawl(max_articles_per_publisher=10, save_to_file="my_corpus.json"):
    print(article)

• Add max_articles_per_publisher parameter to crawl by @MaxDall in #710

Check out our documentation for more details!

Publishers

This update brings 14 new regions and 24 additional publishers, pushing our total to 150 supported publishers!

Added Regions

• Add PL by @addie9800 in #698
• Add PT by @addie9800 in #699
• Add CZ by @horychtom in #725
• Add MX + minor bug fixes by @addie9800 in #734
• Add GL by @addie9800 in #735
• Add ISL by @addie9800 in #736
• Add IL by @addie9800 in #737
• Add PY by @addie9800 in #741
• Add RU by @addie9800 in #757
• Add KR by @addie9800 in #758
• Add KR with MBN by @zxxxv in #765
• Add ZA by @addie9800 in #760
• Add LS by @addie9800 in #762
• Add LU by @addie9800 in #775
• Add LI by @addie9800 in #777

Added Publishers

• Added turkish publisher Anadolu Ajansı by @MSDuran in #722
• Add Tageszeitung by @addie9800 in #738
• Add MallorcaMagazin by @addie9800 in #739
• Add MallorcaZeitung by @addie9800 in #740
• Add DailyMaverick by @addie9800 in #761
• Add LuxemburgerWort by @addie9800 in #776
• Add Spanish publishers by @Finiluh in #768
• Add SalzburgerNachrichten by @addie9800 in #770
• Add DiePresse by @addie9800 in #771
• Add KleineZeitung by @addie9800 in #778

Updated Publishers

• add url_filter to voa by @addie9800 in #715
• add url_filter and RSSFeeds by @addie9800 in #716
• Update BusinessInsider by @addie9800 in #717
• Update author extraction for JyllandsPosten by @addie9800 in #718
• Fix Focus sources by @MaxDall in #732
• Update FAZ parser to version V3 by @MaxDall in #733
• Adjust ZDFParser to be more suitable for live tickers by @MaxDall in #747
• Modify BBC selectors by @addie9800 in #749
• Update RSSFeed for Bild by @addie9800 in #752
• Add V1_1 for NationalPost by @addie9800 in #754
• Update Sitemaps for Tanzania by @addie9800 in #755
• Update _paragraph_selector for JyllandsPosten by @addie9800 in #756
• Update Tanzanian publishers by @addie9800 in #766
• Change name of MBN by @addie9800 in #769
• Add V1_1 for NDR by @addie9800 in #773
• Update Nieuwsblad by @addie9800 in #780

Deprecated Publishers

• Deprecate TheTelegraph by @MaxDall in #711
• Deprecate Nikkei by @addie9800 in #767
• Deprecate TheNamibian by @addie9800 in #779

Bug Fixes & Stability

• Set allow_all=True when robots cannot be loaded by @MaxDall in #709
• Add max_articles_per_publisher parameter to crawl by @MaxDall in #710
• Extend timeout in publisher coverage by @addie9800 in #712
• Properly release resources by @MaxDall in #713
• Docs: Fix date_filter example by @dallasbrittany in #714
• Bug Fixes - Events by @addie9800 in #719
• Register default stop event for WebSource by @MaxDall in #721
• Make network connections interruptible by @MaxDall in #723
• Rework language attribution by @MaxDall in #726
• Make lang attribute deterministic by @MaxDall in #742
• Bug Fix in Source Restriction by @addie9800 in #746
• Bug Fixes from Publisher Coverage by @addie9800 in #753
• Add logging for source restriction by @addie9800 in #774
• Remove duplicate entries in PublisherCollection after merge of #757 by @MaxDall in #781
• Remove Whitespace Normalization in image source parsing by @addie9800 in #692

Cleanup & Maintenance

• Remove leftover ANADOLUAJANSI.json by @MaxDall in #727
• Remove unused imports by @MaxDall in #729
• Update publisher_coverage.yaml by @addie9800 in #750
• Update publisher_coverage.yaml by @addie9800 in #751

Testing

• Add unit test if default_language is ISO 639 language code by @MaxDall in #744

New Contributors

• @dallasbrittany made their first contribution in #714
• @MSDuran made their first contribution in #722
• @horychtom made their first contribution in #725
• @zxxxv made their first contribution in #765
• @Finiluh made their first contribution in #768

Full Changelog: v0.5.0...v0.5.1