v0.30.0
Added
- Yahoo's Slurp crawler is now a recognized regional (Asia) crawler (#568).
- Added to the
REGIONAL_CRAWLERSallow-list (default-off, opt-in), since Yahoo's global web results are served by Bing (already covered by Bingbot) and Slurp's residual crawling is low-volume. - Discovery stats roll Slurp traffic up under the "Yahoo" brand.
- Added to the