Add DuckDuckGo search engine, add extra_deps

[theresnotime]

⚠️ This PR adds a dependency to https://github.com/deedy5/duckduckgo_search

The DuckDuckGo search engine interface returns a [list] of URLs (max. 200), so should be compatible with the copyvios tool — it would be very interesting to compare this to the output of the Google search API.

Future work

• If used, web queries should be proxied (see docs). This PR does not add any proxying of requests.

Notes

• https://help.duckduckgo.com/duckduckgo-help-pages/company/partnerships
• https://help.duckduckgo.com/duckduckgo-help-pages/open-source/opensource-overview/

[A09090091]

Looks good

[earwig]

If I'm understanding this right, that library is scraping DDG's site using an internal endpoint for their JS frontend instead of any sort of public API (see code). DDG used to have a documented API but it no longer functions, as far as I can tell.

There are a few concerns with this:

• It's probably against their TOS. From that now-inaccessible API docs page: "That is, it is not a full search results API or a way to get DuckDuckGo results into your applications beyond our instant answers. Because of the way we generate our search results, we unfortunately do not have the rights to fully syndicate our results, free or paid. For the same reason, we cannot allow framing our results without our branding."
  • If we want to establish a relationship with them, as I think you're suggesting with those links, starting out by violating their terms isn't the best move.
• The internal endpoint could break at any time without warning; i.e., there is no guarantee of future compatibility.

I always understood DDG's to be a wrapper around Bing that provides anonymity and additional higher-level search features that aren't applicable for the copyvio detector, so we really should be looking into Bing instead if we want a Google alternative, and their free tier limits mean we'd most likely need to work something out with the WMF. See my comment here.