Add searx.locale.get_engine_locale in addition to searx.utils.match_language; improve qwant engine #1652

return42 · 2022-08-10T14:19:19Z

What does this PR do?

Various improvements to the qwant engine

[mod] decouple qwant's categories from SearXNG's categories

By using new property qwant_categ: the category of qwant is no longer bound to the category of SearXNG.
[mod] add locale.get_engine_locale to get predictable results ...

A function that is used where match_languages cannot be used or gives worse results. Needed by the next patch ..
[fix] qwant - API error::locale must be one ..

The request function should not request a language (aka locale) that is not supported by qwant. Select a locale like zh-TW ends in qwant's API error: ..
[mod] qwant - add safesearch option

see Add ability to disable or enable safesearch on Qwant web & image search #1640

Related issues

Closes: #1640

tiekoetter · 2022-08-10T18:08:54Z

@return42
I have a few questions / some comments:

Regarding 84d1635: What is the benefit of decoupling the Qwant engine specifically from the categories?
Regarding 564058f: I understand the need to map the locales between what SearXNG uses and what the specific engine uses. But I can't reproduce what you have mentioned should result in an API error. Searching !qw time :zh-TW using the current master branch does not result in an API error. Could further explain this please?
Regarding 6fe8211: Works as it should 👍

I think you have a typo in the commit message of 564058f The quant's _fetch_supported_languages and 8aka locales).

return42 · 2022-08-10T19:16:01Z

Regarding 84d1635: What is the benefit of decoupling the Qwant engine specifically from the categories?

By example, we have:

searxng/searx/settings.yml

Lines 1183 to 1186 in 43c5459

    
           - name: qwant images 
        
             engine: qwant 
        
             shortcut: qwi 
        
             categories: [images, web]

and you wan't to remove qwant-images from category images and place it in mycateg:

 - name: qwant images 
   engine: qwant 
   shortcut: qwi 
   categories: mycateg

then the qwant-images engine wan't work any longer when you use !mycateg:

searxng/searx/engines/qwant.py

Lines 54 to 55 in 43c5459

    
           category_to_keyword = { 
        
               'general': 'web',

Regarding 564058f: I understand the need to map the locales between what SearXNG uses and what the specific engine uses.

OK

But I can't reproduce what you have mentioned should result in an API error. Searching !qw time :zh-TW using the current master branch does not result in an API error. Could further explain this please?

Thanks for pointing out; now I realize that the language issue is only in the qwant-news engine

!qwn paris :zh-TW

I have to modify the patch .. I will have a look the next days ..

Regarding 6fe8211: Works as it should +1

👍

I think you have a typo in the commit message of 564058f The quant's _fetch_supported_languages and 8aka locales).

Oops . thanks for pointing out .. I will fix it.

searx/engines/qwant.py

tiekoetter · 2022-08-13T20:16:35Z

Regarding 793b570:
I have some minor corrections regarding the commit message. I also must say this is the most detailed commit message which I have ever seen!

  A. When user select a language the results should be optimized according to
     the selected language.

  B. When user select a language and a territory the results should be
     optimized with first priority on terrirtory and second on language.

to

  A. When a user selects a language the results should be optimized according to
     the selected language.

  B. When user selects a language and a territory the results should be
     optimized with first priority on territory and second on language.

Example: (Assumption A.)

  User selects region 'zh-TW' what should end in zh_HK

to

Example: (Assumption A.)

  A user selects region 'zh-TW' which should end in zh_HK

Example: (Assumption A.)

  User selects only the language 'zh' what should end in CN

to

Example: (Assumption A.)

  A user selects only the language 'zh' which should end in CN

Example: (Assumption B.)

  User selects region 'fr-BE' what should end in nl-BE

hint:
  priority should be first on territory the user selected.  If user
  prefer 'fr' he will select 'fr' without a region tag.

to

Example: (Assumption B.)

  A user selects region 'fr-BE' which should end in nl-BE

hint:
  priority should be on the territory the user selected.  If the user
  prefers 'fr' he will select 'fr' without a region tag.

Example: (Assumption A.)

  User selects only the language 'fr' what should end in fr_CA

to

Example: (Assumption A.)

  A user selects only the language 'fr' which should end in fr_CA

The difference of priotity on terrirtory is best shown with a engine that
supports the follwoing locales:

to

The difference in priority on the territory is best shown with a engine that
supports the following locales:

Example: (Assumption A.)

   User selects only a language

to

Example: (Assumption A.)

   A user selects only a language

hint: the engine supports fr_FR and fr_CA sine no terrirtory is given, fr_FR
takes priority ..

to

hint: the engine supports fr_FR and fr_CA since no territory is given, fr_FR
takes priority ..

Example: (Assumption B.)

  User selects region 'fr-BE' what should end in nl-BE

to

Example: (Assumption B.)

  A user selects region 'fr-BE' which should end in nl-BE

Lokks both function return the same value, but match_language depends oon the
order of the dictionary (what is not predictable):

to

Looks like both functions return the same value, but match_language depends on the
order of the dictionary (which is not predictable):

tiekoetter

LGTM
If #1652 (comment) is addressed

By using new property `qwant_categ:` the category of qwant is no longer bound to the category of SearXNG. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

The match_language function sometimes returns incorrect results which is why a new function get_engine_locale is required. A bugfix of the match_language is not easily possible, because there is almost no documentation for it and already the call parameters are undefined. E.g. the function processes values like the ones from yahoo:: "yahoo": [ "ar", ... "zh_chs", "zh_cht" ] The get_engine_locale has been documented in detail, there is a clear description of the assumptions as well as the requirements and approximation rules (read doc-string for more details):: Argument ``engine_locales`` is a python dict that maps *SearXNG locales* to corresponding *engine locales*: <engine>: { # SearXNG string : engine-string 'ca-ES' : 'ca_ES', 'fr-BE' : 'fr_BE', 'fr-CA' : 'fr_CA', 'fr-CH' : 'fr_CH', 'fr' : 'fr_FR', ... 'pl-PL' : 'pl_PL', 'pt-PT' : 'pt_PT' } .. hint:: The *SearXNG locale* string has to be known by babel! In the following you will find a comparison: >>> import babel.languages >>> from searx.utils import match_language >>> from searx.locales import get_engine_locale Assume we have an engine that supports the follwoing locales: >>> lang_list = { ... "zh-CN": "zh_CN", ... "zh-HK": "zh_HK", ... "nl-BE": "nl_BE", ... "fr-CA": "fr_CA", ... } Assumption: A. When a user selects a language the results should be optimized according to the selected language. B. When user selects a language and a territory the results should be optimized with first priority on territory and second on language. ---- Example: (Assumption A.) A user selects region 'zh-TW' which should end in zh_HK hint: CN is 'Hans' and HK ('Hant') fits better to TW ('Hant') >>> get_engine_locale('zh-TW', lang_list) 'zh_HK' >>> lang_list[match_language('zh-TW', lang_list)] 'zh_CN' ---- Example: (Assumption A.) A user selects only the language 'zh' which should end in CN >>> get_engine_locale('zh', lang_list) 'zh_CN' >>> lang_list[match_language('zh', lang_list)] 'zh_CN' ---- Example: (Assumption B.) A user selects region 'fr-BE' which should end in nl-BE hint: priority should be on the territory the user selected. If the user prefers 'fr' he will select 'fr' without a region tag. >>> get_engine_locale('fr-BE', lang_list, default='unknown') 'nl_BE' >>> match_language('fr-BE', lang_list, fallback='unknown') 'fr-CA' ---- Example: (Assumption A.) A user selects only the language 'fr' which should end in fr_CA >>> get_engine_locale('fr', lang_list) 'fr_CA' >>> lang_list[match_language('fr', lang_list)] 'fr_CA' ---- The difference in priority on the territory is best shown with a engine that supports the following locales: >>> lang_list = { ... "fr-FR": "fr_FR", ... "fr-CA": "fr_CA", ... "en-GB": "en_GB", ... "nl-BE": "nl_BE", ... } ---- Example: (Assumption A.) A user selects only a language >>> get_engine_locale('en', lang_list) 'en_GB' >>> match_language('en', lang_list) 'en-GB' hint: the engine supports fr_FR and fr_CA since no territory is given, fr_FR takes priority .. >>> get_engine_locale('fr', lang_list) 'fr_FR' >>> lang_list[match_language('fr', lang_list)] 'fr_FR' ---- Example: (Assumption B.) A user selects region 'fr-BE' which should end in nl-BE >>> get_engine_locale('fr-BE', lang_list) 'nl_BE' >>> lang_list[match_language('fr-BE', lang_list)] 'fr_FR' ---- If the user selects a language and there are two locales like the following: >>> lang_list = { ... "fr-BE": "fr_BE", ... "fr-CH": "fr_CH", ... } >>> >>> get_engine_locale('fr', lang_list) 'fr_BE' >>> lang_list[match_language('fr', lang_list)] 'fr_BE' Looks like both functions return the same value, but match_language depends on the order of the dictionary (which is not predictable): >>> lang_list = { ... "fr-CH": "fr_CH", ... "fr-BE": "fr_BE", ... } >>> get_engine_locale('fr', lang_list) 'fr_BE' >>> lang_list[match_language('fr', lang_list)] 'fr_CH' >>> The get_engine_locale selects the locale by looking at the "population percent" and this percentage has an higher amount in BE (68.%) compared to CH (21%) Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

The request function should not request a language (aka locale) that is not supported by qwant. Select a locale like zh-TW ends in qwant's API error: ERROR searx.engines.qwant news: exception : \ API error::locale must be one of the following values: \ en_gb, en_ie, en_us, en_ca, en_my, en_au, en_nz, de_de, de_ch, de_at, fr_fr, \ fr_be, fr_ch, fr_ca, fr_ad, fc_ca, co_fr, es_es, es_ar, es_cl, es_co, es_mx, \ es_pe, es_ad, ca_es, ca_ad, ca_fr, eu_es, eu_fr, it_it, it_ch, pt_pt, pt_ad, \ nl_be, nl_nl The existing searx.utils.match_language function is unsuitable for this purpose, it is replaced by function searx.locales.get_engine_locale that is based on the methods from the babel package. The quant's _fetch_supported_languages function has been revised to filter out languages 8aka locales) not supported by qwant. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

Closes: searxng#1640 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 · 2022-08-14T08:58:31Z

If #1652 (comment) is addressed

I fixed the commit message .. thanks for reading with care 👍 👍

I also must say this is the most detailed commit message which I have ever seen!

😆

TL;DR;

I prefer git comments to document a change process. Most people prefer short git comments and left the detailed discussions of the changes in the issues.

I think the collaboration tools of git hosters are valuable for community collaboration. However, the insights gained from the exchange should be additional and directly readable in the commit messages of the changes.

That's why I take the liberty to re-worden the commits and to take passages from the posts to the PR.

The requirement or its implementation (such as a function or class description) belongs in my opinion directly in the source code, as close as possible to the place of implementation. In python code, we can use comments or better doc-strings.

I think the git hosts like gitlab, githost and many others (e.g. gitea instances) are excellent tools to promote collaboration in a community. However, in most cases an account is required and personalization of activities takes place. What is done with the personalized data is usually not transparent and at least the question remains, what motivation drives a top dog like github.

From a privacy point of view, one could, for example, be critical of our hoster (github). However, the choice of hoster is usually more dependent on the question of which hoster the majority of a community will be found on.

This patch replaces the *full of magic* ``utils.match_language`` function by a ``locales.match_locale``. The ``locales.match_locale`` function is based on the ``locales.build_engine_locales`` introduced in 9ae409a [1]. In the past SearXNG did only support a search by a language but not in a region. This has been changed a long time ago and regions have been added to SearXNG core but not to the engines. The ``utils.match_language`` was the function to handle the different aspects of language/regions in SearXNG core and the supported *languages* in the engine. The ``utils.match_language`` did it with some magic and works good for most use cases but files in some edge case. To replace the concurrence of languages and regions in the SearXNG core the ``locales.build_engine_locales`` was introduced in 9ae409a [1]. With the last patches all engines has been migrated to a ``fetch_traits`` and a language/region concept that is based on ``locales.build_engine_locales``. To summarize: there is no longer a need for the ``locales.match_language``. [1] searxng#1652 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

This patch replaces the *full of magic* ``utils.match_language`` function by a ``locales.match_locale``. The ``locales.match_locale`` function is based on the ``locales.build_engine_locales`` introduced in 9ae409a [1]. In the past SearXNG did only support a search by a language but not in a region. This has been changed a long time ago and regions have been added to SearXNG core but not to the engines. The ``utils.match_language`` was the function to handle the different aspects of language/regions in SearXNG core and the supported *languages* in the engine. The ``utils.match_language`` did it with some magic and works good for most use cases but fails in some edge case. To replace the concurrence of languages and regions in the SearXNG core the ``locales.build_engine_locales`` was introduced in 9ae409a [1]. With the last patches all engines has been migrated to a ``fetch_traits`` and a language/region concept that is based on ``locales.build_engine_locales``. To summarize: there is no longer a need for the ``locales.match_language``. [1] searxng#1652 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 mentioned this pull request Aug 10, 2022

Add ability to disable or enable safesearch on Qwant web & image search #1640

Closed

return42 requested a review from tiekoetter August 10, 2022 15:26

return42 force-pushed the mod-qwant branch from 6fe8211 to 5ab6f1c Compare August 12, 2022 16:41

return42 commented Aug 12, 2022

View reviewed changes

searx/engines/qwant.py Show resolved Hide resolved

tiekoetter approved these changes Aug 13, 2022

View reviewed changes

return42 added 4 commits August 14, 2022 10:26

[mod] decouple qwant's categories from SearXNG's categories

75bb8c4

By using new property `qwant_categ:` the category of qwant is no longer bound to the category of SearXNG. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

[mod] qwant - add safesearch option

27385e7

Closes: searxng#1640 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

return42 force-pushed the mod-qwant branch from 5ab6f1c to 27385e7 Compare August 14, 2022 08:37

return42 merged commit 2bd3c20 into searxng:master Aug 14, 2022

return42 deleted the mod-qwant branch August 14, 2022 08:58

dalf changed the title ~~Improve qwant engine~~ Add searx.locale.get_engine_locale in addition to searx.utils.replace match_language; improve qwant engine Aug 14, 2022

dalf changed the title ~~Add searx.locale.get_engine_locale in addition to searx.utils.replace match_language; improve qwant engine~~ Add searx.locale.get_engine_locale in addition to searx.utils.match_language; improve qwant engine Aug 14, 2022

This was referenced Aug 14, 2022

[fix] harden get_engine_locale: handle UnknownLocaleError exceptions #1664

Merged

[fix] typo in get_engine_locale #1666

Merged

return42 mentioned this pull request Feb 7, 2023

[mod] replace utils.match_language by locales.match_locale return42/searxng#186

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add searx.locale.get_engine_locale in addition to searx.utils.match_language; improve qwant engine #1652

Add searx.locale.get_engine_locale in addition to searx.utils.match_language; improve qwant engine #1652

return42 commented Aug 10, 2022 •

edited

tiekoetter commented Aug 10, 2022 •

edited

return42 commented Aug 10, 2022

tiekoetter commented Aug 13, 2022

tiekoetter left a comment

return42 commented Aug 14, 2022 •

edited

Add searx.locale.get_engine_locale in addition to searx.utils.match_language; improve qwant engine #1652

Add searx.locale.get_engine_locale in addition to searx.utils.match_language; improve qwant engine #1652

Conversation

return42 commented Aug 10, 2022 • edited

What does this PR do?

Related issues

tiekoetter commented Aug 10, 2022 • edited

return42 commented Aug 10, 2022

tiekoetter commented Aug 13, 2022

tiekoetter left a comment

Choose a reason for hiding this comment

return42 commented Aug 14, 2022 • edited

return42 commented Aug 10, 2022 •

edited

tiekoetter commented Aug 10, 2022 •

edited

return42 commented Aug 14, 2022 •

edited