Problem with lobste.rs (and possibly other search engines): timeout

[x-0n]

Issue:

When running a search, lobste.rs fails for me. In the results page, I get notified that

"Engines cannot retrieve results: lobste.rs (timeout)".

in /var/log/uwsgi/uwsgi.log it looks like this:

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (2): lobste.rs:443
WARNING:searx.search:engine timeout: lobste.rs
ERROR:searx.search:engine lobste.rs : HTTP requests timeout(search duration : 8.756943225860596 s, timeout: 8.0 s) : ReadTimeout

In my settings.yml, I have the following settings:

  - name : lobste.rs
    engine : xpath
    search_url : https://lobste.rs/search?utf8=%E2%9C%93&q={query}&what=stories&order=relevance
    results_xpath : //li[contains(@class, "story")]
    url_xpath : .//span[@class="link"]/a/@href
    title_xpath : .//span[@class="link"]/a
    content_xpath : .//a[@class="domain"]
    categories : it
    shortcut : lo
    timeout : 10.0

But checking prefs from web UI looks like this:

Expected result

• lobste.rs should respect the timeout set in settings.yml and thereby not overrun the max timeout

Steps to reproduce

• Activate lobste.rs in prefs
• run any search query

Version

• docker image searx/searx:latest (0.15.0-186-42d5e2c0)

[return42]

No need to activate lobste.rs in the prefs, just use !lo foo. With I can confirm the behavior see: !lo foo

[return42]

I tested from different nodes from DE, never got response under 6sec.

curl https://lobste.rs/search?q=foo >/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 37971    0 37971    0     0   6027      0 --:--:--  0:00:06 --:--:--  8254

Set timeout to 10sec and the got error

File "/share/searx/searx/engines/xpath.py", line 53, in extract_url
    raise Exception('Empty url resultset')

fixed url_xpath (see below) helped, but results_xpath returns on some elements \n which cause the next exception.

I will stop here, maybe there's someone who wants to pick up the baton.

modified   searx/settings.yml
@@ -421,11 +421,13 @@ engines:
     engine : xpath
     search_url : https://lobste.rs/search?utf8=%E2%9C%93&q={query}&what=stories&order=relevance
     results_xpath : //li[contains(@class, "story")]
-    url_xpath : .//span[@class="link"]/a/@href
+    url_xpath : .//span[contains(concat(' ', @class, ' '), ' link ')]/a/@href
     title_xpath : .//span[@class="link"]/a
     content_xpath : .//a[@class="domain"]
     categories : it
     shortcut : lo
+    timeout: 10
+    disabled: True
 
   - name : microsoft academic
     engine : microsoft_academic

[unixfox]

@return42 why don't you push your commit into the master branch of searx?

[return42]

because it is only a partial bugfix, not the solution .. read again ..

fixed url_xpath helped, but results_xpath returns on some elements \n which cause the next exception.

[kvch]

I could not reproduce the timeout problem you are describing. Whenever I changed the timeout and restarted searx, the UI displayed the correct value and the search was either done or not depending on the setting.

I have submitted a PR with an XPATH fix which closes the issue. @x-0n if the timeout problem still persists, feel free to reopen this issue.