Suddenly stopped working #115

kurtericmiller · 2020-06-30T20:40:51Z

Greetings,
Can ping duckduckgo.com (no replies but contactable)
Can use duckduckgo.com from chrome

some debug output:
PING google.com (172.217.20.110) 56(84) bytes of data.
64 bytes from ams17s01-in-f14.1e100.net (172.217.20.110): icmp_seq=1 ttl=117 time=39.3 ms
64 bytes from ams17s01-in-f14.1e100.net (172.217.20.110): icmp_seq=2 ttl=117 time=38.9 ms
64 bytes from ams17s01-in-f14.1e100.net (172.217.20.110): icmp_seq=3 ttl=117 time=39.2 ms
64 bytes from ams17s01-in-f14.1e100.net (172.217.20.110): icmp_seq=4 ttl=117 time=38.8 ms

--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 38.834/39.043/39.297/0.202 ms
[kmiller@KEM-B9]~:ping -c4 duckduckgo.com
PING duckduckgo.com (40.114.177.156) 56(84) bytes of data.

--- duckduckgo.com ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3061ms

[kmiller@KEM-B9]~:ddgr -d ubuntu 20.04
[DEBUG] ddgr version 1.8.1 Python version 3.8.2
[DEBUG] q:ubuntu 20.04, region:us-en, page:0, curindex:1, safe:1
[DEBUG] nextParams:
[DEBUG] proxy:{'https': None}
[DEBUG] ua:ddgr/1.8.1 (textmode; Linux x86_64; 1024x768)
[DEBUG] Response body written to '/tmp/ddgr-response-tixs8svt'.
[DEBUG] Prev nextParams:
[DEBUG] Next nextParams:
No results.
ddgr (? for help)

cat /tmp/ddgr-response-tixs8svt

<title>DuckDuckGo</title>

About DuckDuckGo

      <input name="q" autocomplete="off" id="search_form_input_homepage" class="search__input" type="text" autofocus />

      <input name="b" id="search_button_homepage" class="search__button search__button--html" value="" title="Search" alt="Search" type="submit" />

  
  
  
  
  

    </form>
</div>

<!--[if IE 6]>
    <p style="text-align:center;">This browser is not supported. Please visit <a href="/lite">/lite</a>.</p>
<![endif]-->

<div id="html-only-notice">

</div>

The text was updated successfully, but these errors were encountered:

jarun · 2020-07-01T00:49:34Z

There is some change in the service. i can fetch the first page with the following patch:

diff --git a/ddgr b/ddgr
index 39510e6..647718e 100755
--- a/ddgr
+++ b/ddgr
@@ -705,7 +705,7 @@ class DdgConnection:
     """
 
     def __init__(self, proxy=None, ua=''):
-        self._u = 'https://duckduckgo.com/html'
+        self._u = 'https://html.duckduckgo.com/html'
 
         self._proxies = {
             'https': proxy if proxy is not None else (os.getenv('https_proxy')

However, fetching the next page still fails and I am not sure why. Please see if you can debug and figure this out. I can't spend time on this.

jarun · 2020-07-01T00:54:28Z

Other things that would fail are bangs, tab-completion, ~~instant results~~.

jetzerb · 2020-07-05T21:10:30Z

Yes, it must be due to a change in DDG's API. I get the same "No Results" message from the version packaged for Ubuntu 20.04

$ dpkg -l ddgr
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version                   Architecture Description
+++-==============-=========================-============-=================================
ii  ddgr           1.7+git20190928.bccdc92-2 all          DuckDuckGo from the terminal

$ ddgr -d Ubuntu 20.04
[DEBUG] ddgr version 1.7 Python version 3.8.2
[DEBUG] q:Ubuntu 20.04, region:us-en, page:0, curindex:1, safe:1
[DEBUG] nextParams:
[DEBUG] proxy:{'https': None}
[DEBUG] ua:ddgr/1.7 (textmode; Linux x86_64; 1024x768)
[DEBUG] Response body written to '/tmp/ddgr-response-sxfn32q0'.
[DEBUG] Prev nextParams: 
[DEBUG] Next nextParams: 
No results.
ddgr (? for help)

NOTE: When I run the latest code that includes the update referenced above, I have no problem navigating to the next or previous page, and bangs seem to work fine.

aalapd · 2020-07-06T12:45:38Z

Same problem here.
$ ddgr hello
No results.
ddgr (? for help)

brandong954 · 2020-07-06T19:04:26Z

Same here

jarun · 2020-07-07T01:03:51Z

Guys, I know it's there. Unfortunately, I do not have the time to debug and fix every aspect of it.
Someone has to pitch in and fix fetching next pages, bangs, auto-completion on tab.

jetzerb · 2020-07-07T14:35:42Z

@jarun: Could you expand on the troubles you're experiencing? I just tried a couple things now, using a freshly-cloned ddgr repo:

Next/Prev pages Sometimes work, but the responses coming back from DDG seem strange. The number of results varies. I imagine this is why n/p seems to work sometimes but not others:

$ rm /tmp/ddgr-response-*
$ for i in $(seq 10); do ddgr --np --debug microsoft; done >/dev/null 2>&1
$ grep -c 'result__a' /tmp/ddgr-response-*
/tmp/ddgr-response-0k1wqttu:19
/tmp/ddgr-response-39i1ed80:21
/tmp/ddgr-response-6461a4gc:21
/tmp/ddgr-response-b99mdhdr:21
/tmp/ddgr-response-ca181t92:20
/tmp/ddgr-response-dnwhc41t:21
/tmp/ddgr-response-eubap2vm:21
/tmp/ddgr-response-irh29qd7:21
/tmp/ddgr-response-uth25bvw:30
/tmp/ddgr-response-zgyjbck9:21

Bangs are working fine for me, both on the commandline (ddgr '!w Galilean moons') and at the ddgr interactive menu.
auto-completion is also working for me on Ubuntu 20.04 running Bash:

$ ddgr --
--colors             --help               --num                --unsafe
--debug              --json               --proxy              --url-handler
--ducky              --nocolor            --reg                --version
--expand             --noprompt           --show-browser-logs  
--gb                 --noua               --site               
--gui-browser        --np                 --time               
$ ddgr --d
--debug  --ducky  
$ ddgr --ducky

(that last command auto-completed after I entered --du and hit tab)

jarun · 2020-07-07T15:07:10Z

Thanks for checking it out. In that case I think the only issue is next and prev. You can compare the data sent from firefox (for next) in developer mode with what ddgr sends. Please see if that gives a hint.

jarun · 2020-07-07T15:35:41Z

Sorry, I didn't mean option completion by auto-completion. in ddgr you could also complete possible search terms earlier. That is not working.

Bangs are also not working. Try the 4th example: https://github.com/jarun/ddgr#examples

jetzerb · 2020-07-07T15:56:28Z

I wasn't aware ddgr had search-term auto-completion. How did that work? Just type (a portion of) a word and hit tab? Where? On the commandline? At the interactive prompt?

Regarding bangs: I'm running inside a Linux Docker container without a GUI browser so I can't test your 4th example exactly, but both ddgr \!w hello world and ddgr '!w hello world' work for me. Lynx opens and displays the "Hello World!" page:

#
"Hello, World!" program - Wikipedia (p1 of 7)
   #Edit this page Wikipedia (en) Wikipedia Atom feed

"Hello, World!" program

   From Wikipedia, the free encyclopedia
     (Redirected from Hello world)
   Jump to navigation Jump to search
   "Hello World" redirects here. For other uses, see Hello World (disambiguation).
   Traditional beginners' computer program
. . . snip . . .

jarun · 2020-07-07T16:07:53Z

You are right! Bangs are working. I am on zsh and missed the \.

Just type (a portion of) a word and hit tab? Where? On the commandline? At the interactive prompt?

That's correct. At he commandline. And I see it is working in bash.

~$ ddgr he
headphones           hello                hello\ neighbor      hertz                
hearts               hello\ fresh         hemorrhagic\ stroke  hertz\ car\ sales

In fact I missed both these cases as I switched to zsh nowadays.

So we are back to only next and prev are not working.

jetzerb · 2020-07-07T16:13:02Z

Interesting...I tried tab completion at the commandline earlier and it didn't seem like it was working. Tried again now and discovered that I have to hit tab twice before it offers suggestions.

jarun · 2020-07-07T16:19:57Z

That's correct.

jarun · 2020-07-10T18:08:35Z

@cmonfortep noticed you actively contribute to the DDG android app project. Can you/refer someone who can help us with fetching the next and previous pages?

cmonfortep · 2020-07-13T13:40:51Z

Hi @jarun. I will try to ping someone from the backend team and see if they can comment here. 👍

jarun · 2020-07-13T14:06:19Z

Thank you so much!

muraii · 2020-07-13T21:14:19Z

For what it's worth, I'm using Ubuntu 18.04 which is capped at ddgr v1.2.something. The __init__ function was already using the html.duckduckgo.com/html URI but I was still getting the "No results" behavior. Since I'm not updating to 20.04 any time soon, I instead decided to keep a package-managed version of ddgr but not use Ubuntu's packaging.

Enter pipx. Now I've installed ddgr v.1.8.1, edited the __init__ to point to html.duckduckgo.com/html and everything works, even next and prev. Well, at least on search term "farts".

Is there any kind of debug information I can provide regarding getting next and prev working generally?

jarun · 2020-07-13T23:24:33Z

everything works, even next and prev

Not really. You are only viewing the results of the first fetch (first 30). Do next 3 times and then you'll see the next 30 results fetch fail.

Is there any kind of debug information I can provide regarding getting next and prev working generally?

I think fetching next page fails because there is some change required in the data we send to fetch the next page. The server is expecting something else. You can debug by comparing with the data Firefox sends (you cna see it in Web Developer -> Network).

Note: I am not in a position to work on this now. There's enough info on how/what to debug in this thread. Someone has to pick it up.

muraii · 2020-07-14T01:38:04Z

Quite right. Thanks for the clarification.

…

On Mon, Jul 13, 2020 at 5:24 PM Jana ***@***.***> wrote: everything works, even next and prev Not really. You are only viewing the results of the first fetch (first 30). Do next 3 times and then you'll see the next 30 results fetch fail. Is there any kind of debug information I can provide regarding getting next and prev working generally? I think fetching next page fails because there is some change in the data we send to fetch the next page. The server is expecting something else. You can debug by comparing with the data Firefox sends (you cna see it in Web Developer -> Network). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADCTGIBGPQENWXNMC6YYODR3OJT3ANCNFSM4OMUF7HA> .

jarun · 2020-07-20T13:50:21Z

Fixed at #117. 👍

jn0 · 2020-07-21T16:01:51Z

hm-hm... 1.8.1 from snap on ubuntu 18.04 still fails:

jno@shuttle:~$ ddgr -d ubuntu 20.04
ERROR: ld.so: object 'libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
[DEBUG] ddgr version 1.8.1 Python version 3.5.2
[DEBUG] q:ubuntu 20.04, region:us-en, page:0, curindex:1, safe:1
[DEBUG] nextParams:
[DEBUG] proxy:{'https': None}
[DEBUG] ua:ddgr/1.8.1 (textmode; Linux x86_64; 1024x768)
[DEBUG] Response body written to '/tmp/ddgr-response-1xytet86'.
[DEBUG] Prev nextParams: 
[DEBUG] Next nextParams: 
No results.
ddgr (? for help) 
ddgr (? for help) 
jno@shuttle:~$

(ignore the ERROR - it always here)

And I couldn't find that /tmp/ddgr-response-1xytet86 (even before leaving the program)

And, yes, it has still usr/local/bin/ddgr: self._u = 'https://duckduckgo.com/html' while declares 1.8.1… :(

jarun · 2020-07-21T16:08:42Z

v1.8.1 will not work. Wait for the next release or use master. The fix is in master.

cmonfortep · 2020-07-22T06:28:28Z

@jarun sorry I didn't respond you back about this. I reported the issue internally but I've not been actively following it (I'm enjoying some days off 😅). Glad to see this was solved.

jarun · 2020-07-22T07:59:15Z

No problem! Thank you!

jarun · 2020-07-25T20:20:29Z

Guys, I have released ddgr v1.9 tonight: https://github.com/jarun/ddgr/releases/tag/v1.9

Everything works as before!

Many thanks to @RTek-dev for vital PR #117. 👍

jarun added the help wanted label Jul 1, 2020

jarun closed this as completed Jul 20, 2020

jarun removed the help wanted label Jul 21, 2020

saloniamatteo mentioned this issue Aug 14, 2020

Consistently getting "No results." #119

Closed

github-actions bot locked and limited conversation to collaborators Aug 26, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Suddenly stopped working #115

Suddenly stopped working #115

kurtericmiller commented Jun 30, 2020 •

edited

jarun commented Jul 1, 2020

jarun commented Jul 1, 2020 •

edited

jetzerb commented Jul 5, 2020 •

edited

aalapd commented Jul 6, 2020 •

edited

brandong954 commented Jul 6, 2020

jarun commented Jul 7, 2020

jetzerb commented Jul 7, 2020

jarun commented Jul 7, 2020

jarun commented Jul 7, 2020

jetzerb commented Jul 7, 2020

jarun commented Jul 7, 2020

jetzerb commented Jul 7, 2020

jarun commented Jul 7, 2020

jarun commented Jul 10, 2020

cmonfortep commented Jul 13, 2020

jarun commented Jul 13, 2020

muraii commented Jul 13, 2020

jarun commented Jul 13, 2020 •

edited

muraii commented Jul 14, 2020 via email

jarun commented Jul 20, 2020

jn0 commented Jul 21, 2020

jarun commented Jul 21, 2020

cmonfortep commented Jul 22, 2020

jarun commented Jul 22, 2020

jarun commented Jul 25, 2020 •

edited

Suddenly stopped working #115

Suddenly stopped working #115

Comments

kurtericmiller commented Jun 30, 2020 • edited

jarun commented Jul 1, 2020

jarun commented Jul 1, 2020 • edited

jetzerb commented Jul 5, 2020 • edited

aalapd commented Jul 6, 2020 • edited

brandong954 commented Jul 6, 2020

jarun commented Jul 7, 2020

jetzerb commented Jul 7, 2020

jarun commented Jul 7, 2020

jarun commented Jul 7, 2020

jetzerb commented Jul 7, 2020

jarun commented Jul 7, 2020

jetzerb commented Jul 7, 2020

jarun commented Jul 7, 2020

jarun commented Jul 10, 2020

cmonfortep commented Jul 13, 2020

jarun commented Jul 13, 2020

muraii commented Jul 13, 2020

jarun commented Jul 13, 2020 • edited

muraii commented Jul 14, 2020 via email

jarun commented Jul 20, 2020

jn0 commented Jul 21, 2020

jarun commented Jul 21, 2020

cmonfortep commented Jul 22, 2020

jarun commented Jul 22, 2020

jarun commented Jul 25, 2020 • edited

kurtericmiller commented Jun 30, 2020 •

edited

jarun commented Jul 1, 2020 •

edited

jetzerb commented Jul 5, 2020 •

edited

aalapd commented Jul 6, 2020 •

edited

jarun commented Jul 13, 2020 •

edited

jarun commented Jul 25, 2020 •

edited