Implements subpagination for ddgr #22

mosegontar · 2017-11-19T05:17:01Z

Here's a provisional implementation of subpagination for ddgr.

Pagination is set by using the -n (or --num) option which takes an integer N to paginate the results by. For example: ddgr python -n 5 will paginate by 5 results at a time.

If the number of fetched results left to display is less than the pagination value, new results from the next Ddg results page are fetched and appended to the list of results already returned. Use of the p command in the omniprompt will show the previous N results, where N is the pagination value.

If the f command is used in the omni prompt, the index is reset to 0 so that the first set of results are displayed.

Please let me know if this is in line with what you are thinking/what changes should be made.

… set of results

…w search

jarun · 2017-11-19T06:56:52Z

I think we are on the same page. When we press p, we never fetch results, am I right?

I will review this in 1/2 days.

jarun · 2017-11-19T11:06:36Z

The indices do not start from 0. We have to hide (and store) the true first real index of the current page (as offset)) and start from 1 in each page. When users enters the visible index to open a link in browser, we have to add that to current offset.

If we don't do that, we will see this:

ddgr (? for help) n

 (41) Hello World:) (2013) - IMDb  [www.imdb.com]
Directed by Rafael Mathé, Etienne Larraguetta. With Mikael Alhawi, Remi Baconnet, Jean-Paul Bezzina, Tatiana Campana. In a close future, a private company developed a technology aimed at boosting
our brain capacity. But it requires from its clients to store their memory data on one single server.

 (42) Hello, World | whitehouse.gov  [obamawhitehouse.archives.gov]
Though it wasn't true for President Obama when he became the first president to write a line of code back in December, it's traditional for the first program a new coder writes to be one that prints
out the message "Hello, world."

 (43) Hello, world! - The complete C# Tutorial  [csharp.net-tutorials.com]
A chapter on Hello, world! in the complete Microsoft C# .NET tutorial using Visual Studio Express 2012

 (44) Hello World Personalized Baby Book | I See Me!  [www.iseeme.com]
Hello, World! makes a perfect gift to welcome a new baby into the world! Parents will treasure this adorable keepsake book, as it is personalized with the new baby's name and photograph.

 (45) Lesson: The "Hello World!" Application (The Java™ Tutorials ...  [docs.oracle.com]
This beginner Java tutorial describes getting started with Java and setting up your Netbeans IDE

 (1) The History of Hello World - The Software Guild  [www.thesoftwareguild.com]
Hello World is a program used to simply explain core structure and syntax of a programming language. Learn about the history, modern use, and more here!

 (2) Hello-Hello: Best Language Learning apps for iPad, iPhone ...  [hello-hello.com]
Learn a new language anytime, anywhere with Hello-Hello. Top-selling apps for schools, libraries, companies and more!

 (3) Hello World - CircleCI  [circleci.com]
Hello World. This document describes how to configure your project to run on CircleCI 2.0. Prerequisites. If this will be your very first CircleCI project, complete the steps in the Sign Up & Try
CircleCI document.

 (4) "Hello, world." - vndb.org  [vndb.org]
Title "Hello, world." Aliases: ハローワールド: Length: Very long (> 50 hours) Developer: Nitroplus: Publishers Nitroplus: Relations: Shares characters

 (5) Hello World (TV Series 2016- ) - IMDb  [www.imdb.com]
The leading information resource for the entertainment industry. Find industry contacts & talent representation. Manage your photos, credits, & more

ddgr (? for help)

jarun · 2017-11-19T11:09:03Z

Why don't we update the index when we add it to our cache?

jarun · 2017-11-19T11:12:24Z

We try to fetch whenever we hit the last page. As we are caching the results, can we store the index (initialized to a negative value) to indicate which is our last index the first time we hit it? That way we can save the redundant fetch every time user reaches the last page.

jarun · 2017-11-20T08:14:44Z

Self note: document the option in README, manpage, add to completion-scripts. (owner @jarun)

jarun · 2017-11-20T09:31:04Z

It would be a good idea to default num to 10. 30/40 results per page needs scrolling up/down every time.

jarun · 2017-11-20T09:31:40Z

Let n=0 disable num as a fallback.

…value; calculate true index when opening a url

mosegontar · 2017-11-21T03:19:17Z

I've made the following updates

When paginating and fetching a new page of results, we update DdgCmd._urltable dict with the new set of results. The key for each result is equal to its index value + the previous total number of results retrieved. So if page 1 has 30 results, the 1st result on Page 2 will be added to the dict as 31.
Updated do_open and cmdloop to calculate correct index to open when paginating.
Added a display_index parameter to Result.print() to set the displayed index value instead of the true index value.
Set default --num value to 10.

There is a significant item remaining to be fixed: if pagination values are large, they can cause the index to increase to a value larger than the size of the results, and this causes problems with the output.

jarun · 2017-11-23T14:18:00Z

Self-notes (owner @jarun):

I'll take up any review comments I might add.
We need to fix the large num issue.
Even if we are on the first page, f/p refreshes the list. Need to fix that.

jarun · 2017-11-23T14:40:04Z

ddgr

@@ -1139,18 +1140,25 @@ class DdgCmd(object):
        parser = DdgParser()
        parser.feed(page.text)

-        self.results = parser.results
+        if self.options.pagination:


Where is this getting set???

jarun · 2017-11-23T14:43:32Z

ddgr

@@ -1267,8 +1290,15 @@ class DdgCmd(object):
    @no_argument
    def do_next(self):
        # If no results were fetched last time, we have hit the last page already
-        if self._ddg_url._qrycnt == 0:
+        if self._ddg_url._qrycnt == 0 and self.index >= len(self.results):


What does this additional condition do???

I think we will keep having these issues in deciphering stuff if we never get a single line of code comment.

You add the check self.options.pagination in every other place and then fuse this kind of a condition without any check???

Yes, I think this line is the issue with the huge pagination value.

I have hidden it by limiting n to 25. If anyone is trying to set n greater than that he is trying to automate stuff. He should patch the code for himself. ;)

jarun · 2017-11-23T17:22:31Z

ddgr

+        if self.options.pagination:
+            self.results.extend(parser.results)
+            offset = len(self._urltable)
+            for r in self.results:


We are re-indexing results from offset??? Why???

jarun · 2017-11-23T18:18:17Z

I'll probably have to do a massive rework on this. Anyway...

jarun · 2017-11-23T19:07:41Z

ddgr

                print('No results.', file=sys.stderr)
            else:
                sys.stderr.write(prelude)
-                for r in self.results:
-                    r.print()
+                for i, r in enumerate(results):


Broken for n=0

jarun · 2017-11-23T21:46:11Z

I think we are reasonably good now! 👍

jarun · 2017-11-23T22:52:52Z

ddgr

            print(json.dumps(results_object, indent=2, sort_keys=True, ensure_ascii=False))
        else:
            # Regular output
-            if not self.results:
+            if not self.results or self.index > len(self.results):


Shouldn't the second check be self.index >= len(self.results)?

I think we can get rid of the check completely.

mosegontar added 6 commits November 18, 2017 23:55

Add provisional subpagination

5ea045e

Always display same number of results when paginating, unless at last…

b8dce64

… set of results

If pagination used, reset index to 0 and results to empty array on ne…

a4c5005

…w search

Simplify display_results

512bb28

Add comment to explain negative pagination value in do_previous

9215d5e

Update do_first to account for pagination option

a1d41b3

mosegontar changed the title ~~This PR implements subpagination for ddgr~~ Implements subpagination for ddgr Nov 19, 2017

mosegontar added 3 commits November 20, 2017 20:33

Make sure displayed index begins with 1 and ends with the pagination …

9e0fb2a

…value; calculate true index when opening a url

Set --num default to 10

cc94a36

Add isdigit() check in cmdloop

58944e0

Remove print(cmd) statement

423f4b8

jarun reviewed Nov 23, 2017

View reviewed changes

jarun merged commit 82e05b9 into jarun:master Nov 23, 2017

jarun reviewed Nov 23, 2017

View reviewed changes

github-actions bot locked and limited conversation to collaborators Jun 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implements subpagination for ddgr #22

Implements subpagination for ddgr #22

mosegontar commented Nov 19, 2017

jarun commented Nov 19, 2017 •

edited

jarun commented Nov 19, 2017 •

edited

jarun commented Nov 19, 2017

jarun commented Nov 19, 2017

jarun commented Nov 20, 2017

jarun commented Nov 20, 2017

jarun commented Nov 20, 2017 •

edited

mosegontar commented Nov 21, 2017

jarun commented Nov 23, 2017 •

edited

jarun Nov 23, 2017

jarun Nov 23, 2017

jarun Nov 23, 2017

jarun Nov 23, 2017

jarun Nov 23, 2017

jarun Nov 23, 2017

jarun Nov 23, 2017

jarun commented Nov 23, 2017

jarun Nov 23, 2017

jarun commented Nov 23, 2017

jarun Nov 23, 2017 •

edited

jarun Nov 23, 2017

Implements subpagination for ddgr #22

Implements subpagination for ddgr #22

Conversation

mosegontar commented Nov 19, 2017

jarun commented Nov 19, 2017 • edited

jarun commented Nov 19, 2017 • edited

jarun commented Nov 19, 2017

jarun commented Nov 19, 2017

jarun commented Nov 20, 2017

jarun commented Nov 20, 2017

jarun commented Nov 20, 2017 • edited

mosegontar commented Nov 21, 2017

jarun commented Nov 23, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jarun commented Nov 23, 2017

Choose a reason for hiding this comment

jarun commented Nov 23, 2017

jarun Nov 23, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jarun commented Nov 19, 2017 •

edited

jarun commented Nov 19, 2017 •

edited

jarun commented Nov 20, 2017 •

edited

jarun commented Nov 23, 2017 •

edited

jarun Nov 23, 2017 •

edited