Dealing with cursors in ConnectionField resolve functions

[bigblind]

In the Star Wars relay example, it looks like you expect the resolve_ships method of the Faction class to return every ship. While this is feasible with a small dataset like that, it's not an assumption you can make. Does the resolve function get access to the first and after argument? Also, how do we supply pagination info?

Looking at the code, it looks like connection_from_list deals with the resolved data as a complete list. Is there a way to get around this? If the resolve function only gets the arguments, but no way to supply pagination info, should we just return 1 more object than was requested, so that connection_from_list knows there's more and can correctly create the pagination info?

[syrusakbary]

Hi @bigblind,
connection_from_list expect an iterable (this iterable had to implement len and a item getter like iterable[20:30] though). Could be a list, could be a lazy list, or anything that you could iter through.
However, we don't iter through the iterable until the end (when we show the results).

In the example of Starwars relay the resolve_ships return a complete list. But the swapi example the method returns an iterable.

So, for example, when you are using django querysets, the list is only "resolved" (aka queried in the DB) when we iter though it (when we return the resolved result), and after the list is sliced.

So it will already do in the most optimal way :)

[bigblind]

Can I just create my own field with a type of Connection? The database I'm using, rethinkdb, allows me to do much more optimized queries with cursors than with indices based on len. Basically, with the approach that you use, I need to know the index of every item in the database. With cursors, a rethinkdb query would look like this:

`r.db("my_db").table("users").between(cursor, r.maxval).limit(first+1).run(connection)``

If you're not interested in rethinkdb, I don't expect you to look up how this code works, but I wanted to give a use case for actually doing this.

The reason I'm fetching first+1 items, is so that I know there's a next page of results. if the query returns first items or fewer, I know there are no more items.

[syrusakbary]

Good catch. I will rethink better how we can handle this.
Probably moving some code from the graphql_relay Python library to graphene and wrapping in some class where people could manage it easily is the path to go!
Will keep this issue open until we have a good way for handling this case.

[ekampf]

Similar case here working with AppEngine's NDB (would be the same for HBase or most other nosqls I guess).
I have a huge table. You can iterate it using a starting cursor value (you get the cursor value from the DB) and count of entries you want to fetch. There's an iterator that can do this in batches... but there's no len() operation...

Example fetching a page:

greets, next_curs, more = Greeting.query().fetch_page(10, start_cursor=None)

more will be True\False on wether there's a next page and next_curs is the start_cursor for the next query.

Is there a way to implement this logic of iterating a big list with no len() ?

[jkimbo]

Hi @bigblind . We're currently going through old issues that appear to have gone stale (ie. not updated in about the last 6 months) to try and clean up the issue tracker. If this is still important to you please comment and we'll re-open this.

Thanks!