ForEach only return 100 nodes

[gokhanckgz]

When iterate data which has nodes greater than 100 , foreach function only getting 100 of them.

g.V().ForEach( function(d) { g.Emit(d) })

and

g.V().All()

should be equal I think. But g.Emit(d) inside ForEach always returns only 100 nodes.

Cayley version : 0.7.3 , Backend : mongodb

[steffansluis]

I think this is because of the limit query parameter defined here. Try sending the query to /api/v1/query/gizmo?limit=10000.

[dennwc]

@steffansluis Still, it's unexpected that it affects ForEach but doesn't affect All.

[steffansluis]

If that is indeed the case, then it is a bug. Although I'm not sure if by should be equal I think @gokhanckgz means that both only return 100 results or that one of them behaves differently.

[gokhanckgz]

Yes, they behaves differently. In my case All returns all nodes (about 800) but ForEach not iterating all nodes (always iterates 100 of them).

[gokhanckgz]

I was using v2 HTTP API. But with /api/v1/query/gizmo?limit=10000 and ForEach I can iterate all nodes now , just like you said @steffansluis .

[dennwc]

It was a bug with g.All() not honoring a query limit. Now it also returns only the first 100 nodes. To iterate all nodes, set ?limit=-1.

[mojoex]

Why does g.All() limit and where is it set? Shouldn't it default to all edges? And where is the default limit set? This seems like an unnecessary change given the 'All()' statement literally does mean.. get me All() things.
This was a breaking change in our application - where we have around 50 strongly typed queries using All().

[dennwc]

@mojoex Imagine a large database with millions of nodes executing All(). It shouldn't be a default behavior to return them all. But if you know that a number of nodes is limited, just pass ?limit=-1 with your queries and it will disable the limit.

Note that this does not affect Go applications that use Cayley as a library.

[iddan]

I think this is counter intuitive.
Other solutions to large datasets are:

• SQL doesn't limit by default but some REPLs do.
• BigQuery gives a memory if a query is executed without a limit.
• Mongo returns the first batch of a cursor with a default batch size of 1000 but all results can be iterated seamlessly.
• Most UIs paginate results.

Maybe the Cayley UI should paginate results, and libraries use a cursor but HTTP to return all results by default?

[dennwc]

Yes, pagination is the way to go and #121 now tracks this.