Option for arrayLimit returning data even if it's over the limit

[thelinuxlich]

It would be nice if we had a .run() optarg to let the query return as much documents as possible instead of throwing an error because of arrayLimit.

I mean, if arrayLimit is 100k and the query results in 500k rows, return 100k

[danielmewes]

I suspect that you can emulate this in most (if not all) cases by inserting .limit(100000) terms into your query at the right places (before the command that converts the data into an array). Though I can see that becoming a bit annoying if you're building ad-hoc queries from user requests.

One issue with truncating automatically is that things might no longer behave in the expected way.
For example if you have a non-indexed orderBy("val"), and we simply truncate the input to 100k elements, then the first result in the output is not actually going to be the one with the smallest val.

[thelinuxlich]

Isn't this a temporary limitation because there are features that should be streamable and currently aren't?

[danielmewes]

@thelinuxlich That is true for some terms, but not for others. An un-indexes orderBy can never be streaming for example as far as I can tell. It needs to wait for all inputs before it can determine which input is the smallest.
Would it help us here if it was temporary?

[thelinuxlich]

You're right, but then .limit() before orderBy() would have the same effect, with the disadvantage of routing everything to one shard