Diffrence between find() cursor and aql.execute() returned cursor

[arturasjurev]

Python-arango==4.1.0
Arango=3.3.8

Create database with 1000 documents that has same data, but different keys (incremented 1-1000)

if not db.has_collection('data'):
    db.create_collection('data')
    data = db.collection('data')
    for i in range(0,1000):
        data.insert({'_key': str(i+1), 'data': True})

Extract same data via find() and db.aql.execute()

data = db.collection('data')
cursor = data.find({"data": True})
print('Data collection count : ' + str(data.count()))
print('find() | Cursor id : ' + str(cursor.id))
print('find() | Cursor count : ' + str(cursor.count()))

cursor2 = db.aql.execute(
    'FOR doc IN data FILTER doc.data == true RETURN doc',
    count=True,
    batch_size=100)
print('aql.execute() | Cursor id : '+ str(cursor2.id))
print('aql.execute() | Cursor count : ' + str(cursor2.count()))

Output is :

Data collection count : 1000
find() | Cursor id : None
find() | Cursor count : 100
aql.execute() | Cursor id : 19433873780390
aql.execute() | Cursor count : 1000

So in this case find() function returns only 100 documents, and since it has no ID - it's not possible to use fetch() rest 900documents and next() raises StopIteration exception.

[joowani]

Hi @arturasjurev,

Yea all simple queries like Collection.find have default limit of 100 documents.
You can set the limit parameter to an arbitrarily high value. See the documentation for more details.

[sheirys]

But what if we don't know how many matching documents there is and we need fetch or extract them all? Is it okay to use cursor = data.find({"data": True}, limit=data.count()) or is there any other options apart from using aql?

[joowani]

Hmm I guess I can remove the limit of 100 in the next release (there was a reason for this but I don't remember anymore). For now you have only two options, either to set the limit to a really high number or use AQL.

[sheirys]

In provided example cursor = data.find({"data": True}) value of cursor.id is None. Is this expected behavior?

[joowani]

Yes because cursors from simple queries we have no control over like we do with aql queries. It has limit of 100 and thinks there is nothing to fetch, hence cursor ID = None.

[arturasjurev]

In same scenario as opening comment where data has 1000 documents with mentioned filter, executing :

cursor2 = db.aql.execute(
    'FOR doc IN data FILTER doc.data == true RETURN doc',
    count=True,
    batch_size=300)

cursor2.count() is expected to be 1000 ?

[joowani]

I think so. If not try the full_count parameter.

[joowani]

@arturasjurev @sheirys

I've removed the arbitrary limit of 100 in the latest python-arango version 4.2.0. Now it should return all document matches by default.

Closing this out as there doesn't seem to be anything more to follow up on. Feel free to reopen if you have any other questions.