ArangoDB 3.1 fails with collection.all

[tammoippen]

When running the simple querry all with the method collection.all using a batch size smaller than the actual collection (i.e. the cursor api has to be used to fetch more results), results in a requests.exceptions.ConnectionError and the ArangoDB 3.1 instance shuts down. With ArangoDB 3.0 there is no issue. Using the HTTP API of 3.1 with curl is also fine.

Here is my environment:

# installed via homebrew on OS X 10.10.5
» python --version                                                                                                                                                 
Python 3.5.2

» pip freeze
appnope==0.1.0
decorator==4.0.10
ipython==5.1.0
ipython-genutils==0.1.0
pexpect==4.2.1
pickleshare==0.7.4
prompt-toolkit==1.0.9
ptyprocess==0.5.1
Pygments==2.1.3
python-arango==3.4.0
requests==2.12.3
simplegeneric==0.8.1
six==1.10.0
traitlets==4.3.1
wcwidth==0.1.7

Here is a reproducer:
A modified script crash.py from your README.md:

from arango import ArangoClient

# Initialize the client for ArangoDB
client = ArangoClient(enable_logging=False)

# Create a new database named "my_database"
db = client.create_database('my_database')

# Create a new collection named "students"
students = db.create_collection('students')

# Add a hash index to the collection
students.add_hash_index(fields=['name'], unique=True)

# Insert new documents into the collection
students.insert({'name': 'jane', 'age': 19})
students.insert({'name': 'josh', 'age': 18})
students.insert({'name': 'jake', 'age': 21})

result = students.all(batch_size=1)
print([student['name'] for student in result])

# Starting ArangoDB
» docker run -p 8529:8529 --name arangodb -it -d -e ARANGO_NO_AUTH=1 arangodb:3.1

# After finished launching, run the script:
» python crash.py

Results in

Traceback (most recent call last):
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 391, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 387, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1197, in getresponse
    response.begin()
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 643, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py", line 334, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 391, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 387, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1197, in getresponse
    response.begin()
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "crash.py", line 21, in <module>
    print([student['name'] for student in result])
  File "crash.py", line 21, in <listcomp>
    print([student['name'] for student in result])
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/arango/cursor.py", line 32, in __next__
    return self.next()
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/arango/cursor.py", line 120, in next
    res = self._conn.put("/_api/cursor/{}".format(self.id))
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/arango/connection.py", line 211, in put
    auth=(self._username, self._password)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/arango/http_clients/default.py", line 105, in put
    verify=self._check_cert
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/sessions.py", line 546, in put
    return self.request('PUT', url, data=data, **kwargs)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/Users/tammoippen/Desktop/venv/lib/python3.5/site-packages/requests/adapters.py", line 473, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

And on the arango side (it is stopped):

» docker logs -f arangodb
Initializing database...Hang on...
Database initialized...Starting System...
2016-12-06T14:59:01Z [1] INFO ArangoDB 3.1.3 [linux] 64bit, using VPack 0.1.30, ICU 54.1, V8 5.0.71.39, OpenSSL 1.0.1t  3 May 2016
2016-12-06T14:59:01Z [1] INFO using SSL options: SSL_OP_CIPHER_SERVER_PREFERENCE, SSL_OP_TLS_ROLLBACK_BUG
2016-12-06T14:59:01Z [1] INFO file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2016-12-06T14:59:01Z [1] INFO JavaScript using startup '/usr/share/arangodb3/js', application '/var/lib/arangodb3-apps'
2016-12-06T14:59:02Z [1] INFO In database '_system': Found 14 defined task(s), 1 task(s) to run
2016-12-06T14:59:02Z [1] INFO In database '_system': state standalone/existing, tasks updateUserModels
2016-12-06T14:59:02Z [1] INFO In database '_system': existing cleanup successfully finished
2016-12-06T14:59:02Z [1] INFO using endpoint 'http+tcp://0.0.0.0:8529' for non-encrypted requests
2016-12-06T14:59:03Z [1] INFO ArangoDB (version 3.1.3 [linux]) is ready for business. Have fun!
2016-12-06T14:59:16Z [1] INFO created application directory '/var/lib/arangodb3-apps/_db/my_database' for database 'my_database'
2016-12-06T14:59:16Z [1] INFO In database 'my_database': No version information file found in database directory.
2016-12-06T14:59:16Z [1] INFO In database 'my_database': Found 14 defined task(s), 9 task(s) to run
2016-12-06T14:59:16Z [1] INFO In database 'my_database': state standalone/init, tasks setupGraphs, addDefaultUserOther, createModules, createRouting, insertRedirectionsAll, setupAqlFunctions, createFrontend, setupQueues, setupJobs
2016-12-06T14:59:17Z [1] INFO In database 'my_database': init successfully finished

This happens also with password for root on arangodb.

[tammoippen]

I just saw, that you use the export API. It seems, this is an arangodb 3.1 error, as I can reproduce the error with curl. I will propagate this to Arango.

[tammoippen]

This seems to be an issue on the python-arango side as well. You are using the wrong cursor, see this explanation and this one.

[joowani]

Hi @tammoippen

I appreciate you looking into this. I will make the appropriate fix and push out a new release sometime this week or next.

Thanks!

[joowani]

Hi again @tammoippen

I've pushed out a fix in the latest release 3.4.1.
Let me know how it fares. Thanks.

[bangert]

This works for me.
Thank you very much!

[joowani]

Awesome. Closing this out.