slow inserts

[dimitri-yatsenko]

On my setup, it takes about 20 minutes to insert 1GB (after compression) with big blobs. MATLAB inserts seem much faster. The hold up seems to be on the python side, not MySQL side. Perhaps zlib takes long. Fetching seems to take normal times. Need to profile and identify opportunities for speedup.

[dimitri-yatsenko]

Indeed, inserting large blobs takes about 5 times longer in Python than MATLAB.

Python:

In[14]: timeit.timeit('probe.Test().insert1(dict(id=2, value=numpy.random.randn(1024,1024,4)))', number=1, setup='import probe, numpy')
Out[14]: 22.01270990096964

MATLAB

>> tic, insert(probe.Test, {1 randn(1024,1024,4)}), toc
Elapsed time is 4.306250 seconds.

However, the problem does not appear to be with the serialization.

In[15]: timeit.timeit('datajoint.blob.pack(numpy.random.randn(1024,1024,4))', number=1, setup='import datajoint, numpy')
Out[15]: 2.0954980110982433

The problem must be within pymysql?

[dimitri-yatsenko]

The problem is caused by slow byte string processing. I have submitted this issue to pymysql: PyMySQL/PyMySQL#366

[dimitri-yatsenko]

closing for now. The hexadecimal encoding of binary data works even though it consumes twice the bandwidth. We will wait for the permanent solution from pymysql.

[dimitri-yatsenko]

Looks like PEP-461 should fix this in Python 3.5.1: https://www.python.org/dev/peps/pep-0461/

[dimitri-yatsenko]

looks like Python 3.6 has resolved byte processing issues.