[MRG] Numpy pickle to single file by aabadie · Pull Request #260 · joblib/joblib

aabadie · 2015-10-23T16:22:17Z

I gave a try to gzip.GzipFile to dump numpy object. This PR is a POC of such usage and also contains a fairly large refactoring of numpy_pickle.

In terms of performance, I reused the bench script from #255. With Numpy 1.10, there's no memory copy but the read/write is slightly slower (31s dump, 5.5s read). It also makes the code more 'readable'.
I had to comment out the test regarding the compatibility between pickles.

Waiting for the CI's

This change is

GaelVaroquaux · 2015-10-24T08:02:26Z

the read/write is slightly slower (31s dump, 5.5s read).

How does it compare to what we currently have?

aabadie · 2015-10-26T17:14:32Z

How does it compare to what we currently have?

For an array of 763MB, it adds a read/write time overhead of approximately 10% : 28s write with the actual master against 31s with this PR (4.8s against 5s for reading). This extra time consuming is balanced with a stable memory consumption (for both compress and not compressed serialization), at least for recent versions of numpy, and be explained by the CRC computation used in GzipFile, which is not performed with the direct usage of zlib.
Note also that this PR is not fully complete, it misses improvements:

the control file (containing the main serialized object without arrays/matrices) is not compressed
the whole object (including arrays) could be serialized in a single file using a smart seeking in the file. This could be part of an extra PR.
last, but not least : this PR should also contains a retrocompatibility mecanism with previous versions of the cache format.

aabadie · 2015-11-12T16:54:01Z

the control file (containing the main serialized object without arrays/matrices) is not compressed

This was done in 4f58113

the whole object (including arrays) could be serialized in a single file using a smart seeking in the file. This could be part of an extra PR.

I just pushed caeb704 which does this. But it has several drawbacks:

tests are broken because the actual state doesn't support memory mapping
I had to patch numpy locally in order to make the gzip file compression work. Here, numpy overload GzipFile object seek() function in order to be able to seek using negative values. The problem is that numpy assumes the file only contains one array which is not our case here. The only solution is to include numpy array load/save implementation in joblib.

last, but not least : this PR should also contains a retrocompatibility mecanism with previous versions of the cache format.

This is done in this branch (works in 4f58113)

GaelVaroquaux · 2015-11-12T16:56:37Z

file only contains one array which is not our case here. The only
solution is to include numpy array load/save implementation in
joblib.

Can we then only reimplement the minimal amount that we need, and not
copy the whole codebase?

aabadie · 2015-11-12T17:43:19Z

Can we then only reimplement the minimal amount that we need, and not
copy the whole codebase?

Yes, that's the plan. It should also helps with the memmap case.

aabadie · 2015-11-13T16:03:58Z

As the status of this PR is imho in a good shape, I changed the status to MRG and renamed it to better reflect what it contains : now object are serialized in a single file, even compressed file and memory maps.
Note that there is a basic retrocompatibility mecanism which means that the future new version of joblib will be able to read files containing object cached with older versions. The dump will use the new format version.

All remaining problems mentionned above are solved:

as suggested, the gzip.GzipFile issue was solved by reimplementing (not to say copy-pasting), the minimum amount of required code from numpy in joblib. I'm not if I did that the best way, but at least it works now. The compression level is 3 (it can be from 0 to 9 as said in the doc).
the memory mapping works as well using the same technique. I just had to slightly change the way the implementation of the numpy open_memmap function : I introduced a parameter for setting the offset in the file where the numpy array is serialized.

Maybe some more tests could be added. I'll post here some memory/speed benchmarks just to compare with the actual implmentation.

Appveyor is stuck, I don't know why.

Waiting for you comments.

GaelVaroquaux · 2015-11-13T16:06:22Z

For benchmarks, have a look at the following link (hint: there is a link
to a gist with code to benchmark in it)
http://gael-varoquaux.info/programming/joblib-beta-release-fast-compressed-persistence-python-3.html

aabadie · 2015-11-13T17:27:18Z

Thanks, I'll try it asap !

lesteve · 2015-11-17T14:07:01Z

We need to find a better name than this.

We need to find a better name than this.

What about NumpyArrayWrapper ?

What about NumpyArrayWrapper ?

Sure. The reason of the original name is that the name of the class of a numpy array is 'ndarray': <pre> In [3]: type(np.zeros(1)) Out[3]: numpy.ndarray </pre>

aabadie · 2015-11-17T15:25:32Z

As promised to @GaelVaroquaux, I tested your gist locally, just to compare with other implementations.

See the results:

Read speed with this PR:
Read speed with master:
Write speed with this PR:
write speed with master:

The conclusion is that the speed is clearly slower with this PR. And as you noticed in your blog post, gzip.GZipFile might not be an option for cache compression.

aabadie · 2015-11-17T15:33:59Z

Looking at the plots, it seems that the datasets labels are mixed. I'll update them.

aabadie · 2015-11-17T15:49:00Z

Looking at the plots, it seems that the datasets labels are mixed. I'll update them.

Replying to myself. It double checked and all is ok. It's just the write speed using F-Contiguous data which has a significant impact on performance (especially when uwing pytables with zlib).

GaelVaroquaux · 2015-11-17T15:59:50Z

It doesn't seem to me that the impact of this PR on read/write speed is very large.

GaelVaroquaux · 2015-11-17T16:07:44Z

As a side note, when this PR is merged, it would be useful to test lzma for compression under Python 3: it could be much faster. I have added an enhancement issue for this: #273.

aabadie · 2015-11-17T16:10:33Z

I also want to try the mmap case using read-write mode and especially when the persisted object contains multiple arrays. I'm wondering if the arrays could overlap at the end of the file resulting in a corrupted file.

GaelVaroquaux · 2015-11-17T16:12:00Z

I'm wondering if the arrays could overlap at the end of the file resulting in a corrupted file.

Sound like we need a test :)

ogrisel · 2016-05-04T13:27:51Z

+    env['filename'] = os.path.join(env['dir'], 'test.pkl')
+    print(80 * '_')
+    print('setup numpy_pickle')
+    print(80 * '_')


Please avoid printing stuff in tests.

ogrisel · 2016-05-09T14:03:14Z

Can you also please squash the commits? Commit messages such as "addressing other comments" are really not interesting.

aabadie · 2016-05-09T14:18:24Z

Can you also please squash the commits?

@ogrisel : history rewritten :)

ogrisel · 2016-05-10T11:46:07Z

+
+def _check_compression_format(filename, expected_list):
+    if (not PY3_OR_LATER and (filename.endswith('xz') or
+                              filename.endswith('lzma'))):


filename.endswith('.xz') or filename.endswith('.lzma')

Good catch ! Just pushed the update.

…lzma files

ogrisel · 2016-05-10T13:56:15Z

+
+            # We are careful to open the file handle early and keep it open to
+            # avoid race-conditions on renames.
+            # That said, if data are stored in companion files, which can be


"data" is a mass noun so it should always be singular: "data is stored"

ogrisel · 2016-05-10T14:09:01Z

Thanks @aabdie this LGTM. Merging.

ogrisel · 2016-05-10T14:09:22Z

🍻

GaelVaroquaux · 2016-05-10T14:11:15Z

Yehaaa!!!

lesteve · 2016-05-10T14:19:14Z

Great stuff!

aabadie · 2016-05-10T15:02:40Z

Woohoo !! 🎉

mrocklin · 2016-05-11T15:40:22Z

+                                                  'zerosize_ok'],
+                                           buffersize=buffersize,
+                                           order=self.order):
+                pickler.file_handle.write(chunk.tostring('C'))


Question: is it possible to use the memoryview directly? This might avoid a copy

In [1]: import numpy as np In [2]: x = np.ones(5) In [3]: x.data Out[3]: <memory at 0x7f201c81fe58> In [4]: with open('foo.pkl', 'wb') as f: f.write(x.data)

I think we tried that but it breaks for some versions of Python / numpy if I recall correctly. @aabadie do you confirm?

Anyway this is a small buffer so it introduce minimal memory overhead and I don't think the performance overhead is significant.

Thanks @mrocklin for the suggestion.
@ogrisel, I was running some benches/tests with this change. Indeed, it avoids a memory copy but as you said it's not significant as this code works with a few MB (16). Performance is the same.
It also works on all versions of python/numpy supported by joblib.

Also I think this will result in storing incorrect data in the pickle when the array is not contiguous.

Just tried this using:

import numpy as np import joblib a = np.asarray(np.arange(100000000).reshape((1000, 500, 200)), order='F')[:, :1, :] a.flags # f_contiguous: False, c_contiguous: False, but aligned: True joblib.dump(a, '/tmp/test.pkl') np.allclose(a, joblib.load('/tmp/test.pkl')) # Return True

Seems ok to me.

a = np.asarray(np.arange(100000000).reshape((1000, 500, 200)), order='F')[:, :1, :]

Try with [::2]

Try with [::2]

Works

Interesting, then let's do a PR :)

I think np.nditer must return contiguous chunks whatever the input.

then let's do a PR

Here it is : #352 :)

aabadie force-pushed the gzip_pickling branch from 8dae363 to 3213b72 Compare October 23, 2015 16:27

aabadie force-pushed the gzip_pickling branch from a39fd5f to caeb704 Compare November 12, 2015 16:47

aabadie force-pushed the gzip_pickling branch 7 times, most recently from d331808 to adacbd6 Compare November 13, 2015 15:44

aabadie changed the title ~~[WIP] Gzip pickling~~ [WIP] Numpy pickle to single file Nov 13, 2015

aabadie changed the title ~~[WIP] Numpy pickle to single file~~ [MRG] Numpy pickle to single file Nov 13, 2015

lesteve reviewed Nov 17, 2015
View reviewed changes

GaelVaroquaux mentioned this pull request Nov 17, 2015

Try lzma for compression on Python >= 3.3 in our persistence #273

Closed

ogrisel reviewed May 4, 2016
View reviewed changes

aabadie added 5 commits May 9, 2016 16:12

persistence: refactoring of numpy pickle dump/load functions

87953ae

persistence: adding tests (that work)

c8e0e23

persistence: adding benchmark scripts

af2cc19

persistence: updating documentation

0eed039

persistence: updating CHANGES.rst (see PR joblib#260)

9a88f69

ogrisel reviewed May 10, 2016
View reviewed changes

persistence: improving check compression format detection for xv and …

6181f5d

…lzma files

ogrisel reviewed May 10, 2016
View reviewed changes

mrocklin reviewed May 11, 2016
View reviewed changes

aabadie mentioned this pull request May 12, 2016

[MRG] avoid memory copies when writing chunks of data #352

Merged

lesteve mentioned this pull request Jun 22, 2016

Document the size limitation when using compression under Python 2 #300

Closed

przemyslslaw mentioned this pull request Aug 17, 2016

OverflowError: size does not fit in an int #122

Closed

aabadie mentioned this pull request Nov 3, 2016

[RFC] cpickle based dump/load #421

Open

pyup-bot mentioned this pull request Jul 28, 2017

Pin joblib to latest version 0.11 workforce-data-initiative/skills-airflow#38

Merged

lesteve mentioned this pull request Feb 27, 2025

Use protocol=5 for dumping arrays with dtype=object #1682

Merged

Conversation

aabadie commented Oct 23, 2015

Uh oh!

GaelVaroquaux commented Oct 24, 2015 via email

Uh oh!

aabadie commented Oct 26, 2015

Uh oh!

aabadie commented Nov 12, 2015

Uh oh!

GaelVaroquaux commented Nov 12, 2015

Uh oh!

aabadie commented Nov 12, 2015

Uh oh!

aabadie commented Nov 13, 2015

Uh oh!

GaelVaroquaux commented Nov 13, 2015

Uh oh!

aabadie commented Nov 13, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GaelVaroquaux Nov 18, 2015 via email

Choose a reason for hiding this comment

Uh oh!

aabadie commented Nov 17, 2015

Uh oh!

aabadie commented Nov 17, 2015

Uh oh!

aabadie commented Nov 17, 2015

Uh oh!

GaelVaroquaux commented Nov 17, 2015

Uh oh!

GaelVaroquaux commented Nov 17, 2015

Uh oh!

aabadie commented Nov 17, 2015

Uh oh!

GaelVaroquaux commented Nov 17, 2015 via email

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel commented May 9, 2016

Uh oh!

aabadie commented May 9, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel commented May 10, 2016

Uh oh!

ogrisel commented May 10, 2016

Uh oh!

GaelVaroquaux commented May 10, 2016

Uh oh!

lesteve commented May 10, 2016

Uh oh!

aabadie commented May 10, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel May 12, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aabadie May 12, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel May 12, 2016 •

edited

Loading

aabadie May 12, 2016 •

edited

Loading