[MRG] persistence in/from file objects #351

aabadie · 2016-05-11T13:56:43Z

This PR should fix #341 and fix #312.

Examples of new usage:

import joblib
data = "some data to persist"
# persist to/from a raw file
with open('/tmp/test.pkl', 'wb') as f:
    joblib.dump(data, f)

with open('/tmp/test.pkl', 'rb') as f:
    print(joblib.load(f))

# compressors are also supported:
import gzip
with gzip.GzipFile('/tmp/test.pkl.gz', 'wb') as f:
    joblib.dump(data, f)

with gzip.GzipFile('/tmp/test.pkl.gz', 'rb') as f:
    print(joblib.load(f))

# auto detection of compression also works:
with open('/tmp/test.pkl.gz', 'rb') as f:
    print(joblib.load(f))

# it's also possible to pickle in a buffer in memory using io.BytesIO:
import io
buf = io.BytesIO()
joblib.dump(data, buf)
joblib.load(buf)

Test functions are still missing.

aabadie · 2016-05-12T08:20:13Z

I think this is in a pretty good shape now, so changing the status to MRG.

A few comments though:

the implementation is not perfect (e.g new _COMPRESSOR_OBJS list, etc) so I'd like to have someone else's opinion,
forcing the decompressor to a wrong one (exemple: use gzip compression with dump but try to uncompress with bz2) will raise an exception. Should we handle this case ? My answer on this is 'no' as the message is quite meaningful but maybe other people won't agree with this,
online documentation and CHANGES.rst still need to be updated but this can be done after a preliminary review.

aabadie · 2016-05-18T16:05:15Z

@lesteve, I addressed what we discussed IRL. Feel free to have a look when you have time.

ogrisel · 2016-05-26T09:40:10Z

joblib/numpy_pickle.py

        filename = str(filename)

+    isFilename = isinstance(filename, _basestring)
+    isFileobj = hasattr(filename, "read") and hasattr(filename, "write")


🚨 ⚠️ 🐫 ⚠️ camelcase alert! ⚠️ 🐫 ⚠️ 🚨

Sorry, habits from my old Qt life. I renamed using 🐍 snake_case 🐍 😄

aabadie · 2016-06-09T15:53:20Z

@lesteve, @ogrisel , any comment ?

lesteve · 2016-06-09T16:24:15Z

joblib/numpy_pickle.py

                                                   compress_level)) as f:
            NumpyPickler(f, protocol=protocol).dump(value)

    else:


Use a elif here so that all the conditions are on the same indentation level.

aabadie · 2016-06-17T13:15:53Z

Just for the record, here is the kind of things now possible with this PR: https://gist.github.com/aabadie/074587354d97d872aff6abb65510f618

lesteve · 2016-06-17T15:22:56Z

joblib/test/test_numpy_pickle.py

        nose.tools.assert_equal(len(caught_warnings), 1)
        for warn in caught_warnings:
-            nose.tools.assert_equal(warn.category, DeprecationWarning)
+            nose.tools.assert_equal(warn.category, Warning)


Why DeprecationWarning -> Warning?

Because this is the type of warning raised here and that we want to test: it checks one cannot load a compressed pickle using mmap_mode so I think there's no meaning using a DeprecationWarning in this case.

lesteve · 2016-06-20T09:42:38Z

joblib/numpy_pickle.py

+                'This feature is not supported by joblib.')
+            new_exc.__cause__ = exc
+            raise new_exc
+        # Raise exception "as-is" with python 2


Reraise exception with Python 2

aabadie · 2016-06-23T16:37:38Z

I had issues with doctest and the usage of file object in a context manager. I skipped the lines in the doc but that may not be a solution. Any better idea ?

Otherwise, I rebased on the lastest master (including the flake8_diff.sh recent changes) and it works just fine.

lesteve · 2016-06-24T10:23:22Z

I had issues with doctest and the usage of file object in a context manager. I skipped the lines in the doc but that may not be a solution. Any better idea ?

What was the error? It would be great to understand where it comes from and fix it properly ...

Otherwise, I rebased on the lastest master (including the flake8_diff.sh recent changes) and it works just fine.

Great, thanks a lot!

aabadie · 2016-06-24T12:44:03Z

What was the error? It would be great to understand where it comes from and fix it properly ...

There were 2 things:

with open() as fo returns the file object ([< io.blabla >]) : I fixed it using an ellipsis flags and by adding [<...>] after the with block
in python 2.6, using gzip.GzipFile or bz2.BZ2File doesn't work if you don't use contextlib.closing. So I had to skip the file object compression example.

lesteve · 2016-06-24T13:00:50Z

with open() as fo returns the file object ([]) : I fixed it using an ellipsis flags and by adding [<...>] after the with block

Hmmm I am not sure what we should do in this case:

with gzip.GzipFile('/tmp/test.gz', 'wb') as f:             
    res = joblib.dump({'a': [1, 2, 3], 'b': 'asadf'}, f)

Should res be:

a single element list containing containing f
a single element list containing containing f.name to be more consistent with when dump is called with a filename
because this is a new feature, can we do what we want, in particular returning a list may not make sense, since the list was there for historical reasons (main pickle + companion files for each numpy array)

in python 2.6, using gzip.GzipFile or bz2.BZ2File doesn't work if you don't use contextlib.closing. So I had to skip the file object compression example.

Pfff 😭 is there a way we can just skip doctests in Python 2.6?

lesteve · 2016-06-24T13:30:20Z

Pfff 😭 is there a way we can just skip doctests in Python 2.6?

Seems like scikit-learn has some *fixture* files in the doc folder to skip the doctests. You can probably have a look there. I believe there is something in setup.cfg as well to work with nose and doctest-fixtures.

aabadie · 2016-06-27T09:01:25Z

Should res be:

a single element list containing containing f

a single element list containing containing f.name to be more consistent with when dump is called > with a filename

because this is a new feature, can we do what we want, in particular returning a list may not make > sense, since the list was there for historical reasons (main pickle + companion files for each numpy array)

I think returning nothing in case a file object is given as input is reasonable because this means the user knows in advance the destination of the dump and the returned list is here, as you said, for historical reasons.

aabadie · 2016-06-27T09:14:26Z

Seems like scikit-learn has some fixture files in the doc folder to skip the doctests

Nice, it works like a charm. The setup.cfg file in joblib was already correctly configured.

lesteve · 2016-06-28T12:08:16Z

doc/persistence_fixture.py

+def setup_module(module):
+    """Setup module."""
+    if _compat.PY26:
+        raise SkipTest("Skipping persitence doctest in Python 2.6")


typo in persistence (missing s)

lesteve · 2016-06-28T12:09:34Z

I think returning nothing in case a file object is given as input is reasonable because this means the user knows in advance the destination of the dump and the returned list is here,

Not entirely convinced, but it is a good point that joblib.dump returning something is mostly for companion files (i.e. npy files) since they are not passed in by the user. Let's ask @GaelVaroquaux and @ogrisel whether they have an opinion on this!

GaelVaroquaux · 2016-06-28T14:33:31Z

I think returning nothing in case a file object is given as input
is reasonable because this means the user knows in advance the
destination of the dump and the returned list is here,

The danger with returning the file object is that it risks having users
holding on the open file handles. Under windows, this locks the file, and
hence can raise problems in parallel computing.

aabadie · 2016-06-28T19:53:31Z

The danger with returning the file object is that it risks having users holding on the open file handles. > Under windows, this locks the file, and hence can raise problems in parallel computing.

So I guess we can keep it as it is in this PR (returns nothing).

lesteve · 2016-06-29T14:11:00Z

So I guess we can keep it as it is in this PR (returns nothing).

Let's do it like this in this PR. It'd be good to try to make the joblib.dump return type consistent whether it takes a filename or a file object though in a further PR.

lesteve · 2016-06-29T14:12:43Z

doc/persistence_fixture.py

+def setup_module(module):
+    """Setup module."""
+    if _compat.PY26:
+        raise SkipTest("Skipping persistence doctest in Python 2.6")


Can you add a comment to say that GzipFile can not be used as a context manager in Python 2.6 or some better explanation?

Comment added.

lesteve · 2016-06-29T14:57:24Z

Merging thanks!

* tag '0.10.2': (55 commits) Bump up __version__ Update release to 0.10.2 in CHANGES.rst nosetests should run tests only from the joblib folder. API expose joblib.parallel.ParallelBackendBase and joblib.parallel.AutoBatchingMixin Update to numpydoc 0.6 [numpy_pick_utils] Handle unicode filenames in BinaryZlibFile (joblib#384) Fix format_stack with compiled code (e.g. .so or .pyd) PEP8: cosmit fix (joblib#376) FIX typo Release 0.10.0 FIX: __all__ should hold symbol names as strings Fix bench_auto_batching.py [MRG] Persistence in/from file objects (joblib#351) Minor tweaks in auto batching benchmark script Improve flake8_diff.sh (joblib#371) FIX numpy array persistence with pickle HIGHEST_PROTOCOL (joblib#370) DOC: remove easy_install from joblib installation documentation (joblib#363) MAINT fix typo DOC Add documentation of mmap_mode Explicit handling of job cancellation on first collected exception (joblib#361) ...

aabadie added the in progress label May 11, 2016

aabadie mentioned this pull request May 11, 2016

In-memory caching #341

Closed

aabadie changed the title ~~WIP : persistence in file handles~~ WIP : persistence using file handles May 11, 2016

aabadie force-pushed the file_handle_persistence branch 2 times, most recently from 56d6805 to ba0f914 Compare May 12, 2016 08:01

aabadie changed the title ~~WIP : persistence using file handles~~ [MRG] persistence using file handles May 12, 2016

aabadie force-pushed the file_handle_persistence branch from ba0f914 to 2d9833c Compare May 18, 2016 15:49

aabadie referenced this pull request in GaelVaroquaux/website May 20, 2016

finishing touch

3f381a1

ogrisel reviewed May 26, 2016
View reviewed changes

aabadie force-pushed the file_handle_persistence branch 2 times, most recently from 7996f7b to f33fac6 Compare May 27, 2016 14:14

aabadie changed the title ~~[MRG] persistence using file handles~~ [MRG] persistence using file object Jun 9, 2016

aabadie changed the title ~~[MRG] persistence using file object~~ [MRG] persistence in/from file object Jun 9, 2016

aabadie changed the title ~~[MRG] persistence in/from file object~~ [MRG] persistence in/from file objects Jun 9, 2016

lesteve reviewed Jun 9, 2016
View reviewed changes

aabadie force-pushed the file_handle_persistence branch from 06d650e to 99a275a Compare June 15, 2016 09:26

aabadie added this to the 0.10 milestone Jun 16, 2016

aabadie force-pushed the file_handle_persistence branch from 99a275a to 6e9d362 Compare June 17, 2016 13:18

lesteve reviewed Jun 17, 2016
View reviewed changes

aabadie force-pushed the file_handle_persistence branch from 6e9d362 to 1af045f Compare June 20, 2016 07:53

aabadie added the enhancement label Jun 20, 2016

lesteve reviewed Jun 20, 2016
View reviewed changes

aabadie added 4 commits June 23, 2016 18:16

udpating CHANGES.rst

3dc43be

addressing comments

221aab5

addressing comments

97be6a2

fixing doctest

2002371

aabadie force-pushed the file_handle_persistence branch from e7b1cb3 to 2002371 Compare June 23, 2016 16:16

fixing persistence doctest

bbb0d52

fixing doctest

7d1e947

aabadie force-pushed the file_handle_persistence branch from 3a96f67 to 7d1e947 Compare June 27, 2016 09:14

lesteve reviewed Jun 28, 2016
View reviewed changes

typo

92a4f34

addressing comments

fa71903

lesteve reviewed Jun 29, 2016
View reviewed changes

commenting python 2.6 skip in persitence documentation

99c597c

lesteve merged commit af2e9b7 into joblib:master Jun 29, 2016

lesteve removed the in progress label Jun 29, 2016

aabadie deleted the file_handle_persistence branch September 24, 2016 19:35

pyup-bot mentioned this pull request Jul 28, 2017

Pin joblib to latest version 0.11 workforce-data-initiative/skills-airflow#38

Merged

[MRG] persistence in/from file objects #351

[MRG] persistence in/from file objects #351

Uh oh!

Conversation

aabadie commented May 11, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aabadie commented May 12, 2016

Uh oh!

aabadie commented May 18, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aabadie commented Jun 9, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aabadie commented Jun 17, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aabadie commented Jun 23, 2016

Uh oh!

lesteve commented Jun 24, 2016

Uh oh!

aabadie commented Jun 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lesteve commented Jun 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lesteve commented Jun 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aabadie commented Jun 27, 2016

Uh oh!

aabadie commented Jun 27, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lesteve commented Jun 28, 2016

Uh oh!

GaelVaroquaux commented Jun 28, 2016

Uh oh!

aabadie commented Jun 28, 2016

Uh oh!

lesteve commented Jun 29, 2016

Uh oh!

lesteve Jun 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lesteve commented Jun 29, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

aabadie commented May 11, 2016 •

edited

Loading

aabadie commented Jun 24, 2016 •

edited

Loading

lesteve commented Jun 24, 2016 •

edited

Loading

lesteve commented Jun 24, 2016 •

edited

Loading

lesteve Jun 29, 2016 •

edited

Loading