Skip to content
This repository has been archived by the owner on Feb 10, 2021. It is now read-only.

PR: Make HDFile pickleable #89

Closed
wants to merge 2 commits into from
Closed

PR: Make HDFile pickleable #89

wants to merge 2 commits into from

Conversation

martindurant
Copy link
Member

Essentially re-opens file on deserialization. This means multiple
processes may now have the same file open - should make sure only
one is writing.

Essentially re-opens file on deserialization. This means multiple
processes may now have the same file open - should make sure only
one is writing.
@mrocklin
Copy link
Member

mrocklin commented Aug 9, 2016

This could use a test. I would open a file, pickle and unpickle it, and then verify that it works as expected.

@martindurant
Copy link
Member Author

I found (manually), that writing to the deserialized file makes the original file unwritable - the write appears to succeed but has no effect if f2 is closed, or causes exception if f is still open (LeaseExpiredException).

@martindurant
Copy link
Member Author

https://github.com/Pivotal-Data-Attic/pivotalrd-libhdfs3/pull/43 concat() is coming to libhdfs3, which would allow writing from many processes and joining the files only when all writes are complete.

self.__dict__.update(state)
self.fs.connect()
self._fs = self.fs._handle
self._set_handle()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if the original file object was closed?

@mrocklin
Copy link
Member

I'm going to close this for now as stale.

@mrocklin mrocklin closed this Nov 21, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants