New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

writing data to /tmp peaks_from_model #256

Closed
mdesco opened this Issue Oct 10, 2013 · 8 comments

Comments

Projects
None yet
7 participants
@mdesco
Contributor

mdesco commented Oct 10, 2013

peaks_from_model writes temporary files in /tmp on linux.

On Mac OS X, it is written according to:
print(tempfile.mkdtemp())
which points here on my machine:
/var/folders/XN/XNu2M8uWGV01N+eAyY5bKE+++TI/-Tmp-/

If you run many spherical deconvolution estimation, this folder quickly fills up with data and you can fill up your disk.

Is there a way to do a cleanup of these files when done?
Can we give an option to keep stuff in RAM?

@Garyfallidis

This comment has been minimized.

Member

Garyfallidis commented Oct 10, 2013

Of course there is. You can use os.remove() to remove a file or os.rmdir() will remove an empty directory or shutil.rmtree() will delete a directory and all its contents. The temporary files should be deleted after they have been created. This is definitely a bug. @gabknight can you fix this?

@gabknight

This comment has been minimized.

Contributor

gabknight commented Oct 10, 2013

Yes. I thought we want to keep the data in memmap after the function finished, to reduce memory usage. I can bring it back to the RAM and clean-up the disk after the processing. I will also do it for the data splitting before the call to pool.map to reduce the memory usage (currently it double the data RAM usage to split it in chunks).

@MrBago

This comment has been minimized.

Contributor

MrBago commented Oct 10, 2013

Two ideas to think about related to this. I often try to put a try-finally clause around temporary files so that I know they're always cleaned up. If you want to do something a little more complex, we can think about creating an class that creates a temporary file on initialization and removes the file when the instance gets discarded. I don't know if this use-case warrants the complexity, but it's another option.

@matthew-brett

This comment has been minimized.

Member

matthew-brett commented Oct 10, 2013

nibabel has a context manager for this:

from nibabel.tmpdirs import InTemporaryDirectory
with InTemporaryDirectory():
      # write lots of files
@samuelstjean

This comment has been minimized.

Contributor

samuelstjean commented Oct 17, 2013

So should we try to include this in some manner with #255 ?

@ghost ghost assigned gabknight Oct 17, 2013

@gabknight

This comment has been minimized.

Contributor

gabknight commented Oct 17, 2013

No, better to do 2 PR. I will look at this one.

@Garyfallidis

This comment has been minimized.

Member

Garyfallidis commented Nov 4, 2013

This looks good to me and looking at it a test for memory would be a bit of an overkill. But we should monitor it with real datasets. I will have a look on what it does with the dipy examples.

@arokem

This comment has been minimized.

Member

arokem commented Jan 11, 2014

Closed through #264

@arokem arokem closed this Jan 11, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment