[DOC] Add note about decreasing memory usage

[dPys]

...by decompressing nii's per issue #2218 at @GaelVaroquaux 's request.

[GaelVaroquaux]

LGTM. Thank you very much. It's a pity that you skipped CI, as we would have looked at the docs rendered by the CI.

Another review by someone else?

[kchawla-pi]

I wouldn't wanna merge this without looking at circleci, maybe I can start it manually once I have a stable net connection

…

On Wed, Nov 20, 2019, 13:19 Gael Varoquaux ***@***.***> wrote: LGTM. Thank you very much. It's a pity that you skipped CI, as we would have looked at the docs rendered by the CI. Another review by someone else? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2223?email_source=notifications&email_token=AB6SXRH7PMR45BZ4RQCADULQUUTM7A5CNFSM4JPO2FG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEERZL5Y#issuecomment-555980279>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB6SXRE3U7UJV3CB5GSWYNTQUUTM7ANCNFSM4JPO2FGQ> .

[kchawla-pi]

I can't force build it directly, however there is a workaround.
Merging this PR into the upstream test-circleci branch without the commit log and pushing it will build it.
You can check i tout here once it is done:
https://circleci.com/gh/kchawla-pi/nilearn/644

[kchawla-pi]

Thanks for the force push @dPys

[dPys]

Already one step ahead of you @kchawla-pi !

I had some further text I wanted to add so this may have turned out to be a blessing in disguise.

[codecov]

Codecov Report

Merging #2223 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2223   +/-   ##
=======================================
  Coverage   92.36%   92.36%           
=======================================
  Files         149      149           
  Lines       18991    18991           
  Branches     2307     2307           
=======================================
  Hits        17541    17541           
  Misses        931      931           
  Partials      519      519

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0941897...bf80743. Read the comment docs.

[emdupre]

Is it worth explaining why this is true ?

[dPys]

My sense is that it may not be worth getting too much into the weeds about why. @GaelVaroquaux ?

[GaelVaroquaux]

I agree. Less is more

[dPys]

@GaelVaroquaux -- one final question from me on this:

why not run:

def make_uncompressed_temp(func_file):
    if func_file.endswith('.nii.gz'):
        import tempfile
        import gzip
        import os
        temp_path = "{}/{}.nii".format(tempfile.mkdtemp(), os.path.basename(func_file))
        decomp_func = gzip.open(path)
        with open(temp_path, 'wb') as outfile:
            outfile.write(decomp_func.read())
        decomp_func.close()
        outfile.close()
    else:
        temp_path = func_file
    return temp_path

every time as a default precursor to all Nilearn functions, subsequently deleting the temporary file after operations are performed? Wouldn't that reduce the memory burden universally?

[GaelVaroquaux]

Many reasons why this would be fragile: - It requires a system to clean up the file after, and these are challenging to make fool proof (think: crashing in the middle of reading) - It will slow down the loading (though not much) - It will take disk space, and sometimes there is none available (many processes loading in the same time), or mkdtemp might not return the right place (shared systems not well configured) However, what this gets to is the tradeoff between memory usage and disk usage, which is a classic one. The right way to do this tradeoff would be to increase the swap. Unfortunately, in linux the swap cannot be easily increased dynamically.

[GaelVaroquaux]

LGTM, thank you

[GaelVaroquaux]

The docs can be viewed here: https://5013-1235740-gh.circle-artifacts.com/0/home/circleci/project/doc/_build/html/manipulating_images/input_output.html#niimg-like-objects

They look good. I am merging. I just realized that I'd like this title to be bold, but I'll fix it after merge.

Thanks a lot!

[dPys]

Many reasons why this would be fragile: - It requires a system to clean up the file after, and these are challenging to make fool proof (think: crashing in the middle of reading) - It will slow down the loading (though not much) - It will take disk space, and sometimes there is none available (many processes loading in the same time), or mkdtemp might not return the right place (shared systems not well configured) However, what this gets to is the tradeoff between memory usage and disk usage, which is a classic one. The right way to do this tradeoff would be to increase the swap. Unfortunately, in linux the swap cannot be easily increased dynamically.

Yep, well a useful hack anyway even if it doesn't scale :-) I did think about addressing this problem with swap but without being able to modify swappiness nor an interest in getting into the weeds with it, eventually just decided to try tmp.

Anyway, glad to see this PR merged.

Cheers