OutOfMemory error + memory leak #670
Comments
|
Wow, that's a lot of memory. What's your platform? I assume it's using ImageMagick? Other than recommending that you perhaps try OMETIFF.jl (and use |
|
Yes, looks like it's using ImageMagick. I'm on Ubuntu 16.04 (4.4.0-96-generic). Here's the problematic file. I figured out how to generate files that cause the memory leak. I have a Edit: I spoke to soon. When directly saved as a .tif, there is no memory blow-up, but |
|
I made a minimal example to illustrate the problem using a smaller file (222 MB): julia_memory_blowup.tif using Images
dataFile = "julia_memory_blowup.tif"
tifs = load(dataFile)
println("tifs uses ", sizeof(tifs)/1024^2, " MB")
However, system monitor shows that Julia is actually using 2.8 GB. I then tried using |
|
Sorry for the delay here, I had a grant deadline. The problem seems to be the ImageMagick C library: julia> using ImageMagick
julia> dataFile = "julia_memory_blowup.tif"
"julia_memory_blowup.tif"
julia> wand = ImageMagick.MagickWand()
ImageMagick.MagickWand(Ptr{Void} @0x0000000003b80210)
julia> ccall(ImageMagick.MagickReadImage[], Cint, (Ptr{Void}, Ptr{UInt8}), wand, dataFile)
1For me this is enough to exhibit the 2GB of RAM usage. Since this is a single Not sure what to do. One option would be to write a pure-Julia TIFF reader. Three repositories that might be relevant:
One can easily explore OMETIFF, even though it's not the default loader for "plain" tiff: julia> using FileIO
julia> dataFile = "julia_memory_blowup.tif"
"julia_memory_blowup.tif"
julia> img2 = load(File(format"OMETIFF", dataFile));
INFO: Recompiling stale cache file /home/tim/.julia/lib/v0.6/OMETIFF.ji for module OMETIFF.
Error encountered while loading "julia_memory_blowup.tif".
Fatal error:
ERROR: MethodError: Cannot `convert` an object of type String to an object of type FileIO.LoaderError
This may have arisen from a call to the constructor FileIO.LoaderError(...),
since type constructors fall back to convert methods.The error message suggests some (small) mistake in OMETIFF, but the bigger picture may be that the format may not be compatible. CC @tlnagy. |
|
Ooof. I'm not surprised that |
|
Hah. So that error is That's what I get for ignoring a gap in coverage of my tests. |
|
@timholy The status of OpenImageIO.jl is that it works for some simple cases but development is on hold for the moment - right now I'm trying to put what time I have into Base to make 1.0 as good as it can be :-) |
|
we might need to close this issue since it's already fixed |
|
@johnnychen94 just tested and julia_memory_blowup.tif still consumes > 2GB of memory |
|
It runs well in my machine: Ubuntu 16.04: julia> using Images
julia> @time tifs = load("julia_memory_blowup.tif");
13.358631 seconds (21.26 M allocations: 1.651 GiB, 5.86% gc time)
julia> println("tifs uses ", sizeof(tifs)/1024^2, " MB")
tifs uses 222.0 MBversionjulia> versioninfo()
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.1 (ORCJIT, haswell)Some related issues are fixed by #746: |
|
I don't think it has anything to do with show methods since I was able to replicate it in the REPL with no display. But @johnnychen94 is still mostly right, in that this doesn't appear to be a problem on the Julia side. The only thing we could likely do is develop a completely different image loading library. Since we're all volunteers here...contributions would be welcome! 😄 However, @tbenst it might be worth trying to |
|
Sorry I misunderstood the issue problem here. I thought it's related to |
|
@timholy I certainly appreciate all the amazing work you do on Julia in addition to being a neuroscientist! 😃 Thanks to that thread, I just found that calling |
|
An update on this: With Julia 1.6.0 and ImageIO, which uses TIFFImages made by @tlnagy, loading the 222MB file now takes 500MB memory on macOS and 400MB on Ubuntu so definitely a big improvement. (@v1.6) pkg> st ImageIO FileIO
Status `~/.julia/environments/v1.6/Project.toml`
[5789e2e9] FileIO v1.6.5
[82e4d734] ImageIO v0.5.3 |
|
I assume this is good to close. |
I'm having an issue that is related to #89. When loading a 1.3GB .tif file that contains a sequence of images, Julia eats up all available memory--21 GB of RAM--before I see
LoadError: OutOfMemoryError. In order to free up the memory, I have to end the julia process. The same .tif file opens fine in python using imageio.Let me know how I can be helpful in troubleshooting?
The text was updated successfully, but these errors were encountered: