Error when using `antsApplyTransforms` to large data #564

david-hoffman · 2018-04-10T21:41:13Z

I get this error when using antsApplyTransforms to large data.

** ERROR: NWAD: wrote only 0 of 3650505600 bytes to file

The text was updated successfully, but these errors were encountered:

cookpa · 2018-04-10T21:43:43Z

This appears to be a NIFTI library error. Are you sure you have sufficient disk space to write the file?

david-hoffman · 2018-04-10T21:45:29Z

@cookpa I believe I do (~8TB free)

cookpa · 2018-04-10T22:01:01Z

OK, then I think the error is caused by integer overflow:

https://github.com/InsightSoftwareConsortium/ITK/blob/a652fd46a9b47acf68f290ca41aa8e5f8a38a56d/Modules/ThirdParty/MINC/src/libminc/nifti/nifti1_io.c#L5169

if size_t is 32 bits, then it would not be able to represent the size of the file.

Do you have the latest ANTs? How did you compile?

david-hoffman · 2018-04-10T22:55:22Z

I compiled earlier today following the instructions here: https://github.com/ANTsX/ANTs/wiki/Compiling-ANTs-on-Windows-10

david-hoffman · 2018-04-10T23:34:03Z

There's no error if I run the command on a linux box. Something strange about WSL.

dorianps · 2018-04-10T23:52:33Z

If this is a memory issue, you can try expanding the swap size in Windows. That should be reflected in WSL after a restart. Can't help you with exact commands right now, don't remember them off the top of my head, but in one occasion I had to do that to get passed a limited memory issue. Dorian

…

On Tue, Apr 10, 2018, 19:34 David Hoffman ***@***.***> wrote: There's no error if I run the command on a linux box. Something strange about WSL. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#564 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIqafcku2otFI4Ft08L_6HsseEgt-yDlks5tnUFtgaJpZM4TPCyP> .

david-hoffman · 2018-04-11T00:53:29Z

@dorianps I don't think it's a swap issue, I have plenty of RAM (512 GB). Plus, the swap in WSL is 2GB and the swap on the Linux box I used is 4GB.

cookpa · 2018-04-11T01:11:50Z

Related?

microsoft/WSL#597

It seems the WSL was setting the Linux swap size to a small value, resulting in memory allocation failures. The thread above suggests this is fixed in the "creators update", which might be worth a try if you don't already have it.

dorianps · 2018-04-11T03:16:09Z

The command to check memory and swap sizes in WSL should be: free -m My MS Surface with 8Gb RAM and Win10pro 1709 is showing this (values in Mb):

…

On Tue, Apr 10, 2018 at 9:11 PM, Philip Cook ***@***.***> wrote: Related? microsoft/WSL#597 <microsoft/WSL#597> It seems the WSL was setting the Linux swap size to a small value, resulting in memory allocation failures. The thread above suggests this is fixed in the "creators update", which might be worth a try if you don't already have it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#564 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIqafeT_3SS00O5rKhtq19O6h6Xc1htdks5tnVhYgaJpZM4TPCyP> .

david-hoffman · 2018-04-11T19:08:53Z

This is what I get, again I don't think it's a memory issue

hoffmand@HOFFMAND-WW1:~$ free -m
             total       used       free     shared    buffers     cached
Mem:        524204      82316     441888         17         33        184
-/+ buffers/cache:      82098     442105
Swap:         8192         21       8170

dorianps · 2018-04-11T19:15:26Z

Just wondering, do you have the zlib library installed? It doesn't seem to be among your installation instructions, although without it the error should have appeared during the installation process.

…

On Wed, Apr 11, 2018 at 3:08 PM, David Hoffman ***@***.***> wrote: This is what I get, again I don't think it's a memory issue ***@***.***:~$ free -m total used free shared buffers cached Mem: 524204 82316 441888 17 33 184 -/+ buffers/cache: 82098 442105 Swap: 8192 21 8170 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#564 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIqafb_daxqLkxRbKkiBqZEmqxxJjkljks5tnlTGgaJpZM4TPCyP> .

dorianps · 2018-04-11T19:18:16Z

Sorry, just saw you have listed it there at the end.

…

On Wed, Apr 11, 2018 at 3:14 PM, Dorian Pustina ***@***.***> wrote: Just wondering, do you have the zlib library installed? It doesn't seem to be among your installation instructions, although without it the error should have appeared during the installation process. On Wed, Apr 11, 2018 at 3:08 PM, David Hoffman ***@***.***> wrote: > This is what I get, again I don't think it's a memory issue > > ***@***.***:~$ free -m > total used free shared buffers cached > Mem: 524204 82316 441888 17 33 184 > -/+ buffers/cache: 82098 442105 > Swap: 8192 21 8170 > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#564 (comment)>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/AIqafb_daxqLkxRbKkiBqZEmqxxJjkljks5tnlTGgaJpZM4TPCyP> > . >

david-hoffman · 2018-04-11T19:19:40Z

@dorianps it did show as an error during compilation so I installed it (I also updated the instructions here).

I have run into problems with some python libraries under windows because the default integer is uint32 not uint64. I didn't think this would extend to WSL, but it may have.

dorianps · 2018-04-11T19:20:08Z

The only other thing I can guess is some sort of lack of permissions to write on the drive.

…

On Wed, Apr 11, 2018 at 3:17 PM, Dorian Pustina ***@***.***> wrote: Sorry, just saw you have listed it there at the end. On Wed, Apr 11, 2018 at 3:14 PM, Dorian Pustina ***@***.***> wrote: > Just wondering, do you have the zlib library installed? It doesn't seem > to be among your installation instructions, although without it the error > should have appeared during the installation process. > > On Wed, Apr 11, 2018 at 3:08 PM, David Hoffman ***@***.***> > wrote: > >> This is what I get, again I don't think it's a memory issue >> >> ***@***.***:~$ free -m >> total used free shared buffers cached >> Mem: 524204 82316 441888 17 33 184 >> -/+ buffers/cache: 82098 442105 >> Swap: 8192 21 8170 >> >> — >> You are receiving this because you were mentioned. >> Reply to this email directly, view it on GitHub >> <#564 (comment)>, or mute >> the thread >> <https://github.com/notifications/unsubscribe-auth/AIqafb_daxqLkxRbKkiBqZEmqxxJjkljks5tnlTGgaJpZM4TPCyP> >> . >> > >

david-hoffman · 2018-04-11T19:22:49Z

@dorianps I doubt that, I had just run antsRegistration in the same terminal with no issues (albeit on smaller data).

cookpa · 2018-04-11T19:25:33Z

Thanks for the Wiki update @david-hoffman .

From the WSL issue above,

By default native Linux will not allowing you to map an address range larger than your swap file size.

As I read it, the swap size is the limiting factor here. According to your free output above this is 8Gb, but earlier you mentioned it being 2 Gb. Did you change it? Can you increase it further and see if the problem goes away?

ptsii · 2019-02-09T04:52:54Z

Did anyone solve this? I just got the same issue. I don't think it is memory, as it happened with one file (of several I was trying to save) that is relatively small:
** ERROR: NWAD: wrote only 0 of 3612672 bytes to file
(that can't be big enough to be a swap problem, can it?)

zz10001 · 2019-11-08T01:54:37Z

Did anyone solve this? I just got the same issue. I don't think it is memory, as it happened with one file (of several I was trying to save) that is relatively small:
** ERROR: NWAD: wrote only 0 of 3612672 bytes to file
(that can't be big enough to be a swap problem, can it?)

Have you sovle it? I have met the same trouble like this.

but it really enough memrory and exist this file

dorianps · 2019-11-08T02:14:10Z

Is this still on WSL? I am not using WSL much anymore, but if you are interested there are recent docker images and scripts that run in Windows. https://github.com/dorianps/docker

…

On Thu, Nov 7, 2019, 8:54 PM zz10001 ***@***.***> wrote: Did anyone solve this? I just got the same issue. I don't think it is memory, as it happened with one file (of several I was trying to save) that is relatively small: ** ERROR: NWAD: wrote only 0 of 3612672 bytes to file (that can't be big enough to be a swap problem, can it?) Have you sovle it? I have met the same trouble like this. [image: image] <https://user-images.githubusercontent.com/39954774/68442974-a7b97100-020d-11ea-9d18-c2495f3810ed.png> but it really enough memrory and exist this file — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#564?email_source=notifications&email_token=ACFJU7MN5TLLYQBDE2SCYMDQSTBGBA5CNFSM4EZ4FSH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDOOTPI#issuecomment-551348669>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACFJU7NZKPOTYO5L53EETULQSTBGBANCNFSM4EZ4FSHQ> .

zz10001 · 2019-11-08T02:17:18Z

Sorry, It not on WSL, it occurs when I crop image to 64128128 use simpleitk

cookpa · 2019-11-08T21:39:54Z

I think there's probably more than one way to get this error. If the buffer can only write a subset of the data, this error will be generated. But the numbers reported may not be accurate in case of overflow or if there are I/O errors other than a short write.

If you're seeing this error with ITK outside of ANTs then you might want to ask on the ITK forum.

https://discourse.itk.org

ptsii · 2019-11-08T21:52:53Z

I’ve gotten this error when the disk was too full and the file could not be written.

…

-Tom

On Nov 8, 2019, at 4:39 PM, Philip Cook ***@***.***> wrote: I think there's probably more than one way to get this error. If the buffer can only write a subset of the data, this error will be generated. But the numbers reported may not be accurate in case of overflow or if there are I/O errors other than a short write. If you're seeing this error with ITK outside of ANTs then you might want to ask on the ITK forum. https://discourse.itk.org <https://discourse.itk.org/> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#564?email_source=notifications&email_token=ABHLFXKALPVA76CMUQOVQNDQSXMC7A5CNFSM4EZ4FSH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDTNP6Q#issuecomment-551999482>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHLFXO2ZOBA4NFLKYQHJ7LQSXMC7ANCNFSM4EZ4FSHQ>.

zz10001 · 2019-11-09T06:47:26Z

Thanks for your help.

david-hoffman · 2020-03-08T16:14:52Z

Dead issue

tashrifbillah · 2020-06-09T16:56:12Z

Same problem on Cent OS 7 Linux:

11821 Total elapsed time: 534.8
11822 ** ERROR: NWAD: wrote only 68812448 of 92274688 bytes to file
11823 ** ERROR: NWAD: wrote only 129105568** ERROR: NWAD: wrote only 392864 of 276824064 bytes to file
11824 Transform reader for /tmp/tmp6a11_1hd/warp1.nii.gz caught an ITK exception:
11825
11826 itk::ExceptionObject (0x351f3d0)
11827 Location: "unknown"
11828 File: /data/pnl/soft/pnlpipe3/pnlpipe/soft_dir/ANTs-build/ITKv5/Modules/IO/TransformBase/include/itkTransformFileReader.hxx
11829 Line: 129
11830 Description: itk::ERROR: TransformFileReaderTemplate(0x3512e80): Could not create Transform IO object for reading file /tmp/tmp6      a11_1hd/warp1.nii.gz
11831   Tried to create one of the following:
11832     HDF5TransformIOTemplate
11833     HDF5TransformIOTemplate
11834     MatlabTransformIOTemplate
11835     MatlabTransformIOTemplate
11836     TxtTransformIOTemplate
11837     TxtTransformIOTemplate
11838   You probably failed to set a file suffix, or
11839     set the suffix to an unsupported type.
11840
11841
11842 ** ERROR: NWAD: wrote only 67337888 of 92274688 bytes to file
11843 ** ERROR: NWAD: wrote only 64519840 of 92274688 bytes to file
11844 ** ERROR: NWAD: wrote only 64224928 of 92274688 bytes to file
11845 ** ERROR: NWAD: wrote only 50888352 of 92274688 bytes to file
11846 ** ERROR: NWAD: wrote only 62881440 of 92274688 bytes to file
11847 ** ERROR: NWAD: wrote only 65732256 of 92274688 bytes to file
11848 ** ERROR: NWAD: wrote only 63995552 of 92274688 bytes to file
11849 ** ERROR: NWAD: wrote only 44203680 of 92274688 bytes to file
11850 ** ERROR: NWAD: wrote only 63110816 of 92274688 bytes to file
11851 ** ERROR: NWAD: wrote only 63078048 of 92274688 bytes to file

On the following host:

GiB Mem :    251.6 total,    111.1 free,     79.1 used,     61.5 buff/cache
GiB Swap:     14.6 total,      6.8 free,      7.8 used.    169.0 avail Mem

My NIFTI data type:

data_type       INT16
dim0            3
dim1            176
dim2            256
dim3            256

As @cookpa mentioned, I think it is related to my little swap memory.

Edit:
My issue was space shortage in my /tmp/ directory.

caixh39 · 2021-08-31T02:38:58Z

It seem that each nii.gz file registrated will generate some temp files in /tmp/*Warp.nii.gz and Affine.mat in ubuntu system, as below:

david-hoffman closed this as completed Mar 8, 2020

vkyprmr mentioned this issue Mar 19, 2021

Error: NWAD while doing ants.registration #1164

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error when using `antsApplyTransforms` to large data #564

Error when using `antsApplyTransforms` to large data #564

david-hoffman commented Apr 10, 2018

cookpa commented Apr 10, 2018

david-hoffman commented Apr 10, 2018

cookpa commented Apr 10, 2018

david-hoffman commented Apr 10, 2018

david-hoffman commented Apr 10, 2018

dorianps commented Apr 10, 2018 via email

david-hoffman commented Apr 11, 2018

cookpa commented Apr 11, 2018

dorianps commented Apr 11, 2018 via email

david-hoffman commented Apr 11, 2018

dorianps commented Apr 11, 2018 via email

dorianps commented Apr 11, 2018 via email

david-hoffman commented Apr 11, 2018

dorianps commented Apr 11, 2018 via email

david-hoffman commented Apr 11, 2018

cookpa commented Apr 11, 2018

ptsii commented Feb 9, 2019

zz10001 commented Nov 8, 2019

dorianps commented Nov 8, 2019 via email

zz10001 commented Nov 8, 2019

cookpa commented Nov 8, 2019

ptsii commented Nov 8, 2019 via email

zz10001 commented Nov 9, 2019

david-hoffman commented Mar 8, 2020

tashrifbillah commented Jun 9, 2020 •

edited

Loading

caixh39 commented Aug 31, 2021 •

edited

Loading

Error when using antsApplyTransforms to large data #564

Error when using antsApplyTransforms to large data #564

Comments

david-hoffman commented Apr 10, 2018

cookpa commented Apr 10, 2018

david-hoffman commented Apr 10, 2018

cookpa commented Apr 10, 2018

david-hoffman commented Apr 10, 2018

david-hoffman commented Apr 10, 2018

dorianps commented Apr 10, 2018 via email

david-hoffman commented Apr 11, 2018

cookpa commented Apr 11, 2018

dorianps commented Apr 11, 2018 via email

david-hoffman commented Apr 11, 2018

dorianps commented Apr 11, 2018 via email

dorianps commented Apr 11, 2018 via email

david-hoffman commented Apr 11, 2018

dorianps commented Apr 11, 2018 via email

david-hoffman commented Apr 11, 2018

cookpa commented Apr 11, 2018

ptsii commented Feb 9, 2019

zz10001 commented Nov 8, 2019

dorianps commented Nov 8, 2019 via email

zz10001 commented Nov 8, 2019

cookpa commented Nov 8, 2019

ptsii commented Nov 8, 2019 via email

zz10001 commented Nov 9, 2019

david-hoffman commented Mar 8, 2020

tashrifbillah commented Jun 9, 2020 • edited Loading

caixh39 commented Aug 31, 2021 • edited Loading

Error when using `antsApplyTransforms` to large data #564

Error when using `antsApplyTransforms` to large data #564

tashrifbillah commented Jun 9, 2020 •

edited

Loading

caixh39 commented Aug 31, 2021 •

edited

Loading