OverflowError in remove_background

[rh66]

Hello,
Could someone teach me how to call "remove_background"?
I was able to call remove_backgound with following calls, when I was running the tomopy last release (May 2014), and the data_exchanger cloned in last summer:

d = tomopy.xtomo_dataset(log='debug')
d.dataset(data, white, dark, theta)
d.zinger_removal(median_width=10)
d.normalize()
d.stripe_removal(sigma=0.4) 
d.optimize_center()

d.gridrec()
d.remove_background()

Recently, I would like to use the tilte_correct feature and cloned both tomopy (on Mar 19, 2015) and data_exchanger (on April 3, 2015), and run the same script above, and I got "OverflowError, array too large" (copied in following). Was it because I did not call the remove_background correctly? Or because some environment setting wrong on my computer system?

Thanks!
Rong

---

OverflowError Traceback (most recent call last)
/home/rh66/workspace/tomo/siegmund/test_new_tomopy.py in ()
91
92 if name == "main":
---> 93 main()
94

/home/rh66/workspace/tomo/siegmund/test_new_tomopy.py in main()
64 #d.center=99.5
65 d.gridrec()
---> 66 d.remove_background()
67
68

/nfs/opt/python2.7/packages/tomopy/xtomo/xtomo_postprocess.pyc in _remove_background(self, num_cores, chunk_size, overwrite)
102 _axis = 0 # Slice axis
103 data_recon = distribute_jobs(self.data_recon, _func, _args, _axis,
--> 104 num_cores, chunk_size)
105
106 # Update provenance.

/nfs/opt/python2.7/packages/tomopy/tools/multiprocess_shared.pyc in distribute_jobs(data, func, args, axis, num_cores, chunk_size)
92 arg += [(range(ind_start, ind_end), data.shape, args)]
93
---> 94 shared_arr = mp.Array(ctypes.c_float, data.size) # takes time
95 arr = tonumpyarray(shared_arr, data.shape)
96 arr[:] = data

/opt/rh/python27/root/usr/lib64/python2.7/multiprocessing/init.pyc in Array(typecode_or_type, size_or_initializer, *_kwds)
258 '''
259 from multiprocessing.sharedctypes import Array
--> 260 return Array(typecode_or_type, size_or_initializer, *_kwds)
261
262 #

/opt/rh/python27/root/usr/lib64/python2.7/multiprocessing/sharedctypes.pyc in Array(typecode_or_type, size_or_initializer, **kwds)
113 if kwds:
114 raise ValueError('unrecognized keyword argument(s): %s' % kwds.keys())
--> 115 obj = RawArray(typecode_or_type, size_or_initializer)
116 if lock is False:
117 return obj

/opt/rh/python27/root/usr/lib64/python2.7/multiprocessing/sharedctypes.pyc in RawArray(typecode_or_type, size_or_initializer)
80 type_ = typecode_to_type.get(typecode_or_type, typecode_or_type)
81 if isinstance(size_or_initializer, (int, long)):
---> 82 type_ = type_ * size_or_initializer
83 obj = new_value(type)
84 ctypes.memset(ctypes.addressof(obj), 0, ctypes.sizeof(obj))

OverflowError: array too large

[dgursoy]

Have you tried with a smaller data array?

[rh66]

Hello Doga,

The full image size is 2560 by 2160. I initially ran 100 slices.

I did two tests after reading your email:

1. using a set of images with size of 1143 by 1040, 100 slices;

2. using the full image size of 2560 by 2160, but reduce slice number from 100 to 10.

In both cases, the remove_background finished without error!

My situation is: I need to process the full size of the images because the large size of the sample. I also need to correct the image tilt (I am thinking of using "d.correct_tilt") which I assume I have to use a reasonable large number of slices for the angle correction, right? What do you suggest?

By the way, I remember sometime before I was told to use only 180 degree of data. Can we process data with larger rotation angle now?

Thanks a lot.

Rong

---

From: Doga Gursoy notifications@github.com
Sent: Thursday, April 9, 2015 9:54 AM
To: tomopy/tomopy
Cc: Rong Huang
Subject: Re: [tomopy] OverflowError in remove_background (#73)

Have you tried with a smaller data array?

Reply to this email directly or view it on GitHubhttps://github.com//issues/73#issuecomment-91238754.

[dgursoy]

As I remember, the function requires a lot of memory to process, so unfortunately you can't process all images at once. But you can apply it iteratively.

It's been a while I haven't used or checked correct_tilt. So I can't comment on that much. But I'll try to look into that.

You can sure process data with any angle. Just define your theta correctly.

[rh66]

Dear Doga,

If I understand you correctly, increase my PC RAM should be helpful for this issue, right?

I guess somewhere tomopy probably changed the way of how the processes were handled. While I use the same PC to process the same data, the older tomopy works fine, but the currently cloned tomopy does not.

Thanks a lot for the help!

Best regards,

Rong

---

From: Doga Gursoy notifications@github.com
Sent: Friday, April 10, 2015 5:58 PM
To: tomopy/tomopy
Cc: Rong Huang
Subject: Re: [tomopy] OverflowError in remove_background (#73)

As I remember, the function requires a lot of memory to process, so unfortunately you can't process all images at once. But you can apply it iteratively.

It's been a while I haven't used or checked correct_tilt. So I can't comment on that much. But I'll try to look into that.

You can sure process data with any angle. Just define your theta correctly.

Reply to this email directly or view it on GitHubhttps://github.com//issues/73#issuecomment-91703821.

[dgursoy]

Of course, but you may still need iterations for the full dataset. I suggest you to monitor the memory consumption for a specified number of slices. From that you can estimate roughly the required memory for the whole data.

I am not sure exactly what has changed. Maybe multiprocessing could have caused that. You can try remove_background(num_cores=1). It'll take more time but may save you some memory.

[rh66]

Dear Doga,

Thank you for the very helpful discussions!

Best regards,

Rong

---

From: Doga Gursoy notifications@github.com
Sent: Saturday, April 11, 2015 12:58 AM
To: tomopy/tomopy
Cc: Rong Huang
Subject: Re: [tomopy] OverflowError in remove_background (#73)

Of course, but you may still need iterations for the full dataset. I suggest you to monitor the memory consumption for a specified number of slices. From that you can estimate roughly the required memory for the whole data.

I am not sure exactly what has changed. Maybe multiprocessing could have caused that. You can try remove_background(num_cores=1). It'll take more time but may save you some memory.

Reply to this email directly or view it on GitHubhttps://github.com//issues/73#issuecomment-91761255.

[dgursoy]

You're welcome Rong.