Read MS file error?

[cnwangfeng]

Hi Tim,

I want to open MS file by function create_batchvisitility_from _ms. The MS file was gotten from EoR Blind challenge which is about 474MB. My computer has 32GB memory. However, the ARL reports out of memory:

*** mach_vm_map(size=18446744066256998400) failed (error code=3)
python(57221,0x11d8545c0) malloc: *** set a breakpoint in malloc_error_break to debug
Terminated: 15

I tracked the codes and error would occur at Line 835 of memory_data_models.py.

        desc = [('index', '>i8'),
                ('uvw', '>f8', (nants, nants, 3)),
                ('time', '>f8'),
                ('integration_time', '>f8'),
                ('vis', '>c16', (nants, nants, nchan, npol)),
                ('weight', '>f8', (nants, nants, nchan, npol)),
                ('imaging_weight', '>f8', (nants, nants, nchan, npol))]
        data = numpy.zeros(shape=[ntimes], dtype=desc)

nants =4704, nchan =1 , npol = 4, ntimes = 8

Or it's not a bug which means I should find a big memory server to run it.

Feng Wang

[timcornwell]

Hi, Can you give me access to the data? It’s asking to allocate 16 Exabytes of memory! Regards, Tim

…

On 11 Jan 2019, at 1:52 pm, Feng Wang ***@***.***> wrote: Hi Tim, I want to open MS file by function create_batchvisitility_from _ms. The MS file was gotten from EoR Blind challenge which is about 474MB. My computer has 32GB memory. However, the ARL reports out of memory: *** mach_vm_map(size=18446744066256998400) failed (error code=3) python(57221,0x11d8545c0) malloc: *** set a breakpoint in malloc_error_break to debug Terminated: 15 I tracked the codes and found that the possible error was in Line 835 of memory_data_models.py. desc = [('index', '>i8'), ('uvw', '>f8', (nants, nants, 3)), ('time', '>f8'), ('integration_time', '>f8'), ('vis', '>c16', (nants, nants, nchan, npol)), ('weight', '>f8', (nants, nants, nchan, npol)), ('imaging_weight', '>f8', (nants, nants, nchan, npol))] data = numpy.zeros(shape=[ntimes], dtype=desc) nants =4704, nchan =1 , npol = 4, ntimes = 8 Feng Wang — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#43>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADv3XXxn1vdje8BJoKZLTn-Xz16sOuc3ks5vCJcygaJpZM4Z7fvk>.

[cnwangfeng]

Hi, https://www.dropbox.com/sh/7nful5ib7boyyhg/AACgo6fp3RJgBC7qQWzxrooOa?dl=0&lst= I am using : eor_p_noise_p_fg_t_obsv_1000.0_hr.ms 407,086,215 bytes (407.3 MB on disk) for 89 items Feng

…

On 11 Jan 2019, 14:20 +0000, Tim Cornwell ***@***.***>, wrote: Hi, Can you give me access to the data? It’s asking to allocate 16 Exabytes of memory! Regards, Tim > On 11 Jan 2019, at 1:52 pm, Feng Wang ***@***.***> wrote: > > Hi Tim, > > I want to open MS file by function create_batchvisitility_from _ms. The MS file was gotten from EoR Blind challenge which is about 474MB. My computer has 32GB memory. However, the ARL reports out of memory: > > *** mach_vm_map(size=18446744066256998400) failed (error code=3) > python(57221,0x11d8545c0) malloc: *** set a breakpoint in malloc_error_break to debug > Terminated: 15 > > I tracked the codes and found that the possible error was in Line 835 of memory_data_models.py. > > desc = [('index', '>i8'), > ('uvw', '>f8', (nants, nants, 3)), > ('time', '>f8'), > ('integration_time', '>f8'), > ('vis', '>c16', (nants, nants, nchan, npol)), > ('weight', '>f8', (nants, nants, nchan, npol)), > ('imaging_weight', '>f8', (nants, nants, nchan, npol))] > data = numpy.zeros(shape=[ntimes], dtype=desc) > nants =4704, nchan =1 , npol = 4, ntimes = 8 > > Feng Wang > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub <#43>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADv3XXxn1vdje8BJoKZLTn-Xz16sOuc3ks5vCJcygaJpZM4Z7fvk>. > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[bnikolic]

Maybe a negative number has crept in somewhere so it is allocating two's complement of the right size?

[timcornwell]

It seems that there is a limitation of structured arrays that: ValueError: invalid shape in fixed-type tuple: dtype size in bytes must fit into a C int. The vis, weight and imaging weight together overflow this. Removing two of these is ok. So it’s very close to the limit. Not obvious what to do. I’ll have to ponder this. The problem has not been visible so far because the maximum number of antennas used has been probably 512. However, clearly the following will also overflow if nchan is big enough. ('vis', '>c16', (nants, nants, nchan, npol)),

…

On 11 Jan 2019, at 2:47 pm, bnikolic ***@***.***> wrote: Maybe a negative number has crept in somewhere so it is allocating two's complement of the right size? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#43 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADv3XS98IniKmpy2cMTNSeNohcUQRtQMks5vCKQJgaJpZM4Z7fvk>.

[fdulwich]

Hi Feng, Tim,

Thanks for sharing the link to the MS file. I had a look since I was curious, and I noticed that it's actually a concatenation of 21 single-channel OSKAR simulations, each with 224 stations. Unfortunately the CASA concat task that was used to generate it also duplicates rows in the ANTENNA table to make it very much longer, so it makes it look like there are (224 * 21 =) 4704 antennas, which is not the intention here. I guess it's not possible to assume that the number of rows in the ANTENNA table is the same as the actual number of antennas. Would it be possible to find the number of unique antenna positions to get this instead, or maybe if it's easier, use the total number of rows in the ANTENNA table divided by the total number of rows in the DATA_DESCRIPTION table? Someone with a better knowledge of what's allowed in the Measurement Set could say whether that's a good idea or not - I'm sure Tim would know.

(Incidentally, this also explains why you ran out of memory, as the ARL data structure would need almost 50 GB if it really was with 4704 antennas!)

Cheers,
Fred

[cnwangfeng]

Hi all,

I have further tracked the codes and I agreed with Fred that the wrong values of ANTENNA number would the reason for the error.

Feng

[timcornwell]

I would expect that this could be fixed on the CASA side, probably with some python access of the MSes. Thanks Fred!

…

On 11 Jan 2019, at 4:22 pm, Feng Wang ***@***.***> wrote: Hi all, I have further tracked the codes and I agreed with Fred that the wrong values of ANTENNA number would the reason for the error. Feng — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#43 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADv3XYH-n_wo6FJVdymuZENXMFF0fO9oks5vCLpRgaJpZM4Z7fvk>.

[cnwangfeng]

I suppose to close this issue because I think there are some obvious mistakes in that specified MS file. It is not worth to modify ARL codes to read the data from a wrong MS file.

1. Two fields, i.e., anntena1 and anntenna2, written in the main data, are greater than the number of antennas.
2. Wrong data concatenation.