Cubical stuck in process: waiting for I/O on tile

[viralp]

Hi, My cubical job is not progressing and halt with the following message

INFO 02:28:08 - solver [14.7/235.8 22.7/689.8 69.1Gb] D0T22F3 (end solve) G: 50 iters, conv 0.00%, g/fl 9.17% [MISSING:17(14.17%) LOWSNR:11(9.17%)], d/fl 14.17%, PGE 0.175 (flagged LOWSNR:11); DD: 68 iters, conv 100.00%, d/fl 16.67%, PGE 0.0 0.0488; G: 50 iters, conv 0.00%, g/fl 9.17% [MISSING:17(14.17%) LOWSNR:11(9.17%)], d/fl 14.17%, PGE 0.172 (flagged LOWSNR:11); DD: 29 iters, conv 100.00%, d/fl 16.67%, PGE 0.0 0.0488, stall 85.78%, chi^2 0.6512 -> 0.6491 (1.011), noise 0.296 -> 0.236
INFO 02:28:18 - solver [14.8/235.8 22.7/689.9 69.2Gb] D0T22F3 has 50826 (0.14%) new data flags: gain flags LOWSNR:11(4.58%) (4.58% total)
INFO 02:28:19 - main [1.1/222.3 7.3/676.4 69.2Gb] finished processing tile 11/24
INFO 02:28:19 - main [1.1/222.3 7.3/676.4 69.2Gb] waiting for I/O on tile 12/24

ddcal_0.log
ddcal_0.parset.txt

[o-smirnov]

On stevie or where?

[viralp]

On the ike

[bennahugo]

Its a bug in the application of E-jones according to your log. I'm not sure under which circumstances this becomes a non-one-dimensional array though? INFO 02:15:33 - FITSBeamInterpolat [x01] [39.2/329.3 58.1/774.4 74.6Gb] computing beam sample times for 204960 timeslots INFO 02:15:33 - FITSBeamInterpolat [x01] [39.2/329.3 58.1/774.4 74.6Gb] DtBeamMin=5.00 min results in 3 samples INFO 02:15:33 - FITSBeamInterpolat [x01] [39.2/329.3 58.1/774.4 74.6Gb] FITSParAngleIncrement=5.00 deg results in 2 samples INFO 02:15:33 - main [x01] [39.2/329.3 58.1/774.4 74.6Gb] I/O handler for load 12 save 10 failed with exception: operands could not be broadcast together with shapes (3,) (2,) INFO 02:15:33 - main [x01] [39.2/329.3 58.1/774.4 74.6Gb] Traceback (most recent call last): File "/home/parekh/CubiCal/cubical/workers.py", line 469, in _io_handler tile.load(load_model=load_model) File "/home/parekh/CubiCal/cubical/data_handler/ms_tile.py", line 1014, in load model = subset.load_ddfacet_models(uvwco, loaded_models, model_source, cluster, imod, idir, model_type) File "/home/parekh/CubiCal/cubical/data_handler/ms_tile.py", line 163, in load_ddfacet_models self.time_col) File "/home/parekh/CubiCal/cubical/degridder/DDFacetSim.py", line 564, in simulate provider=dh.degrid_opts["BeamModel"]) File "/home/parekh/CubiCal/cubical/degridder/DDFacetSim.py", line 364, in __load_jones t0 = sample_times - dt ValueError: operands could not be broadcast together with shapes (3,) (2,)

…

On Tue, Jan 5, 2021 at 5:50 AM Viral Parekh ***@***.***> wrote: On the ike — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#427 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4RE6VAJWGQKZDXPOYVQIDSYKD7LANCNFSM4VTK7EXA> .

-- -- Benjamin Hugo PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[bennahugo]

Actually I see the bug: - the timestamp should be modified by an infinitesimal amount if the size is leq 1, otherwise - the last dt should be set to dt = np.zeros_like(sample_times) dt[:-1] = 0.5 * (sample_times[1:] - sample_times[:-1]) dt[-1] = dt[-2] Could you make the modification Viral

…

On Tue, Jan 5, 2021 at 9:05 AM Benna Hugo ***@***.***> wrote: Its a bug in the application of E-jones according to your log. I'm not sure under which circumstances this becomes a non-one-dimensional array though? INFO 02:15:33 - FITSBeamInterpolat [x01] [39.2/329.3 58.1/774.4 74.6Gb] computing beam sample times for 204960 timeslots INFO 02:15:33 - FITSBeamInterpolat [x01] [39.2/329.3 58.1/774.4 74.6Gb] DtBeamMin=5.00 min results in 3 samples INFO 02:15:33 - FITSBeamInterpolat [x01] [39.2/329.3 58.1/774.4 74.6Gb] FITSParAngleIncrement=5.00 deg results in 2 samples INFO 02:15:33 - main [x01] [39.2/329.3 58.1/774.4 74.6Gb] I/O handler for load 12 save 10 failed with exception: operands could not be broadcast together with shapes (3,) (2,) INFO 02:15:33 - main [x01] [39.2/329.3 58.1/774.4 74.6Gb] Traceback (most recent call last): File "/home/parekh/CubiCal/cubical/workers.py", line 469, in _io_handler tile.load(load_model=load_model) File "/home/parekh/CubiCal/cubical/data_handler/ms_tile.py", line 1014, in load model = subset.load_ddfacet_models(uvwco, loaded_models, model_source, cluster, imod, idir, model_type) File "/home/parekh/CubiCal/cubical/data_handler/ms_tile.py", line 163, in load_ddfacet_models self.time_col) File "/home/parekh/CubiCal/cubical/degridder/DDFacetSim.py", line 564, in simulate provider=dh.degrid_opts["BeamModel"]) File "/home/parekh/CubiCal/cubical/degridder/DDFacetSim.py", line 364, in __load_jones t0 = sample_times - dt ValueError: operands could not be broadcast together with shapes (3,) (2,) On Tue, Jan 5, 2021 at 5:50 AM Viral Parekh ***@***.***> wrote: > On the ike > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#427 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AB4RE6VAJWGQKZDXPOYVQIDSYKD7LANCNFSM4VTK7EXA> > . > -- -- Benjamin Hugo PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

-- -- Benjamin Hugo PhD. student, Centre for Radio Astronomy Techniques and Technologies Department of Physics and Electronics Rhodes University Junior software developer Radio Astronomy Research Group South African Radio Astronomy Observatory Black River Business Park Observatory Cape Town

[bennahugo]

Ok I've reproduced this. @viralp says the fix I proposed does not work so I will debug this further now