job started at 2020-09-20 10:17:07.035392 Instantiating container [calibrate_1st_ms_with_cubical-13963662854276816005772266725404]. The container ID is printed below. running docker create --user 0:0 --shm-size 1gb -v /home/arnab/HI_abs_cubical/.stimela_workdir-16005772266706834/stimela_parameter_files/calibrate_1st_ms_with_cubical-13963662854276816005772266725404.json:/stimela_mount/configfile:ro -v /home/arnab/stimela_env/venv3/lib/python3.6/site-packages/stimela/cargo/cab/cubical/src:/stimela_mount/code:ro -v /home/arnab/HI_abs_cubical/.stimela_workdir-16005772266706834/passwd:/etc/passwd:rw -v /home/arnab/HI_abs_cubical/.stimela_workdir-16005772266706834/group:/etc/group:rw -v /home/arnab/stimela_env/venv3/bin/stimela_runscript:/docker_run:ro -v /home/arnab/HI_abs_cubical/msdir:/stimela_mount/msdir:rw -v /home/arnab/HI_abs_cubical/input:/stimela_mount/input:ro -v /home/arnab/HI_abs_cubical/output:/stimela_mount/output:rw -v /home/arnab/HI_abs_cubical/output/tmp:/stimela_mount/output/tmp:rw -e CONFIG=/stimela_mount/configfile -e HOME=/stimela_mount/output -e STIMELA_MOUNT=/stimela_mount -e MSDIR=/stimela_mount/msdir -e INPUT=/stimela_mount/input -e OUTPUT=/stimela_mount/output -e TMPDIR=/stimela_mount/output/tmp --rm -w /stimela_mount/output --name calibrate_1st_ms_with_cubical-13963662854276816005772266725404 stimela/cubical:1.6.7 /docker_run 5325c260d0cf887f9b8bbbc1337fc2c5d3287b7098250b3c1b37bbebfb3b43fe Starting container [calibrate_1st_ms_with_cubical-13963662854276816005772266725404]. Timeout set to -1. The container ID is printed below. running docker start -a calibrate_1st_ms_with_cubical-13963662854276816005772266725404 2020-09-20 04:47:08: Initial memory state: 2020-09-20 04:47:08: total used free shared buff/cache available 2020-09-20 04:47:08: Mem: 251G 49G 1.0G 660K 201G 200G 2020-09-20 04:47:08: Swap: 30G 1.6G 28G 2020-09-20 04:47:08: Running gocubical --sol-jones G --data-ms /stimela_mount/msdir/J1521_5508.ms --data-column DATA --data-time-chunk 120 --data-freq-chunk 512 --data-single-chunk None --sel-taql None --model-list MODEL_DATA --model-ddes never --model-beam-l-axis None --model-beam-m-axis None --model-pa-rotate 0 --model-feed-rotate 0 --montblanc-dtype float --weight-column WEIGHT --madmax-enable 0 --madmax-estimate corr --madmax-plot 0 --madmax-threshold 0,20 --sel-diag 0 --sol-term-iters 50,50,50 --bbc-save-to /stimela_mount/output/ --dist-ncpu 24 --dist-max-chunks 4 --out-name /stimela_mount/output/selfcal_HI_abs --out-overwrite 0 --out-mode sc --out-casa-gaintables 1 --out-plots 1 --log-memory 1 --log-boring 1 --g-time-int 16 --g-freq-int 32 --g-clip-low 0.3 --g-clip-high 2.5 --g-solvable 1 --g-type complex-2x2 --g-save-to /stimela_mount/output/gain --g-update-type full --g-max-prior-error 0.3 --g-max-post-error 0.3 INFO 04:47:21 - cc [0.1 0.1 0.0Gb] set global console verbosity level 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] started /usr/local/bin/gocubical --sol-jones G --data-ms /stimela_mount/msdir/J1521_5508.ms --data-column DATA --data-time-chunk 120 --data-freq-chunk 512 --data-single-chunk None --sel-taql None --model-list MODEL_DATA --model-ddes never --model-beam-l-axis None --model-beam-m-axis None --model-pa-rotate 0 --model-feed-rotate 0 --montblanc-dtype float --weight-column WEIGHT --madmax-enable 0 --madmax-estimate corr --madmax-plot 0 --madmax-threshold 0,20 --sel-diag 0 --sol-term-iters 50,50,50 --bbc-save-to /stimela_mount/output/ --dist-ncpu 24 --dist-max-chunks 4 --out-name /stimela_mount/output/selfcal_HI_abs --out-overwrite 0 --out-mode sc --out-casa-gaintables 1 --out-plots 1 --log-memory 1 --log-boring 1 --g-time-int 16 --g-freq-int 32 --g-clip-low 0.3 --g-clip-high 2.5 --g-solvable 1 --g-type complex-2x2 --g-save-to /stimela_mount/output/gain --g-update-type full --g-max-prior-error 0.3 --g-max-post-error 0.3 INFO 04:47:21 - main [0.1 0.1 0.0Gb] output basename explicitly set to /stimela_mount/output/selfcal_HI_abs, --out-dir setting ignored INFO 04:47:21 - main [0.1 0.1 0.0Gb] Selected Options: INFO 04:47:21 - main [0.1 0.1 0.0Gb] [data] Visibility data options INFO 04:47:21 - main [0.1 0.1 0.0Gb] - ms ................................................ = /stimela_mount/msdir/J1521_5508.ms INFO 04:47:21 - main [0.1 0.1 0.0Gb] - column ............................................ = DATA INFO 04:47:21 - main [0.1 0.1 0.0Gb] - time-chunk ........................................ = 120 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - freq-chunk ........................................ = 512 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - rebin-time ........................................ = 1 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - rebin-freq ........................................ = 1 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - chunk-by .......................................... = SCAN_NUMBER INFO 04:47:21 - main [0.1 0.1 0.0Gb] - chunk-by-jump ..................................... = 1 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - single-chunk ...................................... = None INFO 04:47:21 - main [0.1 0.1 0.0Gb] - single-tile ....................................... = -1 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - normalize ......................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] [sel] Data selection options INFO 04:47:21 - main [0.1 0.1 0.0Gb] - field ............................................. = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - ddid .............................................. = None INFO 04:47:21 - main [0.1 0.1 0.0Gb] - taql .............................................. = None INFO 04:47:21 - main [0.1 0.1 0.0Gb] - chan .............................................. = INFO 04:47:21 - main [0.1 0.1 0.0Gb] - diag .............................................. = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] [out] Options for output products INFO 04:47:21 - main [0.1 0.1 0.0Gb] - dir ............................................... = cubical INFO 04:47:21 - main [0.1 0.1 0.0Gb] - name .............................................. = /stimela_mount/output/selfcal_HI_abs INFO 04:47:21 - main [0.1 0.1 0.0Gb] - overwrite ......................................... = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] - backup ............................................ = 1 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - mode .............................................. = sc INFO 04:47:21 - main [0.1 0.1 0.0Gb] - apply-solver-flags ................................ = True INFO 04:47:21 - main [0.1 0.1 0.0Gb] - column ............................................ = CORRECTED_DATA INFO 04:47:21 - main [0.1 0.1 0.0Gb] - derotate .......................................... = None INFO 04:47:21 - main [0.1 0.1 0.0Gb] - model-column ...................................... = INFO 04:47:21 - main [0.1 0.1 0.0Gb] - weight-column ..................................... = INFO 04:47:21 - main [0.1 0.1 0.0Gb] - reinit-column ..................................... = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] - subtract-model .................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - subtract-dirs ..................................... = : INFO 04:47:21 - main [0.1 0.1 0.0Gb] - correct-dir ....................................... = -1 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - plots ............................................. = 1 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - casa-gaintables ................................... = True INFO 04:47:21 - main [0.1 0.1 0.0Gb] [model] Calibration model options INFO 04:47:21 - main [0.1 0.1 0.0Gb] - list .............................................. = MODEL_DATA INFO 04:47:21 - main [0.1 0.1 0.0Gb] - ddes .............................................. = never INFO 04:47:21 - main [0.1 0.1 0.0Gb] - beam-pattern ...................................... = None INFO 04:47:21 - main [0.1 0.1 0.0Gb] - beam-l-axis ....................................... = None INFO 04:47:21 - main [0.1 0.1 0.0Gb] - beam-m-axis ....................................... = None INFO 04:47:21 - main [0.1 0.1 0.0Gb] - feed-rotate ....................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - pa-rotate ......................................... = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] [montblanc] Montblanc simulation options INFO 04:47:21 - main [0.1 0.1 0.0Gb] - device-type ....................................... = CPU INFO 04:47:21 - main [0.1 0.1 0.0Gb] - dtype ............................................. = float INFO 04:47:21 - main [0.1 0.1 0.0Gb] - mem-budget ........................................ = 1024 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - verbosity ......................................... = WARNING INFO 04:47:21 - main [0.1 0.1 0.0Gb] - threads ........................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - pa-rotate ......................................... = None INFO 04:47:21 - main [0.1 0.1 0.0Gb] [weight] Weighting options INFO 04:47:21 - main [0.1 0.1 0.0Gb] - column ............................................ = WEIGHT INFO 04:47:21 - main [0.1 0.1 0.0Gb] - fill-offdiag ...................................... = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] - legacy-v1-2 ....................................... = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] [flags] General flagging options INFO 04:47:21 - main [0.1 0.1 0.0Gb] - apply ............................................. = -cubical INFO 04:47:21 - main [0.1 0.1 0.0Gb] - auto-init ......................................... = legacy INFO 04:47:21 - main [0.1 0.1 0.0Gb] - save .............................................. = cubical INFO 04:47:21 - main [0.1 0.1 0.0Gb] - save-legacy ....................................... = auto INFO 04:47:21 - main [0.1 0.1 0.0Gb] - reinit-bitflags ................................... = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] - warn-thr .......................................... = 0.3 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - see-no-evil ....................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] [degridding] Options for the degridder. Only in use when predicting from DicoModels using DDFacet INFO 04:47:21 - main [0.1 0.1 0.0Gb] - OverS ............................................. = 11 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - Support ........................................... = 7 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - Nw ................................................ = 100 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - wmax .............................................. = 0.0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - Padding ........................................... = 1.7 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - NDegridBand ....................................... = 16 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - MaxFacetSize ...................................... = 0.25 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - MinNFacetPerAxis .................................. = 1 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - NProcess .......................................... = 8 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - BeamModel ......................................... = None INFO 04:47:21 - main [0.1 0.1 0.0Gb] - NBand ............................................. = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - FITSFile .......................................... = beam_$(corr)_$(reim).fits INFO 04:47:21 - main [0.1 0.1 0.0Gb] - FITSFeed .......................................... = None INFO 04:47:21 - main [0.1 0.1 0.0Gb] - FITSFeedSwap ...................................... = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] - DtBeamMin ......................................... = 5.0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - FITSParAngleIncDeg ................................ = 5.0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - FITSLAxis ......................................... = -X INFO 04:47:21 - main [0.1 0.1 0.0Gb] - FITSMAxis ......................................... = Y INFO 04:47:21 - main [0.1 0.1 0.0Gb] - FITSVerbosity ..................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - FeedAngle ......................................... = 0.0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - FlipVisibilityHands ............................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] [postmortem] Options for "postmortem" flagging based on solution statistics INFO 04:47:21 - main [0.1 0.1 0.0Gb] - enable ............................................ = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] - tf-chisq-median ................................... = 1.2 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - tf-np-median ...................................... = 0.5 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - time-density ...................................... = 0.5 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - chan-density ...................................... = 0.5 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - ddid-density ...................................... = 0.5 INFO 04:47:21 - main [0.1 0.1 0.0Gb] [madmax] Options for the "Mad Max" flagger INFO 04:47:21 - main [0.1 0.1 0.0Gb] - enable ............................................ = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - residuals ......................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - estimate .......................................... = corr INFO 04:47:21 - main [0.1 0.1 0.0Gb] - diag .............................................. = True INFO 04:47:21 - main [0.1 0.1 0.0Gb] - offdiag ........................................... = True INFO 04:47:21 - main [0.1 0.1 0.0Gb] - threshold ......................................... = [0, 20] INFO 04:47:21 - main [0.1 0.1 0.0Gb] - global-threshold .................................. = 12 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - plot .............................................. = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - plot-frac-above ................................... = 0.01 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - plot-bl ........................................... = INFO 04:47:21 - main [0.1 0.1 0.0Gb] - flag-ant .......................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - flag-ant-thr ...................................... = 5 INFO 04:47:21 - main [0.1 0.1 0.0Gb] [sol] Solution options which apply at the solver level INFO 04:47:21 - main [0.1 0.1 0.0Gb] - jones ............................................. = G INFO 04:47:21 - main [0.1 0.1 0.0Gb] - precision ......................................... = 32 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - delta-g ........................................... = 1e-06 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - delta-chi ......................................... = 1e-06 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - chi-int ........................................... = 5 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - last-rites ........................................ = True INFO 04:47:21 - main [0.1 0.1 0.0Gb] - stall-quorum ...................................... = 0.99 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - term-iters ........................................ = [50, 50, 50] INFO 04:47:21 - main [0.1 0.1 0.0Gb] - flag-divergence ................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - min-bl ............................................ = 0.0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - max-bl ............................................ = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - subset ............................................ = INFO 04:47:21 - main [0.1 0.1 0.0Gb] [bbc] Options for baseline-based corrections (a.k.a. BBCs, a.k.a. interferometer gains). INFO 04:47:21 - main [0.1 0.1 0.0Gb] - load-from ......................................... = INFO 04:47:21 - main [0.1 0.1 0.0Gb] - compute-2x2 ....................................... = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] - apply-2x2 ......................................... = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] - save-to ........................................... = /stimela_mount/output/ INFO 04:47:21 - main [0.1 0.1 0.0Gb] - per-chan .......................................... = True INFO 04:47:21 - main [0.1 0.1 0.0Gb] - plot .............................................. = True INFO 04:47:21 - main [0.1 0.1 0.0Gb] [dist] Parallelization and distribution options INFO 04:47:21 - main [0.1 0.1 0.0Gb] - ncpu .............................................. = 24 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - nworker ........................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - nthread ........................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - max-chunks ........................................ = 4 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - min-chunks ........................................ = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - pin ............................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - pin-io ............................................ = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] - pin-main .......................................... = io INFO 04:47:21 - main [0.1 0.1 0.0Gb] - safe .............................................. = True INFO 04:47:21 - main [0.1 0.1 0.0Gb] [log] Options related to logging INFO 04:47:21 - main [0.1 0.1 0.0Gb] - memory ............................................ = True INFO 04:47:21 - main [0.1 0.1 0.0Gb] - stats ............................................. = chi2:.3f INFO 04:47:21 - main [0.1 0.1 0.0Gb] - stats-warn ........................................ = chi2:10 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - boring ............................................ = True INFO 04:47:21 - main [0.1 0.1 0.0Gb] - append ............................................ = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] - verbose ........................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - file-verbose ...................................... = None INFO 04:47:21 - main [0.1 0.1 0.0Gb] [debug] Debugging options for the discerning masochist INFO 04:47:21 - main [0.1 0.1 0.0Gb] - pdb ............................................... = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] - panic-amplitude ................................... = 0.0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - stop-before-solver ................................ = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] - escalate-warnings ................................. = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] [misc] Miscellaneous options INFO 04:47:21 - main [0.1 0.1 0.0Gb] - random-seed ....................................... = None INFO 04:47:21 - main [0.1 0.1 0.0Gb] - parset-version .................................... = 0.1 INFO 04:47:21 - main [0.1 0.1 0.0Gb] [g] Options for G-Jones term INFO 04:47:21 - main [0.1 0.1 0.0Gb] - solvable .......................................... = 1 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - type .............................................. = complex-2x2 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - delay-estimate-pad-factor ......................... = 8 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - load-from ......................................... = INFO 04:47:21 - main [0.1 0.1 0.0Gb] - xfer-from ......................................... = INFO 04:47:21 - main [0.1 0.1 0.0Gb] - save-to ........................................... = /stimela_mount/output/gain INFO 04:47:21 - main [0.1 0.1 0.0Gb] - dd-term ........................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - fix-dirs .......................................... = INFO 04:47:21 - main [0.1 0.1 0.0Gb] - update-type ....................................... = full INFO 04:47:21 - main [0.1 0.1 0.0Gb] - estimate-pzd ...................................... = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] - time-int .......................................... = 16 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - freq-int .......................................... = 32 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - max-prior-error ................................... = 0.3 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - max-post-error .................................... = 0.3 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - low-snr-warn ...................................... = 75 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - high-gain-var-warn ................................ = 30 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - clip-low .......................................... = 0.3 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - clip-high ......................................... = 2.5 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - clip-after ........................................ = 5 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - max-iter .......................................... = 20 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - pin-slope-iters ................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - epsilon ........................................... = 1e-06 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - delta-chi ......................................... = 1e-06 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - conv-quorum ....................................... = 0.99 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - ref-ant ........................................... = None INFO 04:47:21 - main [0.1 0.1 0.0Gb] - prop-flags ........................................ = default INFO 04:47:21 - main [0.1 0.1 0.0Gb] - diag-only ......................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - offdiag-only ...................................... = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] - robust-cov ........................................ = compute INFO 04:47:21 - main [0.1 0.1 0.0Gb] - robust-scale ...................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - robust-npol ....................................... = 2 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - robust-int ........................................ = 1 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - robust-flag-weights ............................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - robust-cov-thresh ................................. = 1 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - robust-sigma-thresh ............................... = 3 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - robust-save-weights ............................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - estimate-delays ................................... = False INFO 04:47:21 - main [0.1 0.1 0.0Gb] [de] de INFO 04:47:21 - main [0.1 0.1 0.0Gb] - dd-term ........................................... = 1 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - clip-low .......................................... = 0.0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - clip-high ......................................... = 0 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - delta-chi ......................................... = 1e-05 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - max-prior-error ................................... = 0.44 INFO 04:47:21 - main [0.1 0.1 0.0Gb] - max-post-error .................................... = 0.44 INFO 04:47:21 - main [0.1 0.1 0.0Gb] Enabling G-Jones INFO 04:47:21 - main [0.1 0.1 0.0Gb] mode: SolveAndCorrect INFO 04:47:21 - main [0.1 0.1 0.0Gb] solver is apply-only type: False INFO 04:47:21 - main [0.1 0.1 0.0Gb] solver requires model: True INFO 04:47:21 - data_handler [0.1 0.1 0.0Gb] reading MS /stimela_mount/msdir/J1521_5508.ms INFO 04:47:21 - data_handler [0.1 0.1 0.0Gb] detected circular (rl) feeds INFO 04:47:21 - data_handler [0.1 0.1 0.0Gb] fields are *0: J1521+5508 INFO 04:47:21 - data_handler [0.1 0.1 0.0Gb] MS contains 1 spectral windows INFO 04:47:21 - data_handler [0.1 0.1 0.0Gb] max freq chunk size is 512 channels and/or -- MHz INFO 04:47:21 - data_handler [0.1 0.1 0.0Gb] DDID 0: 512/512 channels selected INFO 04:47:21 - data_handler [0.1 0.1 0.0Gb] found 1 frequency chunks: 0 512 INFO 04:47:22 - data_handler [0.2 0.1 0.0Gb] applying TAQL query 'FIELD_ID == 0 && DATA_DESC_ID IN [0]' (586815/586815 rows selected) INFO 04:47:22 - data_handler [0.2 0.1 0.0Gb] 32 antennas, 586815 rows, 1/1 DDIDs, 1349 timeslots, 2 corrs INFO 04:47:22 - data_handler [0.2 0.1 0.0Gb] DDID central frequencies are at 0.69 GHz INFO 04:47:22 - data_handler [0.2 0.1 0.0Gb] Input model feed rotation disabled, PA rotation disabled INFO 04:47:22 - data_handler [0.2 0.1 0.0Gb] Output visibilities derotation disabled INFO 04:47:22 - data_handler [0.2 0.1 0.0Gb] Using 1 model(s) for 1 directions(s) (DDEs explicitly disabled) INFO 04:47:22 - data_handler [0.2 0.1 0.0Gb] model 0 (weight WEIGHT): INFO 04:47:22 - data_handler [0.2 0.1 0.0Gb] direction 0: MODEL_DATA INFO 04:47:22 - main [0.2 0.1 0.0Gb] subtraction directions set to slice(None, None, None) INFO 04:47:22 - param_db [0.2 0.1 0.0Gb] creating /stimela_mount/output/gain in fragmented mode INFO 04:47:22 - gain_machine [0.2 0.1 0.0Gb] G solutions will be saved to /stimela_mount/output/gain ERROR 04:47:22 - casa_db_adaptor [0.2 0.1 0.0Gb] Gaintables cannot be written in Python 3 mode due to current casacore implementation issues INFO 04:47:22 - param_db [0.2 0.1 0.0Gb] creating /stimela_mount/output/ in fragmented mode INFO 04:47:22 - gain_machine [0.2 0.1 0.0Gb] will compute & save suggested baseline-based corrections (BBCs) to /stimela_mount/output/ INFO 04:47:22 - gain_machine [0.2 0.1 0.0Gb] (these can optionally be applied in a subsequent CubiCal run) INFO 04:47:22 - main [0.2 0.1 0.0Gb] multi-process mode: 23+1 workers, single thread INFO 04:47:22 - main [0.2 0.1 0.0Gb] defining chunks (time 120, freq 512, also when SCAN_NUMBER jumps > 1.0) INFO 04:47:22 - data_handler [0.2 0.1 0.0Gb] read indexing columns (586815 total rows) INFO 04:47:22 - data_handler [0.2 0.1 0.0Gb] built timeslot index (1349 unique timestamps) INFO 04:47:22 - data_handler [0.2 0.1 0.0Gb] max chunk size is 120 timeslots and/or -- seconds INFO 04:47:23 - data_handler [0.2 0.2 0.0Gb] found 15 time chunks: 0:0:0 1:52200:120 2:77430:178 3:129630:298 4:154860:356 5:207060:476 6:232290:534 7:284490:654 8:309720:712 9:361920:832 10:387585:891 11:439785:1011 12:465015:1069 13:517215:1189 14:542880:1248 1349 INFO 04:47:23 - data_handler [0.2 0.2 0.0Gb] generated 15 row chunks based on time and DDID INFO 04:47:23 - data_handler [0.2 0.2 0.0Gb] row chunks yield 15 potential tiles INFO 04:47:23 - data_handler [0.2 0.2 0.0Gb] coarsening this to 4 tiles (max 4 chunks per tile, based on 23/4 requested) INFO 04:47:24 - wisdom [0.2 0.2 0.0Gb] Detected a total of 251.78GiB of system memory. INFO 04:47:24 - wisdom [0.2 0.2 0.0Gb] Per-solver (worker) memory use estimated at 7.72GiB: 3.07% of total system memory. INFO 04:47:24 - wisdom [0.2 0.2 0.0Gb] Peak I/O memory use estimated at 10.73GiB: 4.26% of total system memory. INFO 04:47:24 - wisdom [0.2 0.2 0.0Gb] Total peak memory usage estimated at 188.23GiB: 74.76% of total system memory. INFO 04:47:24 - data_handler [0.2 0.2 0.0Gb] inserting new column BITFLAG INFO 04:47:24 - data_handler [0.2 0.2 0.0Gb] inserting new column BITFLAG_ROW INFO 04:47:24 - data_handler [0.2 0.2 0.0Gb] auto-filling bitflag 'legacy' from FLAG/FLAG_ROW column. Please do not interrupt this process! INFO 04:47:24 - data_handler [0.2 0.2 0.0Gb] note that all other bitflags will be cleared by this INFO 04:47:40 - data_handler [0.3 0.2 0.0Gb] auto-fill complete INFO 04:47:40 - data_handler [0.3 0.2 0.0Gb] BITFLAG column defines the following flagsets: legacy:1 INFO 04:47:40 - data_handler [0.3 0.2 0.0Gb] will exclude flagset cubical INFO 04:47:40 - data_handler [0.3 0.2 0.0Gb] flagset 'cubical' not found -- ignoring INFO 04:47:40 - data_handler [0.3 0.2 0.0Gb] applying BITFLAG mask 1 to input data INFO 04:47:40 - data_handler [0.3 0.2 0.0Gb] will save output flags into BITFLAG 'cubical' (2), and into FLAG/FLAG_ROW INFO 04:47:41 - main [0.3 0.2 0.0Gb] waiting for I/O on tile 0/4 INFO 04:47:41 - main [io] [0.3 0.2 0.0Gb] loading tile 0/4 INFO 04:47:41 - ms_tile [io] [0.3 0.2 0.0Gb] tile 0/4: reading MS rows 0~154859 INFO 04:47:41 - data_handler [io] [0.2 0.1 0.0Gb] reading DATA INFO 04:48:07 - data_handler [io] [2.6/2.8 2.5/2.6 0.0Gb] reading BITFLAG INFO 04:48:25 - ms_tile [io] [3.9/4.1 3.8/4.0 0.0Gb] 46.93% input visibilities flagged and/or deselected INFO 04:48:27 - main [0.3 0.2 1.0Gb] Exiting with exception: BrokenProcessPool(A process in the process pool was terminated abruptly while the future was running or pending.) Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/cubical/main.py", line 572, in main stats_dict = workers.run_process_loop(ms, tile_list, load_model, single_chunk, solver_type, solver_opts, debug_opts, out_opts) File "/usr/local/lib/python3.6/dist-packages/cubical/workers.py", line 217, in run_process_loop return _run_multi_process_loop(ms, load_model, solver_type, solver_opts, debug_opts, out_opts) File "/usr/local/lib/python3.6/dist-packages/cubical/workers.py", line 277, in _run_multi_process_loop if not io_futures[itile].result(): File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. 2020-09-20 04:48:27: gocubical exited with code 1 docker returns error code 1 job failed at 2020-09-20 10:18:28.200140 after 0:01:21.164748 Traceback (most recent call last): File "/home/arnab/stimela_env/venv3/lib/python3.6/site-packages/stimela/recipe.py", line 704, in run job.run_job() File "/home/arnab/stimela_env/venv3/lib/python3.6/site-packages/stimela/recipe.py", line 424, in run_job self.job.start(output_wrangler=self.apply_output_wranglers) File "/home/arnab/stimela_env/venv3/lib/python3.6/site-packages/stimela/docker.py", line 171, in start kill_callback=lambda: utils.xrun("docker", ["kill", self.name])) File "/home/arnab/stimela_env/venv3/lib/python3.6/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status)) stimela.utils.StimelaCabRuntimeError: docker returns error code 1