Empty arguments to cubical confuse the Stimela wrapper

[Arnab-half-blood-prince]

Hi,
I am trying to run selfcal using cubical. But getting some unexpected error. Not sure what is the problem.
check the screenshot.

[KshitijT]

@Arnabiitindore, could you please post the log ?

[Arnab-half-blood-prince]

log_cubical_selfcal.txt

[Arnab-half-blood-prince]

This is the recipe I used:
import stimela
from stimela.pathformatter import pathformatter as spf

INPUT="input"
MSDIR="msdir"
OUTPUT="output"

ms12 = "Radio_halo_12SEP.ms"

recipe = stimela.Recipe("Radio halo selfcal with cubical",
ms_dir=MSDIR,
JOB_TYPE="docker")

recipe.add("cab/cubical", "calibrate 1st ms with cubical", {

##data##

'data-ms':ms12,
'data-column':'DATA',
'data-time-chunk':400,
'data-freq-chunk':512,
#'data-rebin-time':1,
#'data-rebin-freq':1,
'data-chunk-by':'SCAN_NUMBER',
'data-chunk-by-jump':0,

##sel##

'sel-field':0,

model

'model-list':spf('MODEL_DATA'),

out

#'out-':'selfcal',
'out-name':'selfcal',
'out-overwrite':False,
'out-mode':'sc',
'out-column':'CORRECTED_DATA',
'out-subtract-model':0,
'out-subtract-dirs':0,
'out-plots':True,
'out-casa-gaintables':True,

weight

'weight-column':'WEIGHT_SPECTRUM',

degridding

'degridding-OverS':11,
'degridding-Support':7,
'degridding-Nw':100,
'degridding-wmax':0.0,
'degridding-Padding':1.7,
'degridding-NDegridBand':16,
'degridding-MaxFacetSize':0.25,
'degridding-MinNFacetPerAxis':1,

sol

'sol-jones':'G',
'sol-precision':'64',
'sol-delta-g': '1e-06',
'sol-delta-chi': '1e-06',
'sol-chi-int': '5',
'sol-last-rites': True,
'sol-stall-quorum':0.99,
'sol-term-iters':[50],
#'sol-flag-divergence':0,
#'sol-min-bl':500.0,
#'sol-max-bl':0,

log

#'log-memory': True,
#'log-boring': True,
#'log-append': False,
#'log-verbose': '',
#'log-file-verbose':'',

debug

#'debug-pdb': False,
#'debug-panic-amplitude': 0.0,
#'debug-stop-before-solver': False,

g

'g-solvable': True,
'g-type':'complex-2x2',
'g-load-from':None,
'g-xfer-from':None,
'g-save-to':'',
'g-dd-term': False,
'g-fix-dirs': '',
'g-update-type':'full',
'g-time-int':32,
'g-freq-int':512,
'g-clip-low': 0.3,
'g-clip-high': 2.5,
'g-clip-after': 5,
'g-max-iter': 50,
'g-conv-quorum': 0.99,
'g-ref-ant': '',
'g-prop-flags': 'default',
'g-diag-only': False,

}, input=INPUT, output=OUTPUT, label="cubical on 1st ms")

STEPS = ["cubical on 1st ms"]
recipe.run(STEPS)

[KshitijT]

g-fix-dirs and g-ref-ant don't have defaults as far as I remember. I think you need to specify them - that's why you are getting the error:
Unexpected number of arguments.

[o-smirnov]

Yeah the workaround is to set both to None for now. Setting them to empty strings confuses the command-line parser.

The problem is here: https://github.com/ratt-ru/Stimela/blob/master/stimela/cargo/cab/cubical/src/run.py#L57. @SpheMakh only half-scabhafied the runscript. The join/shlex.split pattern is Ancient Evil that needs to be eradicated with fire (this problem here is an example of why). Just pass the argument list to prun() directly...

[Arnab-half-blood-prince]

There seems lots of other issues:

1. I checked the run.py file . It soes not contain prun. It uses subprocess instead. I tried to import prun from scabha, but this module is not available.

2. I check all the no default option in cubical documents. Then I found out not all options are there in stimeal: like
   a. madmax-plot-bl - not available in stimela but it is no default option in cubical docu
   b. out-weight-column - not available in stimela but it is no default

3. I put all other no default options as mentioned in cubical documents and passed None. But the error remains

Has anyone tested cubical with stimela before? Or am I missing something?

The log.txt are attached.
log_cubical_selfcal.txt

The recipe:
import stimela
from stimela.pathformatter import pathformatter as spf

INPUT="input"
MSDIR="msdir"
OUTPUT="output"

ms12 = "Radio_halo_12SEP.ms"

recipe = stimela.Recipe("Radio halo selfcal with cubical",
ms_dir=MSDIR,
JOB_TYPE="docker")

recipe.add("cab/cubical", "calibrate 1st ms with cubical", {

##data##

'data-ms':ms12,
'data-column':'DATA',
'data-time-chunk':400,
'data-freq-chunk':512,
#'data-rebin-time':1,
#'data-rebin-freq':1,
'data-chunk-by':'SCAN_NUMBER',
'data-chunk-by-jump':0,
'data-single-chunk':'None',

##sel##

'sel-field':0,
'sel-taql':'None',
'sel-diag':False,

model

'model-list':spf('MODEL_DATA'),
'model-beam-pattern':None,
'model-beam-l-axis':'None',
'model-beam-m-axis':'None',

out

#'out-':'selfcal',
'out-name':'selfcal',
'out-overwrite':False,
'out-mode':'sc',
'out-column':'CORRECTED_DATA',
'out-subtract-model':0,
'out-subtract-dirs':'None',
'out-plots':True,
'out-casa-gaintables':True,
'out-model-column':'None',
'out-reinit-column':False,

weight

'weight-column':'WEIGHT_SPECTRUM',

madmax

'madmax-enable':False,
'madmax-plot':False,

bbc

'bbc-load-from':None,

degridding

'degridding-OverS':11,
'degridding-Support':7,
'degridding-Nw':100,
'degridding-wmax':0.0,
'degridding-Padding':1.7,
'degridding-NDegridBand':16,
'degridding-MaxFacetSize':0.25,
'degridding-MinNFacetPerAxis':1,

sol

'sol-jones':'G',
'sol-precision':'64',
'sol-delta-g': '1e-06',
'sol-delta-chi': '1e-06',
'sol-chi-int': '5',
'sol-last-rites': True,
'sol-stall-quorum':0.99,
'sol-term-iters':[50],
#'sol-flag-divergence':0,
'sol-min-bl':500.0,
'sol-max-bl':0,
'sol-subset':'None',

log

'log-memory': True,
'log-boring': True,
'log-append': False,
'log-verbose': '',
#'log-file-verbose':'',

debug

'debug-pdb': False,
'debug-panic-amplitude': 0.0,
'debug-stop-before-solver': False,

g

'g-solvable': True,
'g-type':'complex-2x2',
'g-load-from':None,
'g-xfer-from':None,
'g-save-to':'',
'g-dd-term': False,
'g-fix-dirs': 'None',
'g-update-type':'full',
'g-time-int':32,
'g-freq-int':512,
'g-clip-low': 0.3,
'g-clip-high': 2.5,
'g-clip-after': 5,
'g-max-iter': 50,
'g-conv-quorum': 0.99,
'g-ref-ant': '26,15,16,8',
'g-prop-flags': 'default',
'g-diag-only': False,

}, input=INPUT, output=OUTPUT, label="cubical on 1st ms")

STEPS = ["cubical on 1st ms"]
recipe.run(STEPS)

[o-smirnov]

I checked the run.py file . It soes not contain prun. It uses subprocess instead. I tried to import prun from scabha, but this module is not available.

Hmmm I was looking at master, but of course you're probably using an older release. What's your version?

It is true that not all CubiCal options are propagated into the Stimela wrapper, probably out of laziness (or a sense that they weren't really needed). If you spot one you need, please raise it here or on an issue. And yes, some of these options will have no default (i.e. an empty string by default), but if they're not exposed by Stimela, then no harm, no foul.

Yes, CubiCal is used with Stimela, and routinely at that. So we just have to figure out your version/config problems.

[Arnab-half-blood-prince]

stimela 1.6.5
The latest probably 1.6.7.
I used pip install in vistual environment before.
Let me try to install the latest then.

Thanks for the help.

[o-smirnov]

You can pip install -e from the master branch, but all the usual caveats of being on the bleeding edge apply... then again, this will make it easier for you to test experimental branches and fixes, if that's your thing.

[Arnab-half-blood-prince]

Yes, I have done it from the master.
It is working fine now following your suggestion prun(arglist) in run.py.
Thanks for the help

[o-smirnov]

Ah good! Feel free to push to a branch with the fix then.

[Arnab-half-blood-prince]

It started and sunning well. But then I am getting a error:
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

The log file is attached. Not sure what is the problem here.
log_cubical.txt

The option I used :
cubical_opts = {
"data-ms": ms,
"data-column": 'DATA',
"data-time-chunk": 120,
"data-freq-chunk": 512,
"data-single-chunk":'None',
"model-list": spf('MODEL_DATA'),
"sel-taql":'None',
"dist-ncpu": 24,
"dist-max-chunks":4,
"log-memory": True,
"sol-jones": 'G',
"sol-term-iters": "50,50,50",
"sel-diag": False,
"out-name":'selfcal_HI_abs',
"out-mode": 'sc',
"out-plots": True,
"out-overwrite":False,
"dist-max-chunks":4,
"out-casa-gaintables": True,
"weight-column": 'WEIGHT',
"montblanc-dtype": 'float',
"bbc-save-to": "",
"bbc-load-from":None,
"g-solvable": True,
"g-type": 'complex-2x2',
"g-update-type": 'full',
"g-time-int": 16,
"g-freq-int": 32,
"out-overwrite":False,
"g-save-to": 'gain',
"g-clip-low": 0.3,
"g-clip-high": 2.5,
"g-max-prior-error": 0.3,
"g-max-post-error": 0.3,
"madmax-enable": False,
"madmax-plot": False,
"madmax-threshold": [0,20],
"madmax-estimate": 'corr',
"log-boring": True,
"dd-dd-term": False,
"model-ddes": 'never',
'model-beam-pattern':None,
'model-beam-l-axis':'None',
'model-beam-m-axis':'None',
#"model-ddes": 'never',

}

[o-smirnov]

It looks like the I/O worker just fell over unceremoniously. My first guess would be out of memory -- how much RAM have you got on the box? Try decreasing --data-time-chunk to load less data at once (generally, time chunk should be a multiple of the solution interval i.e. --g-time-int, your interval is 16 so you could even set the time chunk to be as low as 16).

[Arnab-half-blood-prince]

Hi,
Yes, you are right. I used a smaller chunk and it worked. I also imaged after cubical 2GC with wsclean. And there are nice improvements in the image in comparison with the earlier selfcal image.
Thanks for the help.
I am closing it .

[o-smirnov]

Very glad to hear that!

[o-smirnov]

Actually let's leave this open, as the caveat of the empty arguments causing a mysterious error remains...