Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Number of rows inconsistent after removing auto-correlations #133

Closed
IanHeywood opened this issue Jan 31, 2018 · 6 comments
Closed

Number of rows inconsistent after removing auto-correlations #133

IanHeywood opened this issue Jan 31, 2018 · 6 comments

Comments

@IanHeywood
Copy link
Collaborator

I'm trying G+dE cal on some ASKAP data. Has anyone seen this error before? Note that direction-independent (phase-diag) calibration reading from just the MODEL_DATA column works fine.

gocubical ABC_dEcal.parset --model-list=MODEL_DATA:img_SB4899_pcalfitsmask_0000_dE-sources.pybdsm.lsm.html@dE --sel-chan=0:1619 --g-save-to={data[ms]}/G-field:{sel[field]}:spw0 --out-name cube_dEcal_2017-12-28_163317_0_x2_ABC1234.ms_2018-01-31-10-01-02_spw0
 - 10:01:03 - shm_utils          | reaping 1 shared memory objects associated with 1 dead cubical processes
 - 10:01:03 - cc                 [0.1/0.1 0.5/0.5 0.0Gb] set global console verbosity level 0
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] started /home/heywoodl/venv/cubienv/bin/gocubical ABC_dEcal.parset --model-list=MODEL_DATA:img_SB4899_pcalfitsmask_0000_dE-sources.pybdsm.lsm.html@dE --sel-chan=0:1619 --g-save-to={data[ms]}/G-field:{sel[field]}:spw0 --out-name cube_dEcal_2017-12-28_163317_0_x2_ABC1234.ms_2018-01-31-10-01-02_spw0
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  Selected Options:
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [data] Visibility data options
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - ms ................................................ = 2017-12-28_163317_0_x2_ABC1234.ms
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - column ............................................ = DATA
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - time-chunk ........................................ = 360
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - freq-chunk ........................................ = 1620
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - chunk-by .......................................... = SCAN_NUMBER
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - chunk-by-jump ..................................... = 0
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - single-chunk ...................................... =
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [sel] Data selection options
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - field ............................................. = 0
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - ddid .............................................. = None
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - taql .............................................. =
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - chan .............................................. = 0:1619
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [model] Calibration model options
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - list .............................................. = MODEL_DATA:img_SB4899_pcalfitsmask_0000_dE-sources.pybdsm.lsm.html@dE
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - ddes .............................................. = True
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - beam-pattern ...................................... = None
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - beam-l-axis ....................................... = None
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - beam-m-axis ....................................... = None
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [montblanc] Montblanc simulation options
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - device-type ....................................... = CPU
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - dtype ............................................. = float
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - feed-type ......................................... = linear
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - mem-budget ........................................ = 4096
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - verbosity ......................................... = WARNING
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [weight] Weighting options
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - column ............................................ = WEIGHT_SPECTRUM
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [flags] Flagging options
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - apply ............................................. = legacy
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - auto-init ......................................... = legacy
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - save .............................................. = cubical
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - reinit-bitflags ................................... = False
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - post-sol .......................................... = False
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - tf-chisq-median ................................... = 1.2
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - tf-np-median ...................................... = 0.5
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - time-density ...................................... = 0.5
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - chan-density ...................................... = 0.5
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - ddid-density ...................................... = 0.5
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [sol] Solution options which apply at the solver level
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - jones ............................................. = ['G', 'dE']
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - precision ......................................... = 32
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - delta-g ........................................... = 1e-06
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - delta-chi ......................................... = 1e-06
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - chi-int ........................................... = 5
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - last-rites ........................................ = True
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - stall-quorum ...................................... = 0.99
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - term-iters ........................................ =
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - min-bl ............................................ = 200.0
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - max-bl ............................................ = 1000000.0
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - subset ............................................ =
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [bbc] Options for baseline-based corrections (a.k.a. BBCs, a.k.a. interferometer gains).
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - load-from ......................................... =
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - compute-2x2 ....................................... = False
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - apply-2x2 ......................................... = False
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - save-to ........................................... = {data[ms]}/BBC-field:{sel[field]}-ddid:{sel[ddid]}.parmdb
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - per-chan .......................................... = True
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - plot .............................................. = True
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [dist] Parallelization and distribution options
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - ncpu .............................................. = 3
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - min-chunks ........................................ = 4
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [out] Options for output products
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - name .............................................. = cube_dEcal_2017-12-28_163317_0_x2_ABC1234.ms_2018-01-31-10-01-02_spw0
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - mode .............................................. = sr
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - column ............................................ = CORRECTED_DATA
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - reinit-column ..................................... = False
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - subtract-model .................................... = 0
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - subtract-dirs ..................................... = :
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - plots ............................................. = True
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - plots-show ........................................ = False
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [log] Options related to logging
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - memory ............................................ = True
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - boring ............................................ = False
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - append ............................................ = False
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - verbose ........................................... = 0
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - file-verbose ...................................... = None
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [debug] Debugging options for the discerning masochist
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - pdb ............................................... = False
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [misc] Miscellaneous options
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - random-seed ....................................... = None
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - parset-version .................................... = 0.1
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [g] Options for G-Jones term
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - solvable .......................................... = True
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - type .............................................. = complex-2x2
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - load-from ......................................... =
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - xfer-from ......................................... =
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - save-to ........................................... = {data[ms]}/G-field:{sel[field]}:spw0
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - dd-term ........................................... = 0
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - fix-dirs .......................................... =
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - update-type ....................................... = full
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - time-int .......................................... = 6
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - freq-int .......................................... = 1620
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - clip-low .......................................... = 0.1
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - clip-high ......................................... = 10.0
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - clip-after ........................................ = 5
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - max-iter .......................................... = 20
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - conv-quorum ....................................... = 0.99
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - ref-ant ........................................... = None
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - term-iters ........................................ = 10
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] [de] Options for dE-Jones term
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - solvable .......................................... = True
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - type .............................................. = complex-2x2
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - load-from ......................................... =
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - xfer-from ......................................... =
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - save-to ........................................... = {data[ms]}/{JONES}-field:{sel[field]}-ddid:{sel[ddid]}.parmdb
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - dd-term ........................................... = 1
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - fix-dirs .......................................... =
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - update-type ....................................... = full
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - time-int .......................................... = 1
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - freq-int .......................................... = 1
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - clip-low .......................................... = 0.0
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - clip-high ......................................... = 0.0
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - clip-after ........................................ = 5
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - max-iter .......................................... = 20
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - conv-quorum ....................................... = 0.99
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - ref-ant ........................................... = None
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb]  - term-iters ........................................ = 20
 - 10:01:03 - main               [0.1/0.1 0.5/0.5 0.0Gb] mode: solve and correct residuals
 - 10:01:03 - data_handler       [0.1/0.1 0.5/0.5 0.0Gb] reading MS 2017-12-28_163317_0_x2_ABC1234.ms
 - 10:01:03 - data_handler       [0.1/0.1 0.5/0.5 0.0Gb]   sorting MS by TIME column
 - 10:01:03 - data_handler       [0.1/0.1 0.5/0.5 0.0Gb]   detected linear (xy) feeds
 - 10:01:03 - data_handler       [0.1/0.1 0.5/0.5 0.0Gb]   fields are *0: ABC1234
 - 10:01:03 - data_handler       [0.1/0.1 0.5/0.5 0.0Gb]   MS contains 1 spectral windows of 6480 channels each
 - 10:01:03 - data_handler       [0.1/0.1 0.5/0.5 0.0Gb]   applying a channel selection of 0:1619
 - 10:01:03 - data_handler       [0.1/0.1 0.5/0.5 0.0Gb]   applying TAQL query 'FIELD_ID == 0 && DATA_DESC_ID IN [0]' (379984/379984 rows selected)
 - 10:01:03 - data_handler       [0.1/0.1 0.5/0.5 0.0Gb]   16 antennas, 379984 rows, 1/1 DDIDs, 2794 timeslots, 1619 channels per DDID, 4 corrs
 - 10:01:03 - data_handler       [0.1/0.1 0.5/0.5 0.0Gb]   DDID central frequencies are at 0.82 GHz
 - 10:01:03 - data_handler       [0.1/0.1 0.5/0.5 0.0Gb] BITFLAG column found. Will auto-fill with 'legacy' (1) from FLAG/FLAG_ROW if not filled
 - 10:01:03 - data_handler       [0.1/0.1 0.5/0.5 0.0Gb] Applying BITFLAG ['legacy'] (1) to input data
 - 10:01:03 - data_handler       [0.1/0.1 0.5/0.5 0.0Gb] Will save new flags into BITFLAG 'cubical' (2), and into FLAG/FLAG_ROW
Loading img_SB4899_pcalfitsmask_0000_dE-sources.pybdsm.lsm.html: Tigger sky model
 - 10:01:03 - data_handler       [0.1/0.1 0.6/0.6 0.0Gb] Using 1 model(s) for 5 directions(s)
 - 10:01:03 - data_handler       [0.1/0.1 0.6/0.6 0.0Gb]   model 0 (weight WEIGHT_SPECTRUM):
 - 10:01:03 - data_handler       [0.1/0.1 0.6/0.6 0.0Gb]     direction 0: MODEL_DATA
 - 10:01:03 - data_handler       [0.1/0.1 0.6/0.6 0.0Gb]     direction 1: A250(img_SB4899_pcalfitsmask_0000_dE-sources.pybdsm.lsm.html)
 - 10:01:03 - data_handler       [0.1/0.1 0.6/0.6 0.0Gb]     direction 2: B353(img_SB4899_pcalfitsmask_0000_dE-sources.pybdsm.lsm.html)
 - 10:01:03 - data_handler       [0.1/0.1 0.6/0.6 0.0Gb]     direction 3: C026(img_SB4899_pcalfitsmask_0000_dE-sources.pybdsm.lsm.html)
 - 10:01:03 - data_handler       [0.1/0.1 0.6/0.6 0.0Gb]     direction 4: D189(img_SB4899_pcalfitsmask_0000_dE-sources.pybdsm.lsm.html)
 - 10:01:03 - main               [0.1/0.1 0.6/0.6 0.0Gb] Enabling G,dE-Jones
 - 10:01:03 - main               [0.1/0.1 0.6/0.6 0.0Gb] subtraction directions set to slice(None, None, None)
 - 10:01:03 - param_db           [0.1/0.1 0.6/0.6 0.0Gb] creating 2017-12-28_163317_0_x2_ABC1234.ms/G-field:0:spw0 in fragmented mode
 - 10:01:03 - gain_machine       [0.1/0.1 0.6/0.6 0.0Gb] G solutions will be saved to 2017-12-28_163317_0_x2_ABC1234.ms/G-field:0:spw0
 - 10:01:03 - param_db           [0.1/0.1 0.6/0.6 0.0Gb] creating 2017-12-28_163317_0_x2_ABC1234.ms/dE-field:0-ddid:None.parmdb in fragmented mode
 - 10:01:03 - gain_machine       [0.1/0.1 0.6/0.6 0.0Gb] dE solutions will be saved to 2017-12-28_163317_0_x2_ABC1234.ms/dE-field:0-ddid:None.parmdb
 - 10:01:03 - param_db           [0.1/0.1 0.6/0.6 0.0Gb] creating 2017-12-28_163317_0_x2_ABC1234.ms/BBC-field:0-ddid:None.parmdb in fragmented mode
 - 10:01:03 - gain_machine       [0.1/0.1 0.7/0.7 0.0Gb] will compute & save suggested baseline-based corrections (BBCs) to 2017-12-28_163317_0_x2_ABC1234.ms/BBC-field:0-ddid:None.parmdb
 - 10:01:03 - gain_machine       [0.1/0.1 0.7/0.7 0.0Gb]   (these can optionally be applied in a subsequent CubiCal run)
 - 10:01:03 - main               [0.1/0.1 0.7/0.7 0.0Gb] defining chunks (time 360, freq 1620, also when SCAN_NUMBER jumps > 0.0)
 - 10:01:03 - data_handler       [0.1/0.1 0.7/0.7 0.0Gb]   read indexing columns
 - 10:01:03 - data_handler       [0.1/0.1 0.7/0.7 0.0Gb]   built timeslot index (2794 unique timestamps)
 - 10:01:03 - data_handler       [0.1/0.1 0.7/0.7 0.0Gb]   using 1 freq chunks: 0 1619
 - 10:01:03 - data_handler       [0.1/0.1 0.7/0.7 0.0Gb]   found 30 time chunks: 0 92 184 276 368 460 553 645 737 830 922 1015 1108 1200 1292 1395 1494 1587 1680 1773 1866 1959 2052 2145 2238 2331 2424 2517 2610 2703 2794
 - 10:01:03 - data_handler       [0.1/0.1 0.7/0.7 0.0Gb]   generated 30 row chunks based on time and DDID
 - 10:01:03 - data_handler       [0.1/0.1 0.7/0.7 0.0Gb]   row chunks yield 30 potential tiles
 - 10:01:03 - data_handler       [0.1/0.1 0.7/0.7 0.0Gb]   coarsening this to 8 tiles (min 4 chunks per tile)
 - 10:01:03 - main               [0.1/0.1 0.9/0.9 0.0Gb] waiting for I/O on tile #1
 - 10:01:03 - main               [P1] [0.1/0.1 0.8/0.8 0.0Gb] loading tile #1/8
 - 10:01:03 - data_handler       [P1] [0.1/0.1 0.8/0.8 0.0Gb] reading MS rows 0~50047
 - 10:02:52 - data_handler       [P1] [4.9/4.9 17.7/17.7 2.5Gb]   reading MODEL_DATA for model 0 direction 0
 - 10:04:09 - main               [P1] [9.8/9.8 20.1/22.5 4.9Gb] I/O handler for load 0 save None failed with exception: Number of rows inconsistent after removing auto-correlations.
 - 10:04:09 - main               [P1] [9.8/9.8 20.1/22.5 4.9Gb] Traceback (most recent call last):
  File "/home/heywoodl/Software/CubiCal_oms_deploy/CubiCal/cubical/main.py", line 465, in _io_handler
    Tile.tile_list[load].load(load_model=load_model)
  File "/home/heywoodl/Software/CubiCal_oms_deploy/CubiCal/cubical/data_handler.py", line 358, in load
    expected_nrows, sort_ind, row_identifiers = self.prep_for_montblanc()
  File "/home/heywoodl/Software/CubiCal_oms_deploy/CubiCal/cubical/data_handler.py", line 602, in prep_for_montblanc
    raise ValueError("Number of rows inconsistent after removing auto-correlations.")
ValueError: Number of rows inconsistent after removing auto-correlations.
 - 10:04:09 - main               [0.1/0.1 0.9/0.9 4.9Gb] Exiting with exception: ValueError(Number of rows inconsistent after removing auto-correlations.)
 Traceback (most recent call last):
  File "/home/heywoodl/Software/CubiCal_oms_deploy/CubiCal/cubical/main.py", line 343, in main
    if not done or not io_futures[itile].result():
  File "/home/heywoodl/venv/cubienv/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 455, in result
    return self.__get_result()
  File "/home/heywoodl/venv/cubienv/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 414, in __get_result
    raise exception_type, self._exception, self._traceback
ValueError: Number of rows inconsistent after removing auto-correlations.
@o-smirnov
Copy link
Collaborator

Yep, time to switch to master... oms-deploy has been merged, and this error has been fixed since then...

@IanHeywood
Copy link
Collaborator Author

Thanks, running again now.

I'm getting this since switching back to master and rebuilding:

2018-01-31 10:26:05.846278: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA

Does Montblanc also need rebuilding?

Also, do you know why I might get

gocubical: error: no such option: --dE-save-to

errors when my parset looks like this?

[sol]
_Help = Solution options which apply at the solver level
jones = G,dE
...
[dE]
_Help = Options for dE-Jones term
label = dE

@JSKenyon
Copy link
Collaborator

JSKenyon commented Jan 31, 2018 •
edited
Loading

On the Montblanc front, that "error" is normal. I get it consistently. @sjperkins Would you like to comment on what it means?

Regarding the saving error, I am not sure, but it might have to do with case sensitivity. Would you mind trying to label the section in lower case and then specify the option with lower case? Just to test this theory.

The alternative is that something is going awry with the template. I would check the case thing first though.

@sjperkins
Copy link
Member

2018-01-31 10:26:05.846278: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA

Tensorflow binary wheel are targeted at a reduced instruction set in order to be compatible with a wide range of CPUs. You can read more about this here. Tensorflow 1.6 wheels will be built with AVX instructions.

In fact, this does not affect the Montblanc C++/CUDA tensorflow operators which are built from source on each install with ['-march=native', '-mtune=native']

@IanHeywood
Copy link
Collaborator Author

It's still going a day later, just over half way... 😭

Just dawned on me that I should just average the MS and test drive the interpolation features.

I'll try the dE/de thing when it's finished.

@IanHeywood
Copy link
Collaborator Author

I replaced everything that was dE with de in both parset and command line call, and now the save-to option is behaving now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sjperkins @o-smirnov @IanHeywood @JSKenyon