2023-05-17 20:33:47 CARACal INFO: Invoked as /scratch3/users/tzuyin/c-venv/bin/caracal -ct singularity -c meerkat-defaults.yml -sid /scratch3/users/tzuyin/singularity_images -sw line -ew line 2023-05-17 20:33:49 CARACal WARNING: Config file meerkat-defaults.yml schema version is 1.1.0, current CARACal version is 1.0.4 2023-05-17 20:33:49 CARACal WARNING: Will try to proceed anyway, but please be advised that configuration options may have changed. 2023-05-17 20:33:49 CARACal INFO: Loading pipeline configuration from meerkat-defaults.yml 2023-05-17 20:33:49 CARACal INFO: Version 1.0.6 installed at /scratch3/users/tzuyin/c-venv/lib/python3.9/site-packages/caracal 2023-05-17 20:33:49 CARACal INFO: 2023-05-17 20:33:49 CARACal INFO: general: 2023-05-17 20:33:49 CARACal INFO: title: "" 2023-05-17 20:33:49 CARACal INFO: msdir: data1 2023-05-17 20:33:49 CARACal INFO: rawdatadir: "" 2023-05-17 20:33:49 CARACal INFO: input: input 2023-05-17 20:33:49 CARACal INFO: output: output 2023-05-17 20:33:49 CARACal INFO: prefix: mypipelinerun 2023-05-17 20:33:49 CARACal INFO: prep_workspace: True 2023-05-17 20:33:49 CARACal INFO: init_notebooks: std-progress-report, project-logs 2023-05-17 20:33:49 CARACal INFO: report_notebooks: [detailed-final-report] 2023-05-17 20:33:49 CARACal INFO: final_report: False 2023-05-17 20:33:49 CARACal INFO: backend: docker 2023-05-17 20:33:49 CARACal INFO: 2023-05-17 20:33:49 CARACal INFO: getdata: 2023-05-17 20:33:49 CARACal INFO: dataid: [1671075077_sdp_l0] 2023-05-17 20:33:49 CARACal INFO: extension: ms 2023-05-17 20:33:49 CARACal INFO: report: False 2023-05-17 20:33:49 CARACal INFO: ignore_missing: False 2023-05-17 20:33:49 CARACal INFO: 2023-05-17 20:33:49 CARACal INFO: obsconf: 2023-05-17 20:33:49 CARACal INFO: target: [all] 2023-05-17 20:33:49 CARACal INFO: gcal: [all] 2023-05-17 20:33:49 CARACal INFO: bpcal: [longest] 2023-05-17 20:33:49 CARACal INFO: fcal: [longest] 2023-05-17 20:33:49 CARACal INFO: xcal: [longest] 2023-05-17 20:33:49 CARACal INFO: refant: 0 2023-05-17 20:33:49 CARACal INFO: maxdist: 1000.0 2023-05-17 20:33:49 CARACal INFO: minbase: 150.0 2023-05-17 20:33:49 CARACal INFO: report: False 2023-05-17 20:33:49 CARACal INFO: obsinfo: 2023-05-17 20:33:49 CARACal INFO: enable: True 2023-05-17 20:33:49 CARACal INFO: listobs: True 2023-05-17 20:33:49 CARACal INFO: summary_json: True 2023-05-17 20:33:49 CARACal INFO: vampirisms: False 2023-05-17 20:33:49 CARACal INFO: plotelev: 2023-05-17 20:33:49 CARACal INFO: enable: True 2023-05-17 20:33:49 CARACal INFO: plotter: owlcat 2023-05-17 20:33:49 CARACal INFO: 2023-05-17 20:33:49 CARACal INFO: line: 2023-05-17 20:33:49 CARACal INFO: enable: True 2023-05-17 20:33:49 CARACal INFO: label_in: corr 2023-05-17 20:33:49 CARACal INFO: line_name: HI 2023-05-17 20:33:49 CARACal INFO: restfreq: 1.420405752GHz 2023-05-17 20:33:49 CARACal INFO: ncpu: 0 2023-05-17 20:33:49 CARACal INFO: overwrite_flagvers: False 2023-05-17 20:33:49 CARACal INFO: report: False 2023-05-17 20:33:49 CARACal INFO: rewind_flags: 2023-05-17 20:33:49 CARACal INFO: enable: True 2023-05-17 20:33:49 CARACal INFO: mode: reset_worker 2023-05-17 20:33:49 CARACal INFO: version: auto 2023-05-17 20:33:49 CARACal INFO: mstransform_version: auto 2023-05-17 20:33:49 CARACal INFO: subtractmodelcol: 2023-05-17 20:33:49 CARACal INFO: enable: True 2023-05-17 20:33:49 CARACal INFO: mstransform: 2023-05-17 20:33:49 CARACal INFO: enable: True 2023-05-17 20:33:49 CARACal INFO: col: corrected 2023-05-17 20:33:49 CARACal INFO: obsinfo: True 2023-05-17 20:33:49 CARACal INFO: doppler: 2023-05-17 20:33:49 CARACal INFO: enable: True 2023-05-17 20:33:49 CARACal INFO: telescope: meerkat 2023-05-17 20:33:49 CARACal INFO: mode: frequency 2023-05-17 20:33:49 CARACal INFO: frame: bary 2023-05-17 20:33:49 CARACal INFO: veltype: radio 2023-05-17 20:33:49 CARACal INFO: changrid: auto 2023-05-17 20:33:49 CARACal INFO: uvlin: 2023-05-17 20:33:49 CARACal INFO: enable: True 2023-05-17 20:33:49 CARACal INFO: fitorder: 1 2023-05-17 20:33:49 CARACal INFO: fitspw: "" 2023-05-17 20:33:49 CARACal INFO: exclude_known_sources: False 2023-05-17 20:33:49 CARACal INFO: known_sources_cat: "" 2023-05-17 20:33:49 CARACal INFO: known_sources_radius: 1.0 2023-05-17 20:33:49 CARACal INFO: known_sources_flux: 0.0 2023-05-17 20:33:49 CARACal INFO: known_sources_dv: 30.0 2023-05-17 20:33:49 CARACal INFO: make_cube: 2023-05-17 20:33:49 CARACal INFO: enable: True 2023-05-17 20:33:49 CARACal INFO: image_with: wsclean 2023-05-17 20:33:49 CARACal INFO: use_mstransform: True 2023-05-17 20:33:49 CARACal INFO: stokes: I 2023-05-17 20:33:49 CARACal INFO: spwid: 0 2023-05-17 20:33:49 CARACal INFO: nchans: 0 2023-05-17 20:33:49 CARACal INFO: firstchan: 0 2023-05-17 20:33:49 CARACal INFO: binchans: 1 2023-05-17 20:33:49 CARACal INFO: npix: [1800] 2023-05-17 20:33:49 CARACal INFO: cell: 2.0 2023-05-17 20:33:49 CARACal INFO: padding: 1.2 2023-05-17 20:33:49 CARACal INFO: weight: briggs 2023-05-17 20:33:49 CARACal INFO: robust: 0.3 2023-05-17 20:33:49 CARACal INFO: taper: 0.0 2023-05-17 20:33:49 CARACal INFO: niter: 1000000 2023-05-17 20:33:49 CARACal INFO: gain: 0.1 2023-05-17 20:33:49 CARACal INFO: wscl_onlypsf: False 2023-05-17 20:33:49 CARACal INFO: wscl_mgain: 1.0 2023-05-17 20:33:49 CARACal INFO: wscl_sofia_niter: 2 2023-05-17 20:33:49 CARACal INFO: wscl_sofia_converge: 1.1 2023-05-17 20:33:49 CARACal INFO: wscl_removeintermediate: False 2023-05-17 20:33:49 CARACal INFO: wscl_user_clean_mask: "" 2023-05-17 20:33:49 CARACal INFO: wscl_auto_mask: 10.0 2023-05-17 20:33:49 CARACal INFO: wscl_auto_thr: 0.5 2023-05-17 20:33:49 CARACal INFO: wscl_make_cube: True 2023-05-17 20:33:49 CARACal INFO: wscl_noupdatemod: True 2023-05-17 20:33:49 CARACal INFO: wscl_multiscale: False 2023-05-17 20:33:49 CARACal INFO: wscl_multiscale_scales: "" 2023-05-17 20:33:49 CARACal INFO: wscl_multiscale_bias: 0.6 2023-05-17 20:33:49 CARACal INFO: wscl_nrdeconvsubimg: 1 2023-05-17 20:33:49 CARACal INFO: wscl_beam: 0, 0, 0 2023-05-17 20:33:49 CARACal INFO: casa_thr: 10mJy 2023-05-17 20:33:49 CARACal INFO: casa_port2fits: False 2023-05-17 20:33:49 CARACal INFO: sofia: 2023-05-17 20:33:49 CARACal INFO: enable: True 2023-05-17 20:33:49 CARACal INFO: flag: False 2023-05-17 20:33:49 CARACal INFO: flagregion: 0, 0, 0, 0, 0, 0 2023-05-17 20:33:49 CARACal INFO: rmsMode: mad 2023-05-17 20:33:49 CARACal INFO: thr: 4.0 2023-05-17 20:33:49 CARACal INFO: merge: False 2023-05-17 20:33:49 CARACal INFO: mergeX: 2 2023-05-17 20:33:49 CARACal INFO: mergeY: 2 2023-05-17 20:33:49 CARACal INFO: mergeZ: 3 2023-05-17 20:33:49 CARACal INFO: minSizeX: 3 2023-05-17 20:33:49 CARACal INFO: minSizeY: 3 2023-05-17 20:33:49 CARACal INFO: minSizeZ: 3 2023-05-17 20:33:49 CARACal INFO: cubelets: True 2023-05-17 20:33:49 CARACal INFO: mom0: True 2023-05-17 20:33:49 CARACal INFO: mom1: True 2023-05-17 20:34:01 CARACal INFO: output directory for logs is output/logs-20230517-203349 2023-05-17 20:34:01 CARACal INFO: output/logs links to logs-20230517-203349 2023-05-17 20:34:01 CARACal INFO: output/log-caracal.txt links to logs-20230517-203349/log-caracal.txt 2023-05-17 20:34:01 CARACal INFO: Copying MeerKAT input files into input folder 2023-05-17 20:34:01 CARACal INFO: Creating standard notebook output/mypipelinerun-detailed-final-report.ipynb from template 2023-05-17 20:34:01 CARACal INFO: Creating standard notebook output/mypipelinerun-std-progress-report.ipynb from template 2023-05-17 20:34:01 CARACal INFO: Creating standard notebook output/mypipelinerun-project-logs.ipynb from template 2023-05-17 20:34:01 CARACal INFO: Saving original configuration file as output/cfgFiles/meerkat-defaults-20230517-203349.orig.yml 2023-05-17 20:34:01 CARACal INFO: Saving full configuration as output/cfgFiles/meerkat-defaults-20230517-203349.yml 2023-05-17 20:34:01 CARACal INFO: Configuring worker getdata 2023-05-17 20:34:01 CARACal INFO: Configuring worker obsconf 2023-05-17 20:34:01 CARACal INFO: Configuring worker line 2023-05-17 20:34:01 CARACal INFO: getdata: initializing 2023-05-17 20:34:01 CARACal INFO: getdata: running 2023-05-17 20:34:01 CARACal INFO: getdata: finished 2023-05-17 20:34:01 CARACal INFO: obsconf: initializing 2023-05-17 20:34:01 CARACal INFO: obsinfo file 1671075077_sdp_l0-obsinfo.txt exists, not regenerating 2023-05-17 20:34:01 CARACal INFO: summary file 1671075077_sdp_l0-summary.json exists, not regenerating 2023-05-17 20:34:01 CARACal INFO: elevation plot 1671075077_sdp_l0-elevation-tracks.png exists, not regenerating 2023-05-17 20:34:01 CARACal INFO: MS #0: 1671075077_sdp_l0.ms 2023-05-17 20:34:02 CARACal INFO: 1 spectral windows, with NCHAN=4096 2023-05-17 20:34:02 CARACal INFO: CHAN_FREQ from 856000000.0 Hz to 1711791015.625 Hz with average channel width of 208984.375 Hz 2023-05-17 20:34:02 CARACal INFO: target (TARGET): 2023-05-17 20:34:02 CARACal INFO: FRB20211212 (ID=3) : 119.95 minutes | RA=157.35 deg, Dec=1.36 deg 2023-05-17 20:34:02 CARACal INFO: gcal (CALIBRATE_AMPL,CALIBRATE_PHASE): 2023-05-17 20:34:02 CARACal INFO: J1058+0133 (ID=2) : 11.76 minutes | RA=164.62 deg, Dec=1.57 deg 2023-05-17 20:34:02 CARACal INFO: fcal (CALIBRATE_FLUX): 2023-05-17 20:34:02 CARACal INFO: J0408-6545 (ID=0) : 10.16 minutes | RA=62.08 deg, Dec=-65.75 deg 2023-05-17 20:34:02 CARACal INFO: bpcal (CALIBRATE_BANDPASS): 2023-05-17 20:34:02 CARACal INFO: J0408-6545 (ID=0) : 10.16 minutes | RA=62.08 deg, Dec=-65.75 deg 2023-05-17 20:34:02 CARACal INFO: obsconf: running 2023-05-17 20:34:02 CARACal INFO: obsconf: finished 2023-05-17 20:34:02 CARACal INFO: line: initializing 2023-05-17 20:34:02 CARACal INFO: MS #0: 1671075077_sdp_l0-FRB20211212-corr.ms 2023-05-17 20:34:02 CARACal INFO: 1 spectral windows, with NCHAN=1024 2023-05-17 20:34:02 CARACal INFO: CHAN_FREQ from 856313476.5625 Hz to 1711477539.0625 Hz with average channel width of 835937.5 Hz 2023-05-17 20:34:02 CARACal INFO: Target RA, Dec for Doppler correction: 157.351 deg, 1.360 deg 2023-05-17 20:34:03 CARACal INFO: Calculated common Doppler-corrected channel grid for all input .MS: 1022 channels starting at 857065474.062Hz and with channel width 835855.637Hz. 2023-05-17 20:34:03 CARACal INFO: removing pre-existing data1/1671075077_sdp_l0-FRB20211212-corr_mst.ms 2023-05-17 20:34:03 CARACal.Stimela.modelsub-ms0 INFO: job started at 2023-05-17 20:34:03.950667 # WARNING: Overriding HOME environment variable with SINGULARITYENV_HOME is not permitted # INFO: Converting SIF file to temporary sandbox... # WARNING: underlay of /etc/localtime required more than 50 (105) bind mounts # Successful read/write open of default-locked table /stimela_mount/msdir/1671075077_sdp_l0-FRB20211212-corr.ms: 27 columns, 7046424 rows # INFO: Cleaning up image... # Wrtiting to column CORRECTED_DATA (rows 0 to 704641) # Wrtiting to column CORRECTED_DATA (rows 704642 to 1409283) # Wrtiting to column CORRECTED_DATA (rows 1409284 to 2113925) # Wrtiting to column CORRECTED_DATA (rows 2113926 to 2818567) # Wrtiting to column CORRECTED_DATA (rows 2818568 to 3523209) # Wrtiting to column CORRECTED_DATA (rows 3523210 to 4227851) # Wrtiting to column CORRECTED_DATA (rows 4227852 to 4932493) # Wrtiting to column CORRECTED_DATA (rows 4932494 to 5637135) # Wrtiting to column CORRECTED_DATA (rows 5637136 to 6341777) # Wrtiting to column CORRECTED_DATA (rows 6341778 to 7046419) # Wrtiting to column CORRECTED_DATA (rows 7046420 to 7046423) 2023-05-17 21:28:55 CARACal.Stimela.modelsub-ms0 INFO: job complete at 2023-05-17 21:28:55.094462 after 0:54:51.143795 2023-05-17 21:28:55 CARACal.Stimela.mstransform-ms0 INFO: job started at 2023-05-17 21:28:55.094854 # WARNING: Overriding HOME environment variable with SINGULARITYENV_HOME is not permitted # INFO: Converting SIF file to temporary sandbox... # 2023-05-17 21:29:03: Initial memory state: # 2023-05-17 21:29:03: total used free shared buff/cache available # 2023-05-17 21:29:03: Mem: 503Gi 2.5Gi 76Gi 2.0Mi 424Gi 497Gi # 2023-05-17 21:29:03: Swap: 0B 0B 0B # # ========================================= # The start-up time of CASA may vary # depending on whether the shared libraries # are cached or not. # ========================================= # # Python: users/tzuyinIPython 5.1.0 -- An enhanced Interactive Python. # # CASA 5.8.0-109 -- Common Astronomy Software Applications # # Creating a new telemetry file # Telemetry initialized. Telemetry will send anonymized usage statistics to NRAO. # You can disable telemetry by adding the following line to your ~/.casarc file: # EnableTelemetry: False # 2023-05-17 19:29:04 INFO ::casa CASA Version 5.8.0-109 # --> CrashReporter initialized. # 2023-05-17 19:29:06 INFO mstransform:::: # 2023-05-17 19:29:06 INFO mstransform::::+ ########################################## # 2023-05-17 19:29:06 INFO mstransform::::+ ##### Begin Task: mstransform ##### # 2023-05-17 19:29:06 INFO mstransform:::: mstransform(vis="/stimela_mount/msdir/1671075077_sdp_l0-FRB20211212-corr.ms",outputvis="/stimela_mount/msdir/1671075077_sdp_l0-FRB20211212-corr_mst.ms",createmms=False,separationaxis="auto",numsubms="auto", # 2023-05-17 19:29:06 INFO mstransform::::+ tileshape=[0],field="",spw="",scan="",antenna="", # 2023-05-17 19:29:06 INFO mstransform::::+ correlation="",timerange="",intent="",array="",uvrange="", # 2023-05-17 19:29:06 INFO mstransform::::+ observation="",feed="",datacolumn="corrected",realmodelcol=False,keepflags=True, # 2023-05-17 19:29:06 INFO mstransform::::+ usewtspectrum=False,combinespws=False,chanaverage=False,chanbin=1,hanning=False, # 2023-05-17 19:29:06 INFO mstransform::::+ regridms=True,mode="frequency",nchan=1022,start="857065474.062Hz",width="835855.637Hz", # 2023-05-17 19:29:06 INFO mstransform::::+ nspw=1,interpolation="nearest",phasecenter="",restfreq="1.420405752GHz",outframe="bary", # 2023-05-17 19:29:06 INFO mstransform::::+ veltype="radio",preaverage=False,timeaverage=False,timebin="0s",timespan="", # 2023-05-17 19:29:06 INFO mstransform::::+ maxuvwdistance=0.0,docallib=False,callib="",douvcontsub=True,fitspw="", # 2023-05-17 19:29:06 INFO mstransform::::+ fitorder=1,want_cont=False,denoising_lib=True,nthreads=1,niter=1, # 2023-05-17 19:29:06 INFO mstransform::::+ disableparallel=False,ddistart=-1,taql="",monolithic_processing=False,reindex=True) # 2023-05-17 19:29:08 INFO mstransform:::: Parse regridding parameters # 2023-05-17 19:29:08 INFO ParallelDataHelper:::: Parse uvcontsub parameters # 2023-05-17 19:29:08 INFO MSTransformManager::parseMsSpecParams Input file name is /stimela_mount/msdir/1671075077_sdp_l0-FRB20211212-corr.ms # 2023-05-17 19:29:08 INFO MSTransformManager::parseMsSpecParams Data column is CORRECTED # 2023-05-17 19:29:08 INFO MSTransformManager::parseMsSpecParams Output file name is /stimela_mount/msdir/1671075077_sdp_l0-FRB20211212-corr_mst.ms # 2023-05-17 19:29:08 INFO MSTransformManager::parseMsSpecParams Re-index is enabled # 2023-05-17 19:29:08 INFO MSTransformManager::parseMsSpecParams Tile shape is [0] # 2023-05-17 19:29:08 INFO MSTransformManager::parseRefFrameTransParams Regrid MS is activated # 2023-05-17 19:29:08 INFO MSTransformManager::parseRefFrameTransParams Rest frequency is 1.420405752GHz # 2023-05-17 19:29:08 INFO MSTransformManager::parseRefFrameTransParams Output reference frame is bary # 2023-05-17 19:29:08 INFO MSTransformManager::parseRefFrameTransParams Interpolation method is nearest # 2023-05-17 19:29:08 INFO MSTransformManager::parseFreqSpecParams Mode is frequency # 2023-05-17 19:29:08 INFO MSTransformManager::parseFreqSpecParams Number of output channels is 1022 # 2023-05-17 19:29:08 INFO MSTransformManager::parseFreqSpecParams Start is 857065474.062Hz # 2023-05-17 19:29:08 INFO MSTransformManager::parseFreqSpecParams Width is 835855.637Hz # 2023-05-17 19:29:08 INFO MSTransformManager::parseUVContSubParams Continuum subtraction is activated # 2023-05-17 19:29:08 INFO MSTransformManager::colCheckInfo Adding DATA column to output MS from input CORRECTED_DATA column # 2023-05-17 19:29:08 INFO MSTransformManager::open Select data # 2023-05-17 19:29:08 INFO MSTransformManager::createOutputMSStructure Create output MS structure # 2023-05-17 19:29:10 INFO MSTransformManager::checkFillWeightSpectrum Optional column WEIGHT_SPECTRUM found in input MS will be written to output MS # 2023-05-17 19:29:10 INFO MSTransformManager::regridSpwSubTable Regridding SPW with Id 0 # 2023-05-17 19:29:10 INFO MSTransformManager::regridSpwAux Input SPW: 1024 channels, first channel = 8.563134766e+08 Hz, last channel = 1.711477539e+09 Hz, first width = 8.359375000e+05 Hz, last width = 8.359375000e+05 Hz # 2023-05-17 19:29:10 INFO MSTransformManager::regridSpwAux Calculate frequencies in output reference frame # 2023-05-17 19:29:10 INFO MeasIERS::fillMeas(MeasIERS::Files, Double) Requested JD 59928.2 is outside the range of the IERS (Earth axis data) table. # 2023-05-17 19:29:10 INFO MeasIERS::fillMeas(MeasIERS::Files, Double) + Calculations will proceed with less precision # 2023-05-17 19:29:10 SEVERE MeasTable::dUTC(Double) (file ../../measures/Measures/MeasTable.cc, line 4290) Leap second table TAI_UTC seems out-of-date. # 2023-05-17 19:29:10 SEVERE MeasTable::dUTC(Double) (file ../../measures/Measures/MeasTable.cc, line 4290)+ Until the table is updated (see the CASA documentation or your system admin), # 2023-05-17 19:29:10 SEVERE MeasTable::dUTC(Double) (file ../../measures/Measures/MeasTable.cc, line 4290)+ times and coordinates derived from UTC could be wrong by 1s or more. # 2023-05-17 19:29:10 WARN MSTransformRegridder::calcChanFreqs *** Requested new channel width (835856 Hz) is smaller than smallest original channel width # 2023-05-17 19:29:10 WARN MSTransformRegridder::calcChanFreqs+ which is 835857 Hz # 2023-05-17 19:29:10 SEVERE MSTransformRegridder::calcChanFreqs calcChanFreqs failed, check input start and width parameters # 2023-05-17 19:29:11 SEVERE ParallelDataHelper:::: Task mstransform raised an exception of class RuntimeError with the following message: 2023-05-17 19:29:10 SEVERE MSTransformRegridder::calcChanFreqs calcChanFreqs failed, check input start and width parameters # Running CASA task 'mstransform' # Traceback (most recent call last): # INFO: Cleaning up image... # File "/stimela_mount/code/run.py", line 8, in # task.run() # File "/usr/local/lib/python3.8/dist-packages/Crasa/Crasa.py", line 122, in run # raise CasaException("CASA raised a SEVERE exception while running task {0:s}".format(self.task)) # Crasa.Crasa.CasaException: CASA raised a SEVERE exception while running task mstransform 2023-05-17 21:29:12 CARACal.Stimela.mstransform-ms0 ERROR: cd /scratch3/users/tzuyin/reduction/.stimela_workdir-16843484421262705 && singularity run --userns --workdir /scratch3/users/tzuyin/reduction/.stimela_workdir-16843484421262705 --containall returns error code 1 2023-05-17 21:29:12 CARACal.Stimela.mstransform-ms0 ERROR: job failed at 2023-05-17 21:29:12.253840 after 0:00:17.158986 2023-05-17 21:29:12 CARACal ERROR: Job 'mstransform-ms0:: Doppler tracking corrections' failed: cd /scratch3/users/tzuyin/reduction/.stimela_workdir-16843484421262705 && singularity run --userns --workdir /scratch3/users/tzuyin/reduction/.stimela_workdir-16843484421262705 --containall returns error code 1 [PipelineException] 2023-05-17 21:29:12 CARACal INFO: More information can be found in the logfile at output/logs-20230517-203349/log-caracal.txt 2023-05-17 21:29:12 CARACal INFO: exiting with error code 1