Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MS descriptors don't support optional columns #271

Closed
sjperkins opened this issue Nov 9, 2022 · 2 comments · Fixed by #270
Closed

MS descriptors don't support optional columns #271

sjperkins opened this issue Nov 9, 2022 · 2 comments · Fixed by #270

Comments

@sjperkins
Copy link
Member 

    I'm getting this error when running the command @sjperkins :

2022-11-02 10:34:29,585 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21032324 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,592 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21037373 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,597 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21037373 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,602 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21042422 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,607 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21042422 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,613 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21047471 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,617 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21047471 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,624 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21088424 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,629 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21088424 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,634 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21092912 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,640 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21092912 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,646 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21097400 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,651 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21097400 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,657 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21101888 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,661 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21101888 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,666 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21160232 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,672 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21160232 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,678 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21164720 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,683 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21164720 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,689 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21169208 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,694 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21169208 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,699 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21173696 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,704 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21173696 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,711 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21232040 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,716 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21232040 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,721 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21236528 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,726 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21236528 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,732 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21241016 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,736 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21241016 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,743 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21245504 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,748 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21245504 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,753 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21303848 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,758 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21303848 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,764 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21308336 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,769 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21308336 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,775 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21312824 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,780 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21312824 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:29,786 - dask-ms - WARNING - Ignoring 'WEIGHT_SPECTRUM': Unable to infer shape of column 'WEIGHT_SPECTRUM' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21317312 of column WEIGHT_SPECTRUM in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f19'
2022-11-02 10:34:29,792 - dask-ms - WARNING - Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 21317312 of column FLAG_CATEGORY in /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg/table.f18'
2022-11-02 10:34:31,247 - dask-ms - INFO - Input: 'measurementset' file:///home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg
2022-11-02 10:34:31,247 - dask-ms - INFO - Output: 'measurementset' file:///home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg_time
2022-11-02 10:35:21,005 - dask-ms - WARNING - The shape of column 'ASSOC_SPW_ID' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,011 - dask-ms - WARNING - The shape of column 'ASSOC_NATURE' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,015 - dask-ms - WARNING - The shape of column 'ASSOC_SPW_ID' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,021 - dask-ms - WARNING - The shape of column 'ASSOC_NATURE' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,025 - dask-ms - WARNING - The shape of column 'ASSOC_SPW_ID' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,031 - dask-ms - WARNING - The shape of column 'ASSOC_NATURE' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,036 - dask-ms - WARNING - The shape of column 'ASSOC_SPW_ID' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,042 - dask-ms - WARNING - The shape of column 'ASSOC_NATURE' is unconstrained (ndim == -1). Assuming shape is (31,) from exemplar
2022-11-02 10:35:21,521 - dask-ms - WARNING - Ignoring SOURCE
2022-11-02 10:35:21,525 - dask-ms - WARNING - Ignoring 'TARGET': Unable to infer shape of column 'TARGET' due to:
'TableProxy::getCell: no such row'
2022-11-02 10:35:21,526 - dask-ms - WARNING - Ignoring 'ENCODER': Unable to infer shape of column 'ENCODER' due to:
'TableProxy::getCell: no such row'
2022-11-02 10:35:21,527 - dask-ms - WARNING - Ignoring 'POINTING_OFFSET': Unable to infer shape of column 'POINTING_OFFSET' due to:
'TableProxy::getCell: no such row'
2022-11-02 10:35:21,527 - dask-ms - WARNING - Ignoring 'DIRECTION': Unable to infer shape of column 'DIRECTION' due to:
'TableProxy::getCell: no such row'
Traceback (most recent call last):
  File "/home/vicente/anaconda3/envs/pyralysis2/bin/dask-ms", line 8, in <module>
    sys.exit(main())
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/apps/entrypoint.py", line 9, in main
    return EntryPoint(sys.argv[1:]).execute()
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/apps/entrypoint.py", line 33, in execute
    cmd.execute()
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/apps/convert.py", line 415, in execute
    writes = self.convert_table(self.args)
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/apps/convert.py", line 500, in convert_table
    writes.append(writer(datasets, out_store))
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/dask_ms.py", line 102, in xds_to_table
    out_ds = write_datasets(
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/writes.py", line 760, in write_datasets
    tp = _updated_table(table, datasets, columns, descriptor)
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/writes.py", line 338, in _updated_table
    table_proxy.addcols(_table_desc, dminfo=_dminfo).result()
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/daskms/table_proxy.py", line 114, in _impl
    return getattr(table, method)(*args, **kwargs)
  File "/home/vicente/anaconda3/envs/pyralysis2/lib/python3.8/site-packages/casacore/tables/table.py", line 1226, in addcols
    self._addcols(tdesc, dminfo, addtoparent)
RuntimeError: Invalid Table operation: Data manager name StandardStMan is already used in table /home/vicente/Documentos/Ayudantia/complete_data/HLTau_B6cont.calavg_time/POINTING

Originally posted by @miguelcarcamov in #159 (comment)
@sjperkins
Copy link
Member Author

@sjperkins

Maybe if I add the link to the ms here you can traceback the error?

The command line that I'm currently using is:

dask-ms convert HLTau_B6cont.calavg.tav300s -g "FIELD_ID,DATA_DESC_ID,SCAN_NUMBER" -i "ANTENNA1,ANTENNA2,TIME,FEED1,FEED2" -o output.ms --format ms --force

I'm not creating any folder before that.

Originally posted by @miguelcarcamov in #159 (comment)

@sjperkins
Copy link
Member Author

@miguelcarcamov I've moved the discussion from #159 into this issue as there are a number of separate concerns that must be addressed.

@sjperkins sjperkins mentioned this issue Nov 9, 2022
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support optional CASA columns ratt-ru/dask-ms
1 participant
@sjperkins