auto update casa measure data

[kswang1029]

based on https://casa.nrao.edu/casadocs/casa-6.1.0/external-data/casa-data-repository it is suggested to update casa measure data regularly. The above URL contains the information on how to get it updated.

[kswang1029]

I suggest @ajm-asiaa and one backend developer to work together on this.

[jolopezl]

Some work log for the implementation details:

After a quick conversation with @ajm-asiaa to help me understand the build and packaging to CARTA, we think a reasonable approach could be to update the data using a bash script and let the CASA data reside in $HOME/.carta/casa-data. Then, we can add something to the CARTA start script in scripts/carta and call for a data update, depending on the parameters provided (could be carta --update-casa-data) or something else, for instance, periodically.

@veggiesaurus, as you pointed out too: casa-data needs to reside in $HOME/.carta/data optionally. So, it should first look in the home dir and then fall back to the usual place (e.g. /usr/share/casacore/data).

[ajm-ska]

It seems that unless we modify carta-casacore, we will need to use the -DDATA_DIR="%CASAROOT%" carta-casacore cmake build flag and then call export CASAPATH= before starting CARTA.

I guess a bash startup script could check for the presence of the 'geodetic' folder in $HOME/.carta/casa-data first and if present run export CASAPATH=$HOME/.carta/casa-data, if not present then run export CASAPATH=/usr/share/casacore/data.

I did a few experiments to see how that could be improved.

I tried adding two search paths to -DDATA_DIR. e.g. -DDATA_DIR=~/.carta/casa-data:/usr/local/share/casacore/data although I know the first problem is it would lock it to my $HOME and not keep it as a variable to work on other systems. But either way, it doesn't work as it only looks in the first path and ignores the second. Adding two paths to the export CASAPATH= does not work either.

I should mention that if we use -DDATA_DIR="%CASAROOT%" then that hard-codes in CASAROOT to the carta-casacore build path. For example, if I run export CASAPATH=/usr/local/share/casacore/data and then start the carta_backend, the error is:

WARN	MeasIERS::findTab (file /Users/ajm/carta-casacore/casa6/casatools/casacore/measures/Measures/MeasIERS.cc, line 387)+/Users/ajm/carta-casacore/build/usr/share/casacore/data/geodetic/

So we need to back out of the directory structure by adding ../../../../ e.g. export CASAPATH=../../../../usr/local/share/casacore/data.

You may be wondering, why not just build carta-casacore with -DDATA_DIR="/"?
I tried it and it does not work. It appears we need to use %CASAROOT% in order for export CASAPATH to have any affect.

%CASAHOME% is another variable in the code, but that seems to point towards $HOME/aips++ on top of CASAROOT. e.g.:

WARN	MeasIERS::findTab (file /Users/ajm/carta-casacore/casa6/casatools/casacore/measures/Measures/MeasIERS.cc, line 387)+/Users/ajm/carta-casacore/build/Users/ajm/aips++/data/geodetic/

Now that I have looked at this closely, a loop in a bash script would be the easiest option, but maybe modifying our carta-casacore would be the more elegant solution? At least to get rid of the need to have ../../../../ while CASAPATH option is enabled?

[jolopezl]

@ajm-asiaa, so we should add the following:

• Same auto-update script for all supported distributions (preliminary version in Add casa_data_autoupdate bash script #1008): the auto-updated casa data is user-only. If we want a system-wide update we can rely on distribution-dependent packages.

• Preferences should not go in ~/.carta/backend.json, but in ~/.carta/config/preferences.json. With that change, we can add an option to switch the auto-update and the auto-update interval on and off from the CARTA front-end (add preferences to support auto update casa measure data carta-frontend#1764).

[ajm-ska]

A few points on 6efae98

• The current plan is that the carta script executes casa_data_autoupdate every time (Note the Debian and RPM packages will need to copy casa_data_autoupdate to /usr/bin/`).

• casa_data_autoupdate only takes action if the user has manually added the following lines to their ~/.carta/config/preferences.json file. Instructions about this would be added to the User Manual. It is still undecided whether there would be a carta-frontend option to control this.

    "auto_update_casa_data": true,
    "auto_update_casa_data_interval": 30

• It relies on the change to carta-casacore that checks for the presence of a ~/.carta/casarc file. When present, this file overwrites the default /usr/share/casacore/data/geodetic location and points towards a user-accessible location:

measures.observatory.directory: ~/.carta/data/geodetic

• From testing, CARTA appears to work perfectly fine with only geodetic/Observatories present. I'm not sure if the rest of the geodetic folder and all of the ephemerides folder are actually needed.

• The only contents of casa-data that appear to regularly update are geodetic/TAI_UTC. Does CARTA use that?

• As the old https://svn.cv.nrao.edu/svn/casa-data repository will apparently become private soon, we need to get the updated measures data elsewhere. Here we can get it from ftp://ftp.astron.nl/outgoing/Measures/WSRT_Measures.ztar using wget. That might be good because I imagine it is more likely for wget to be installed on a computer rather than svn. However, it appears even though WSRT_Measures.ztar is updated daily, only the contents of geodetic/TAI_UTC is actually updated.
  It is unfortunate that we need to duplicate most of the data by downloading the entire 19MB file.
  From the NRAO SVN repo, we could just download the contents of geodetic/Observatories and geodetic/TAI_UTC which are only a few kB.

• The "datediff" function needs to be improved.

• One shortcoming of this entire scheme is that it will not work for beta releases as they use ~/.carta-beta and we have built carta-casacore with~/.carta/casarc hard-coded. We could easily change carta-casacore when building the macOS Electron and Linux AppImage versions because they bundle separate casacore lib files, but it would be more difficult to have a separate carta-casacore version for the Debian and RPM versions, and probably not worth it. The CARTA beta versions are not intended for long-term use anyway, so it probably isn't a problem.

[confluence]

For beta releases, can the user manually create the appropriate files in .carta (and symlink them to .carta-beta if necessary)?

This should be documented. I don't think that this is likely to be an issue for the Ubuntu packages, since users can install the Kern casacore data package. But it should be available as an option.

In future it may be a good idea for us to revise how we separate the release and beta files, and where we put files which can safely be shared by both.