Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvement: don't use absolute path names in metadata cache #9

Open
awst-baum opened this issue Apr 24, 2019 · 3 comments
Open

Improvement: don't use absolute path names in metadata cache #9

awst-baum opened this issue Apr 24, 2019 · 3 comments

Comments

@awst-baum
Copy link
Contributor

The ISMN reader seems to create a cache of the ISMN data in a folder in a subfolder called python_metadata in a file called metadata.npy.

metadata.npy contains absolute pathnames of the stm files, e.g. /path/to/mydata/ISMN/ISMN_V20180830_GLOBAL/AMMA-CATCH/Banizoumbou/AMMA-CATCH_AMMA-CATCH_Banizoumbou_sm_0.050000_0.050000_CS616-1_19780101_20180830.stm.

In our setup, we use the same NFS mount containing ISMN data in several systems, and not all systems use the same mountpoint (e.g. one might have the data at /path/to/mydata and the other have it at /some/completely/different/path/to/my/data). Also, we use linux softlinks, so we might also access the data as /softlink/to/data/....

Once the metadata folder is created by one of the systems, the other systems may produce errors if they don't use the same path to access the data because they can't find the stm files at the paths cached in the metadata.npy file.

Would it be possible to not use absolute file paths or make it possible in some other way to use the same ISMN folder from several systems?

@wpreimes
Copy link
Member

maybe add an option to allow user to select where to store/load the metadata from?

@wpreimes
Copy link
Member

Generally I think the metadata handling could be improved, e.g raising a meaningful error when the data has been moved and the stored paths are not valid anymore.

@cpaulik
Copy link
Collaborator

cpaulik commented Apr 24, 2019

I would just start with relative paths. That should be pretty straightforward to implement.

Custom metadata path might be a little bit trickier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants