New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve the data management toolbox #369

Merged
merged 1 commit into from Oct 21, 2015

Conversation

Projects
None yet
3 participants
@cdeil
Member

cdeil commented Oct 20, 2015

This PR improves the data management toolbox in Gammapy.
The setup is a bit elaborate, but I think it will work nicely for all HESS FITS data, and hopefully it's intuitive and simple to use.

The main idea is to introduce a file index table containing the actual filenames.
This extra level of indirection means that Gammapy doesn't have to support multiple file / folder layouts, it's up to the data provider (HESS in this case) to produce a file index table in the appropriate format (which I'm doing for HAP and PARISANALYSIS).
It will also make it faster and easier to download, check and select data.

I've also implemented #354 here by bundling pathlib (this version: https://pypi.python.org/pypi/pathlib2/, https://github.com/mcmtroffaes/pathlib2/blob/develop/pathlib2.py) as gammapy/extern/pathlib.py.

@cdeil cdeil self-assigned this Oct 20, 2015

@cdeil cdeil added this to the 0.4 milestone Oct 20, 2015

Show outdated Hide outdated gammapy/conftest.py Outdated

cdeil added a commit that referenced this pull request Oct 21, 2015

Merge pull request #369 from cdeil/datastore-hess
Improve the data management toolbox

@cdeil cdeil merged commit 08c9f29 into gammapy:master Oct 21, 2015

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi Oct 27, 2015

Contributor

Hi,

I am testing the DataManager/DataStore at the moment, I found some issues and I am not sure wheater I should fix them or if this is stuff that you're working on

I left some inline comments

Contributor

joleroi commented Oct 27, 2015

Hi,

I am testing the DataManager/DataStore at the moment, I found some issues and I am not sure wheater I should fix them or if this is stuff that you're working on

I left some inline comments

elif path2.is_file():
filename = path2
else:
raise FileNotFoundError('File not found at {} or {}'.format(path1, path2))

This comment has been minimized.

@joleroi

joleroi Oct 27, 2015

Contributor

For me (using Python 2.7.10) this raises

NameError: global name 'FileNotFoundError' is not defined

@joleroi

joleroi Oct 27, 2015

Contributor

For me (using Python 2.7.10) this raises

NameError: global name 'FileNotFoundError' is not defined

This comment has been minimized.

@cdeil

cdeil Oct 27, 2015

Member

I already fixed this in #364 and changed quite a few other things (e.g. how test data is accessed).
Probably easiest if I merge that PR and then we briefly talk and both continue from master ...

@cdeil

cdeil Oct 27, 2015

Member

I already fixed this in #364 and changed quite a few other things (e.g. how test data is accessed).
Probably easiest if I merge that PR and then we briefly talk and both continue from master ...

This comment has been minimized.

@cdeil

cdeil Oct 27, 2015

Member

PS: I'll try to get #364 to a state where travis-ci passes and it can be merged before lunch ...

@cdeil

cdeil Oct 27, 2015

Member

PS: I'll try to get #364 to a state where travis-ci passes and it can be merged before lunch ...

This comment has been minimized.

@joleroi

joleroi Oct 27, 2015

Contributor

Sounds good to me

@joleroi

joleroi Oct 27, 2015

Contributor

Sounds good to me

Command line tools
------------------
* ``gammapy-data-manage`` -- Manage data locally and on servers

This comment has been minimized.

@joleroi

joleroi Oct 27, 2015

Contributor

this points to 'gammapy-data-manage = gammapy.obs.data_manager:main', which does not exist

@joleroi

joleroi Oct 27, 2015

Contributor

this points to 'gammapy-data-manage = gammapy.obs.data_manager:main', which does not exist

This comment has been minimized.

@cdeil

cdeil Oct 27, 2015

Member

This command line tool to manage data (download files, generate test files) doesn't exist yet.
I've uncommented that line and added a TODO comment in #364

@cdeil

cdeil Oct 27, 2015

Member

This command line tool to manage data (download files, generate test files) doesn't exist yet.
I've uncommented that line and added a TODO comment in #364

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment