New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add DataStore subset method #502

Merged
merged 5 commits into from Apr 11, 2016

Conversation

Projects
None yet
2 participants
@joleroi
Contributor

joleroi commented Apr 5, 2016

This PR introduces the functionality necessary to create a new DataStore from an existing one using a subset of observations.

@joleroi joleroi self-assigned this Apr 5, 2016

@joleroi joleroi added the feature label Apr 5, 2016

@cdeil cdeil added this to the 0.4 milestone Apr 5, 2016

Parameters
----------
obs_tabe : `~gammapy.data.ObservationTable`

This comment has been minimized.

@cdeil

cdeil Apr 5, 2016

Member

Typo: obs_tabe -> obs_table

----------
obs_tabe : `~gammapy.data.ObservationTable`
Table of observation to create the subset
dir : str, Path

This comment has been minimized.

@cdeil

cdeil Apr 5, 2016

Member

dir -> outdir (that's the name of the argument)

@cdeil

This comment has been minimized.

Member

cdeil commented Apr 5, 2016

You should filter the obs and HDU index table as well, and store those as FITS files in outdir, no?
So that the new data store is complete on it's own.

hdutable = self.hdu_table
hdutable.add_index('OBS_ID')
temp = hdutable.loc[obs_ids]
copydirs = set(temp['FILE_DIR'])

This comment has been minimized.

@cdeil

cdeil Apr 5, 2016

Member

For now I think this is OK.

For the future, it might be nice to re-introduce "schemes" how to store data and implement them.
E.g. one HDU per file, one obs per file, or all obs in one file with very many HDUs.

But this is just nice to have ... for now +1 to do it this way and move on ...

@cdeil

This comment has been minimized.

Member

cdeil commented Apr 5, 2016

(I'm offline now ... thank you and good night!)

def test_datastore_subset(tmpdir, data_manager):
"""Test creating a datastore as subset of another datastore"""
data_store = data_manager['hess-crab4-hd-hap-prod2']
selected_runs = data_store.obs_table.select_obs_id([23523, 23592])

This comment has been minimized.

@cdeil

cdeil Apr 5, 2016

Member

"runs" -> "obs" (let's try and be consistent in Gammapy / the specs)

@@ -304,6 +306,25 @@ def check_available_event_lists(self, logger=None):
return file_available
def create_new_store_from_obs_table(self, obs_table, outdir):

This comment has been minimized.

@cdeil

cdeil Apr 5, 2016

Member

Maybe just "copy_obs" to be a bit shorter? (no strong opinion ... pick which one you like best)

@cdeil

This comment has been minimized.

Member

cdeil commented Apr 5, 2016

(I didn't see the "WIP" in the title. I'll shut up now. Sorry!)

@joleroi

This comment has been minimized.

Contributor

joleroi commented Apr 6, 2016

Hi, sorry this was more complicated than I though.

  • Can you please have a look and check if the test actually tests for a valid datastore?
  • Can you also have a look at the implementation, since you've written all the DataStore stuff (especially the HDUindex table). There's probably room for improvement.

@joleroi joleroi changed the title from WIP: DataStore subset to DataStore subset Apr 6, 2016

@joleroi

This comment has been minimized.

Contributor

joleroi commented Apr 6, 2016

I left some comments where I found broken stuff. Can you judge if we still need all of this?

@joleroi joleroi merged commit 35d4795 into gammapy:master Apr 11, 2016

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@joleroi joleroi deleted the joleroi:subdatastore branch Apr 11, 2016

@cdeil

This comment has been minimized.

Member

cdeil commented Apr 11, 2016

Thanks!
(I'll look over the other PRs now and re-start travis-ci where it's useful.)

@cdeil cdeil changed the title from DataStore subset to Add DataStore subset method Apr 20, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment