New tool: enrico_getdata?

[cdeil]

We should make a tool that gets the data for the ROI, energy range, event class the user wants from one of these locations:

• FSSC (automate http requests so that the user doesn't have to fill out the web form and download files by hand)
• Weekly data
• Preprocessed data

Making this a separate tool from enrico_like gives more flexibility for the user how to get started with the analysis, i.e. where to get the input photon and spacecraft files.

[cdeil]

Making Fermi data server queries from Python was implemented in astroquery.fermi by @keflavich .

The implementation and documentation is not finished (e.g. astropy/astroquery#181), but once it is we should integrate it into Enrico.

If astropy and astroquery installation works with the upcoming Fermi LAT ScienceTools software (@kialio and @davidsanchez could maybe check?) we can simply use astroquery as a dependency for Enrico ... if not I'll try to extract a copy of astroquery.fermi and put it into Enrico.

[keflavich]

Let me know how the install goes; the latest PR made astroquery compatible with astropy < 0.3.

[kialio]

The data server won't change when the new data release happens. Should be exactly the same but it will be serving P7Rep instead of P7.

One thing to note: We've had a few cases in the past year of users not thinking through the scripting process and submitting 100's (and one time 1000's) of queries in less than a minute. This pretty much looks like a dds attack and overwhelms our servers. We've implemented checks for this and block IPs that do it. I'm not sure if you can implement something in enrico that prohibits this kind of behavior but if you can, that would be good. I'll cross-post this to astroquery as well.

In the end, if a user wants to submit 1000's of queries, it would be more efficient for the user to grab the weekly files instead. These are updated as fast as the data in the data server. ie. the most recent weekly file might not be a full week, it just includes the most recent data.

[davidsanchez]

To get the data, I use enrico_download which download the weekly file and I
think this is quite useful.

2013/9/13 Jeremy Perkins notifications@github.com

The data server won't change when the new data release happens. Should be
exactly the same but it will be serving P7Rep instead of P7.

One thing to note: We've had a few cases in the past year of users not
thinking through the scripting process and submitting 100's (and one time
1000's) of queries in less than a minute. This pretty much looks like a dds
attack and overwhelms our servers. We've implemented checks for this and
block IPs that do it. I'm not sure if you can implement something in enrico
that prohibits this kind of behavior but if you can, that would be good.
I'll cross-post this to astroquery as well.

In the end, if a user wants to submit 1000's of queries, it would be more
efficient for the user to grab the weekly files instead. These are updated
as fast as the data in the data server. ie. the most recent weekly file
might not be a full week, it just includes the most recent data.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/9#issuecomment-24394036
.

David Sanchez