-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GTI required or optional with EVENTS? #20
Comments
I had a look at the ctools code and in fact, GTI is not required at all. If no GTI extension is found, ctools uses TSTART and TSTOP keywords to build the GTI internally. Therefore, we could change the GTI extension to be optional (in case it is identical with TSTART and TSTOP). |
OK, ... I'll make the change in the spec later today. |
I removed the EVENTS / GTI disclaimer for ctools in 9535bd7 |
While working on supporting GTIs in a separate file in ctools, I have discussed with Jürgen about this. Especially regarding having GTIs optional and in a separate file, he is very sceptical. I looked again over http://gamma-astro-data-formats.readthedocs.org/en/latest/data_storage/index.html, here some comments (not sure where to post them otherwise, you may just echo them where appropriate):
|
Just to clarify: The idea is to make a GTI extension required in the same file as the events and to put the GTI extension name as a DSS keyword. |
Is having multiple EVENTS and GTI in one FITS file a use case we want to support? Yes, pointing from EVENTS to GTI is an option. In OGIP they do that a lot, they also point to the IRFs from the EVENTS header. Is there a description / webpage / document / spec / whatever of "DSS"? |
Yes this is still supported. Of course these EVENTS and GTIs have to have different extension names.
I agree but I would argue that GTIs are special and very specific to the event list (unlike IRFs, which can be similar for different observations). GTIs are more like cut parameters and always have to be linked to the corresponding event list.
https://confluence.slac.stanford.edu/display/ST/Data+SubSpace+keywords
Ctools already uses DSS selection quite extensively, e.g. when applying ctselect. I can add this to the spec. |
At the moment for HESS we have one run = one event list = one start / stop time = GTI superfluous.
That's a major addition. Probably a good one. |
Well ctools can handle missing GTIs. But as far as I understand, this is not the way to go. Eventually all observations need GTIs. Maybe for CTA we will have only one observation per night with multiple GTIs - no one really knows. GTIs are generic enough to cover all the use cases we can currently think of.
Not sure how Nathan does this exactly but I guess he has a python script that does all the work in splitting and assigning IRFs using certain criteria.
Ok |
Just a comment: it is looking very likely that with CTA we will drop the concept of "runs", meaning that in a particular observation there may be events that are not appropriate for the attached IRF (e.g. events reconstructed during slewing that would need to be treated differently). For that reason, GTIs, even if trivially describing the run start and stop time, are very important. The first usable event may actually come thousands of events into the file, for example. Therefore, even if it seems trivial, the software should be implemented to use them and not rely on the run start and stop headers (those can even be dropped, or considered only "human readable", while the tools should access the GTI to get the time bounds in a machine-readable format). That also means that at least in a future release, there should be an extra loop somewhere over GTIs within a file, rather than assuming the events are contiguous. That even allows us to immediately do things like light-curve binning, PSR phases, and "cloud" removal, which are all nice features to have! |
@kosack - Thanks for your comment! I wasn't aware that it might be common to have science tools apply GTIs.
I'm 50:50 on this. Keeping the option to just have EVENTS with single start / stop time in the header and no GTI could still be allowed / the fallback if no GTI is present. But I'm also OK with making GTI required and to make it very clear that science tools should read GTI, not EVENTS header keywords for this. I still have several questions:
|
I guess for now lifetime is computed using GTIs (regardless where they come from). Eventually there will be more detailed information about the deadtime but currently it is assumed to be constant.
As Jürgen suggested, we could use the DSS keywords (like it is done in Fermi): DSTYP1 = 'TIME ' DSUNI1 = 's ' DSVAL1 = 'TABLE ' DSREF1 = ':GTI ' The DSREF keyword gives the extension name of the GTI (required to be in the same file).
For now I would keep it that way. If the user wants to split up runs into chunks and use different IRFs it is possible but more effort which is fine for now.
I don't have much time this week but I will try. |
No, at the moment I'm using This is a bit related to the idea to have extra TECH3 files ... info like deadtime fraction and good time intervals could be there. Is it a good idea to keep deadtime fraction and livetime header keywords in the EVENTS HDU?
Can you please split that out into a separate issue (or even better yet: pull request)?
Why treat the GTI differently from the IRFs? Having all HDUs in the HDU index table is convenient for data distribution / synchronisation. If you want the links from EVENTS HDU to the GTI, I'd suggest to also add links from EVENTS to the IRFs, and to keep the HDU index as-is. Although: flexibility in which files HDUs are located / complexity is introduced when introducing a linking scheme. |
As long as there is no "spacecraft" or whatever file these keywords should stay in the FITS header.
Yes
Because they are specific to the event lists (IRFs such as background can apply to different observations). From the event file it should be clear which GTIs correspond to the event lists. Event list shouldn't exist without a GTI definition.
Coherent handling of GTIs and linking them to the event list. We can of course list the GTIs in the HDU index but I would strongly prefer to also link the GTIs from the EVENT header.
No, GTIs are different. Following the Fermi-LAT scheme GTIs are unique to the event list. IRFs, as said above) do not have to be linked. The user can e.g. choose to use different background models etc. |
Let's do that.
That's not clear yet. There could be different GTIs (depending on data quality or other criteria) for a given EVENTS HDU as well. I agree that one can make arguments that GTI and BACKGROUND is special and should be treated differently than other IRFs. But it's not clear yet / there's no agreement yet how it should work / we're not currently doing this in the HDU index spec. So bottom line ... pull requests welcome. |
No, if the user just updates GTIs according to some quality parameters.
It doesn't matter to me, we can list the GTIs in the HDU spec for completeness. But I think it is important that GTIs are linked from the events header too.
Sounds reasonable. |
Here's a non-official diagram I made recently to show how GTIs might relate to observation blocks(which are just generalized runs that may have more events than the science tools would normally use in them). It's just a brainstorm right now, but illustrates some complex use cases. One might want for example to analyze data during slewing to make a slew survey, or select data when one set of telescopes in an array have stabilized and are tracking, while the others are still moving. Those special cases require different IRFs than the standard ones, so would need different GTI definitions. Those would be realized via a tool e.g. |
@kosack - Thanks! At the DL3 file / science tools level, analysing data taken while the telescopes slew (or the array changes in any other way) could be handled by defining (probably small) periods of time during which the IRFs are sufficiently constant. This could be done by having small "observations" each with one start / stop and be handled by the existing format / science tools. Alternatively large extensions are needed so that one observation has multiple GTIs, each with their per-GTI IRFs, or even introducing TECH3 files that describe how the array changes, and have a larger IRF lookup database to compute IRFs for a given point in time on the fly. |
@cdeil Yes that is the idea - you wouldn't have to have a single GTI file that has the "Standard" ,"slew", etc gtis. You just have the GTI that the user selected. So they would do:
etc.. So the science tools don't need to understand the difference between GTI types, that's up to the user to associate. They need a |
I guess that does lead to one possible advanced case: the user may want to generate several GTI extensions and not always overwrite one. I don't know if that is a use case in other telescopes, but it could be useful to have the name of the GTI extension flexible ("GTI" by default, but the user could select another if they want "MYGTI", etc). I guess that's the same as with the IRFs, where you don't want to hard-code the extension name. |
As part of a HESS presentation today I mentioned this DL3 effort.
@mimayer @kosack @jknodlseder @cboisson @joleroi @GernotMaier - Thoughts? Should we start and change things now? |
Hi Christoph, On Mon, Feb 22, 2016 at 2:56 PM, Christoph Deil notifications@github.com
Larger periods of time would just need the merging of the IRF3 phase space,
|
For simplicity, you can simply make the assumption that there may be multiple GTIs in a run, but they all share the same IRF (e.g. some data may be cut out, but there is by defauly still only one GTI/run). The advanced use of using another GTI subset with different IRF doesn't need to be handled by the software automatically, since the user can do it. Therefore you have:
|
I'm about to make the v0.2 version of the spec, so I'm moving this to v1.0. I think CTA will now for the next year or two drive the DL3 spec and eventually define something. This discussion might be a useful record for previous discussions, but we're not going to resolve this question of time intervals on Github. |
Should GTI (Good time interval) extensions be required or optional?
Are they required to be in the same file as the EVENTS extension?
@mimayer - What's the status in ctools? Are there any concrete plans to change something?
http://gamma-astro-data-formats.readthedocs.org/en/latest/events/index.html#disclaimer-about-names-of-hdu-and-file-structure
Does someone have time to look at the EVENTS and GTI specs from OGIP?
Do existing FTOOLs handling EVENTS work if no GTI is present (for operations where it's not needed)?
This is probably something where we should just follow the existing practices, no?
The text was updated successfully, but these errors were encountered: