Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite MultiFiles to allow reading all subruns #160

Merged
merged 7 commits into from
Dec 3, 2022
Merged

Conversation

maxnoe
Copy link
Member

@maxnoe maxnoe commented Nov 20, 2022

  • Missing test files for reading multiple subruns after one-another
  • Add tests for reading multiple subruns

@maxnoe maxnoe marked this pull request as draft November 20, 2022 10:06
@codecov
Copy link

codecov bot commented Nov 20, 2022

Codecov Report

Base: 87.74% // Head: 88.11% // Increases project coverage by +0.37% 🎉

Coverage data is based on head (f78388d) compared to base (de8015f).
Patch coverage: 94.23% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #160      +/-   ##
==========================================
+ Coverage   87.74%   88.11%   +0.37%     
==========================================
  Files          16       16              
  Lines        1314     1397      +83     
==========================================
+ Hits         1153     1231      +78     
- Misses        161      166       +5     
Impacted Files Coverage Δ
ctapipe_io_lst/tests/test_lsteventsource.py 99.19% <ø> (-0.05%) ⬇️
ctapipe_io_lst/multifiles.py 91.80% <91.66%> (+0.27%) ⬆️
ctapipe_io_lst/tests/test_multifile.py 100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@maxnoe
Copy link
Member Author

maxnoe commented Nov 20, 2022

@FrancaCassol This seems to work nicely:

$ python read_subruns.py                                                                                                                                                        
INFO:ctapipe_io_lst.LSTEventSource:INPUT PATH = /fefs/aswg/data/real/R0/20221030/LST-1.1.Run10003.0000.fits.fz                                                                  
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.1.Run10003.0000.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.2.Run10003.0000.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.3.Run10003.0000.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.4.Run10003.0000.fits.fz                                                            
INFO:ctapipe_io_lst.LSTEventSource:Changed `use_flatfield_heuristic` to False                                                                                                   
True                                                                                                                                                                            
CRITICAL:ctapipe_io_lst.event_time.EventTimeCalculator:Using event 1 as time reference for dragon. timestamp: 1667168918767793176 from ucts counter: 3767792900                 
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.1.Run10003.0001.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.2.Run10003.0001.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.3.Run10003.0001.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.4.Run10003.0001.fits.fz                                                            
WARNING:ctapipe_io_lst.LSTEventSource:Event 90946 has unknown event type, trigger: 00100001                                                                                     
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.1.Run10003.0002.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.2.Run10003.0002.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.3.Run10003.0002.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.4.Run10003.0002.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.1.Run10003.0003.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.2.Run10003.0003.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.3.Run10003.0003.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.4.Run10003.0003.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.1.Run10003.0004.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.2.Run10003.0004.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.3.Run10003.0004.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.4.Run10003.0004.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.1.Run10003.0005.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.2.Run10003.0005.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.3.Run10003.0005.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.4.Run10003.0005.fits.fz                                                            
WARNING:ctapipe_io_lst.LSTEventSource:Event 295516 has unknown event type, trigger: 00100001                                                                                    
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.1.Run10003.0006.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.2.Run10003.0006.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.3.Run10003.0006.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.4.Run10003.0006.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.1.Run10003.0007.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.2.Run10003.0007.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.3.Run10003.0007.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.4.Run10003.0007.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.1.Run10003.0008.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.2.Run10003.0008.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.3.Run10003.0008.fits.fz                                                            
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.4.Run10003.0008.fits.fz
WARNING:ctapipe_io_lst.LSTEventSource:Event 468290 has unknown event type, trigger: 00100001
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.1.Run10003.0009.fits.fz
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.2.Run10003.0009.fits.fz
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.3.Run10003.0009.fits.fz
INFO:ctapipe_io_lst.multifiles.MultiFiles:Opened file /fefs/aswg/data/real/R0/20221030/LST-1.4.Run10003.0009.fits.fz

with:

from ctapipe.io import EventSource
from traitlets.config import Config
import logging

logging.basicConfig(level=logging.INFO)


c = Config()
c.LSTEventSource.MultiFiles.all_streams = True
c.LSTEventSource.MultiFiles.all_subruns = True
c.LSTEventSource.apply_drs4_corrections = False
c.LSTEventSource.pointing_information = False
c.EventSource.input_url = "/fefs/aswg/data/real/R0/20221030/LST-1.1.Run10003.0000.fits.fz"


with EventSource(config=c) as source:
    print(source.multi_file.all_subruns)
    for e in source:
        print(f"Event {e.index.event_id:8d}", end='\r')
    print()

@maxnoe maxnoe marked this pull request as ready for review December 3, 2022 11:36
@maxnoe maxnoe merged commit 0169f3a into master Dec 3, 2022
@maxnoe maxnoe deleted the all_subruns branch December 3, 2022 12:10
@maxnoe maxnoe mentioned this pull request Feb 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants