Coincidences

[153957]

This is meant to add the processing of coincidences to the publicdb.
Daily ESD data will be analysed for coincidences, which will be stored in the ESD.
Histograms will be generated to give an overview of the coincidences.

Currently it is meant to use all stations in the network when looking for coincidences.
We could add the processing of coincidences for cluster, though that should just be a subset of the full network..

The coincidence module in SAPPHiRE does not seems to keep a nice reference to the original data entry,
it simply copies all events that are part of a coincidence to a new table.
This increases disk usage, perhaps we can make it work with nicer pointers back to the original tables.

To be added are views which show the histograms for the daily network-wide coincidences.

[153957]

Points to be resolved:

• Link to the coincidences page from the stations pages
• Improve coincidences in SAPPHiRE to find alternative for duplication of events (to save a lot of disc space)
• Make sure that some edge cases are handled properly
  • Only one station with data
  • Only test stations with data
  • ...
• Add more information to coincidences page
  • Number of stations with data
  • Number of events
  • Coincidence window
• Add info to Help page
• Correctly fill in num_coincidences in NetworkSummary
• Remember to add new histogram types in database when merging to master

[153957]

Note that it was not the observables that were taking so much space but rather the _src_timestamps.

>>> data.root.coincidences._src_timestamps.size_on_disk
75987048L
>>> data.root.coincidences.coincidences.size_on_disk
1768851L
>>> data.root.coincidences.c_index.size_in_memory
300628L
>>> data.root.coincidences.observables.size_on_disk
5566650L

Still the observables where a large fraction.
With these changes:

• removal of _src*/observables tables
• addition of more columns in coincidences
• station_id also in c_index
• new s_index table

The final file size has decreased
ESD with no coincidences: 408 MB
With old coincidences: 494 MB
With new coincidences: 415 MB

>>> data.root.coincidences.coincidences.size_on_disk
4390644L
>>> data.root.coincidences.s_index.size_in_memory
2947L
>>> data.root.coincidences.c_index.size_in_memory
583992L

Both the coincidences and c_index tables have grown a lot, but in total space was saved.

[153957]

Fix migration conflict

[153957]

Use smaller coincidence window than default, because coincidental coincidences become dominant for larger windows. As can be seen in this plot, where the number of found coincidences for all events of a single day is plotted against the size of the coincidence window.

[153957]

Todo:

• Use station numbers in coincidence column names (s2, s3, s501, s502, etc)
• Add nice links to get to the correct page easily

[153957]

Adding support for clusters will be easier once HiSPARC/sapphire#19 is finished.

[153957]

Finishing touches:

• Add (sub)cluster coincidence histograms
• Add downloading of coicnidences (with events), as streaming JSON?

These points will be worked on later (new PR)