Skip to content

Super sorter implemented#57

Merged
lpgaff merged 3 commits intoMiniball:mainfrom
lpgaff:new_time_ordering
Jun 20, 2025
Merged

Super sorter implemented#57
lpgaff merged 3 commits intoMiniball:mainfrom
lpgaff:new_time_ordering

Conversation

@lpgaff
Copy link
Contributor

@lpgaff lpgaff commented Jun 20, 2025

I have updated the time-ordering so that it doesn't fill a ROOT tree in the first instance, but a vector of the MiniballDataPackets class. These data are then time-ordered using the std::sort algorithm, which is super fast because it is done completely in memory. I tested the memory pressure and it is smaller than the histogrammer. We basically open the 2 GB MIDAS file in memory, then have all of that again in the vector, so we have something like 4 GB maximum memory usage. It did end up using something like 9 GB in the end, but there is no leak at least.

To write the output tree then, we loop over the sorted vector and fill the tree. Seems to work, but code needs some cleaning up. People are free to take a look at it for now though, might speed things up online.

I did a test with a run file from the IS708 run last year, results below for a subrun with 6525 x 64kB blocks of data.

ROOT standard sorting:

mb_sort -s settings_20241110.dat -c calibrations_20250304.dat -r  -i R9_14 -f 
94.80s user 3.54s system 99% cpu 1:38.86 total

New super sorter:

mb_sort -s settings_20241110.dat -c calibrations_20250304.dat -r  -i R9_14 -f
36.81s user 2.53s system 97% cpu 40.183 total

That is a factor of 2.5 times faster for small files, no -ebis flag included.


A larger 2 GB file with 32000 x 64 kB blocks of data gives:

ROOT standard sorting:

mb_sort -s settings_20241110.dat -c calibrations_20250304.dat -r  -i R9_13 -f 
426.61s user 11.70s system 99% cpu 7:19.73 total

New super sorter:

mb_sort -s settings_20241110.dat -c calibrations_20250304.dat -r  -i R9_13 -f
147.08s user 7.00s system 98% cpu 2:35.97 total

which is a factor of 2.9 times faster. So consistently better :-)

@lpgaff lpgaff merged commit d1d39c0 into Miniball:main Jun 20, 2025
1 of 2 checks passed
@lpgaff lpgaff deleted the new_time_ordering branch June 20, 2025 01:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant