Skip to content

SuperSuperSorter#59

Merged
lpgaff merged 9 commits intoMiniball:mainfrom
lpgaff:super_sorter
Jul 1, 2025
Merged

SuperSuperSorter#59
lpgaff merged 9 commits intoMiniball:mainfrom
lpgaff:super_sorter

Conversation

@lpgaff
Copy link
Contributor

@lpgaff lpgaff commented Jun 26, 2025

The latest sorting upgrade, now using smart pointers inside the vector so that we don't need to move the data around when we sort it, only the pointers to the memory. Tested with 50 sub runs and it is significantly faster and doesn't appear to be leaking memory.

For a single subrun as comparison:

Super^2-sorter = 14.24s user 1.79s system 93% cpu 17.082 total
Super-sorter   = 36.81s user 2.53s system 97% cpu 40.183 total
Root sorter    = 94.80s user 3.54s system 99% cpu 1:38.86 total

A factor of a 6.6 times faster overall.

This PR also includes Nigel's catch for corrupted input files in Issue #58.

lpgaff added 4 commits June 26, 2025 19:19
…into the vector and sorting them, we are using smart pointers so that the memory doesn't need to be moved around, just the pointers to them!
…of files to be sorted. This checks if we've got a file that wasn't closed properly and then forces it to be resorted if required.
@lpgaff
Copy link
Contributor Author

lpgaff commented Jun 26, 2025

Super^2-sorter = 14.24s user 1.79s system 93% cpu 17.082 total
Super-sorter   = 36.81s user 2.53s system 97% cpu 40.183 total
Root sorter    = 94.80s user 3.54s system 99% cpu 1:38.86 total

Of course, there was a bug in the time ordering and it wasn't doing it correctly... Real result is:

Super-sorter^2 = 36.67s user 3.90s system 97% cpu 41.436 total
Super-sorter   = 36.81s user 2.53s system 97% cpu 40.183 total
Root sorter    = 94.80s user 3.54s system 99% cpu 1:38.86 total

So not much different than before.

@lpgaff
Copy link
Contributor Author

lpgaff commented Jul 1, 2025

With the bugs fixed and using a mapping of data_id in the vector with timestamp, rather than calling it from the DataPackets class each time, we win another few seconds:

Super^2-sorter = 29.17s user 8.82s system 97% cpu 39.152 total
Super-sorter   = 36.81s user 2.53s system 97% cpu 40.183 total
Root sorter    = 94.80s user 3.54s system 99% cpu 1:38.86 total

@lpgaff lpgaff merged commit 819e532 into Miniball:main Jul 1, 2025
1 of 2 checks passed
@lpgaff lpgaff deleted the super_sorter branch July 1, 2025 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant