libidset: add idset_equal(3), improve efficiency of idset_count(3) #2060
As a sanity check I was running a test workload to exercise the ingest->job-manager<->scheduler logic, and while doing some profiling I noticed the workload was spending 10% of the total CPU cycles in
This PR adds a
This small change had a large improvement, so at the risk of premature optimization, I went ahead and proposed it here.
On current master, a set of 4K 1 core jobs submitted to a instance with 4K cores across 128 nodes shows
With the change to
While adding new tests, I noticed that there was already an idset comparator function within the unit test, and since this was needed for sched-simple, I moved that code into
Man page entry for
Hm: Looks like
Until then (or until wider availability of asciidoctor >= 1.5.7), though, splitting .adoc files is the only thing I can think of as well.
Ok (I do need to reword or update spelling dictionary).
I was going down the route of trying to make a
Problem: idset_count(3) needs to iterate the entire vEB tree to get a count of entries in the set on each call, and this is (relatively) slow. Considering use cases of idset_count(3) such as in a sort algorithm, an improvement here may be warranted. Add a count member to the idset structure and keep the embeeded count up to date within the idset_set/clear set of calls. This unfortunately adds some slight overhead to the set and clear operations, since each integer argument to set/clear has to first be checked for membership (or lack thereof) in the idset so that the count can be appropriately updated.
Now that there is a newer implementation of idset_count(3), add some extra checks to handle possible corner cases where the internal idset count accounting could go wrong.
Now that there exists an `idset_equal()` implementation in libidset, use this function instead of a custom comparator in the shed-simple rlist.c code.
@@ Coverage Diff @@ ## master #2060 +/- ## ========================================== + Coverage 80.42% 80.44% +0.02% ========================================== Files 191 191 Lines 30273 30286 +13 ========================================== + Hits 24346 24363 +17 + Misses 5927 5923 -4