zhashx: hash limit increasing much faster than zhash

[chu11]

I've been investigating a performance issue after some code was converted to use the zhashx library instead of zhash (flux-framework/flux-core#3591).

After a lot of digging, it appears the zhashx implementation hash limit is growing much faster/larger than when we used zhash. It appears it has to do with the "chain limit" check that leads to a increased zhashx limit.

In zhashx, I see two locations where a re-hash will be done. One in zhashx_insert():

    size_t limit = primes [self->prime_index];
    if (self->size >= limit * LOAD_FACTOR / 100) {
        //  Create new hash table
        uint new_prime_index = self->prime_index + GROWTH_FACTOR;
        assert (s_zhashx_rehash (self, new_prime_index) == 0);
        self->chain_limit += CHAIN_GROWS;
    }

and the other in s_item_lookup()

    if (len > self->chain_limit) {
        //  Create new hash table
        uint new_prime_index = self->prime_index + GROWTH_FACTOR;
        assert (s_zhashx_rehash (self, new_prime_index) == 0);
        limit = primes [self->prime_index];
        self->cached_index = self->hasher (key) % limit;
    }

note that the path in s_item_lookup() does not increase the chain limit after a rehash.

The default self->chain_limit is 1, so it only takes 1 unlucky hash collision to increase the limit of the hash regardless of how full it is. And b/c we don't increase the chain limit in the latter rehash, the chain limit stays at 1. So a single unlucky hash collision will always keep on increasing the limit of the hash.

As an experiment I did the following two different fixes, each of which resolved the problem:

-#define INITIAL_CHAIN    1    //  Initial chaining limit
+#define INITIAL_CHAIN    5    //  Initial chaining limit

and

@@ -342,6 +342,7 @@ s_item_lookup (zhashx_t *self, const void *key)
             return NULL;
         limit = primes [self->prime_index];
         self->cached_index = self->hasher (key) % limit;
+        self->chain_limit += CHAIN_GROWS;
     }
     return item;
 }

I'm going to propose a PR based on the latter fix, but wanted to post this issue to see if other solutions are preferred or possible.