Choke point at FeatureDictWeighter.countFeature() with large numbers of threads #12
Description
When querying a trained model, ProPPR uses a couple of bloom filters to track the trained and untrained features it encounters. This helps prevent mistakes where the wrong parameter file is used, or where the training set isn't representative of the test set.
Problem is, the bloom filter impl we're using blocks on key comparisons. It's a fast check, so we don't see much slowdown with --threads up to 32, but beyond that the thread blocking gets prohibitive.
Not sure what to do about this yet other than give you an option to disable it (which would obvs make it into people's default settings and defeat the whole purpose).