Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Optimize filter removal #18
Filter removal has always been an issue with Koncorde, because of how I used TypedArrays to index subfilters and conditions. These arrays were necessary as Koncorde was written with Node.js 4, meaning that, at the time, Map/Set were either unavailable, or had subpar performances (I don't remember which).
So, the aim of that PR is to simplify A LOT how filters, subfilters, conditions and keyword storages are indexed.
The benchmark tool has also been updated: it now cleans filter up after each keyword test. This hasn't been done before because the removal of large quantities of filters posed problems to Koncorde: it takes 12 to 20 seconds just to remove 10k filters, which is far too long.
How should this be manually tested?
Use that PR' benchmark tool. You can also use that version to test previous branches of Koncorde, the difference is striking.
Add LGTM badges to the README file
Just for fun, I ran the benchmark with 1 000 000 filters per keywords:
@@ Coverage Diff @@ ## 1-dev #18 +/- ## ========================================== + Coverage 95.18% 96.62% +1.43% ========================================== Files 33 33 Lines 1081 978 -103 ========================================== - Hits 1029 945 -84 + Misses 52 33 -19