Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of tagged streams #1011

Merged
merged 1 commit into from Oct 15, 2022
Merged

Conversation

nukemberg
Copy link
Contributor

Create tags-coll set only once to avoid expensive set creation and allocation

@sanel
Copy link
Contributor

sanel commented Oct 15, 2022

After some testing, this small change was faster indeed :) Please notice that my tests will not use the code from PR, but simulate the difference when set is created on-the-fly (when event is received, old behavior) and when set is created before stream function is defined (new behavior from this PR).

(require '[criterium.core :as c])

(def tags ["foo" "bar" "baz"])  ; default behavior
(def tags-set (set tags))       ; generated set object before stream was created

(c/bench (tagged-any? tags {:tags ["foo"]}))
; Evaluation count : 95891520 in 60 samples of 1598192 calls.
;              Execution time mean : 628.626019 ns
;     Execution time std-deviation : 8.187710 ns
;    Execution time lower quantile : 622.051208 ns ( 2.5%)
;    Execution time upper quantile : 648.455723 ns (97.5%)
;                    Overhead used : 2.654335 ns
; 
; Found 3 outliers in 60 samples (5.0000 %)
; 	low-severe	 2 (3.3333 %)
; 	low-mild	 1 (1.6667 %)
;  Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

(c/bench (tagged-any? tags-set {:tags ["foo"]}))
; Evaluation count : 319278240 in 60 samples of 5321304 calls.
;              Execution time mean : 184.413915 ns
;     Execution time std-deviation : 1.959476 ns
;    Execution time lower quantile : 181.763322 ns ( 2.5%)
;    Execution time upper quantile : 187.707063 ns (97.5%)
;                    Overhead used : 2.654335 ns
; nil

(c/bench (tagged-all? tags {:tags ["foo"]}))
; Evaluation count : 102135000 in 60 samples of 1702250 calls.
;              Execution time mean : 585.095348 ns
;     Execution time std-deviation : 1.638328 ns
;    Execution time lower quantile : 584.169099 ns ( 2.5%)
;    Execution time upper quantile : 589.706420 ns (97.5%)
;                    Overhead used : 2.654335 ns
; 
; Found 3 outliers in 60 samples (5.0000 %)
; 	low-severe	 3 (5.0000 %)
;  Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

(c/bench (tagged-all? tags-set {:tags ["foo"]}))
; Evaluation count : 204666780 in 60 samples of 3411113 calls.
;              Execution time mean : 290.374668 ns
;     Execution time std-deviation : 0.656243 ns
;    Execution time lower quantile : 287.957636 ns ( 2.5%)
;    Execution time upper quantile : 291.137373 ns (97.5%)
;                    Overhead used : 2.654335 ns
; 
; Found 5 outliers in 60 samples (8.3333 %)
; 	low-severe	 4 (6.6667 %)
; 	low-mild	 1 (1.6667 %)
;  Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

It is >3x improvement for tagged-any? and >2x improvement for tagged-all? when they receive a set object :)

cc @jamtur01 @pyr

@pyr
Copy link
Contributor

pyr commented Oct 15, 2022

Yes, this one is a no-brainer, thanks @nukemberg!

@pyr pyr merged commit 762776f into riemann:main Oct 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants