feat: Multivariate Anomaly Detection #343

ab93 · 2024-01-26T19:21:04Z

Additions:

Beta VAE to balance the KLD and Log likelihood loss
RobustMahalanobis Distance for multivariate thresholding
Percentile based thresholding with aggregation
Allow a tag suffix in docker workflow when using the github UI
Exponential moving average in postprocess output
Introduce pytests instead of unittests for newer test cases
Improve synthetic anomaly generator module
Per feature anomaly score when using certain thresholding models
Score config for different feature and window aggreation methods

Removal:

backtesting cli tool (since this will be added in a separate backtesting tool UI)

Demo branch --------- Signed-off-by: s0nicboOm <i.kushalbatra@gmail.com>

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

codecov · 2024-01-26T19:27:18Z

Codecov Report

Attention: 57 lines in your changes are missing coverage. Please review.

Comparison is base (2d84876) 93.40% compared to head (644875d) 92.31%.
Report is 1 commits behind head on main.

Files	Patch %	Lines
numalogic/backtest/_prom.py	62.12%	20 Missing and 5 partials ⚠️
numalogic/synthetic/anomalies.py	69.23%	7 Missing and 1 partial ⚠️
numalogic/transforms/_stateless.py	81.57%	3 Missing and 4 partials ⚠️
numalogic/models/threshold/_median.py	85.36%	3 Missing and 3 partials ⚠️
numalogic/config/factory.py	73.33%	4 Missing ⚠️
numalogic/tools/data.py	50.00%	3 Missing ⚠️
numalogic/udfs/postprocess.py	89.28%	1 Missing and 2 partials ⚠️
numalogic/config/_config.py	94.73%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #343      +/-   ##
==========================================
- Coverage   93.40%   92.31%   -1.10%     
==========================================
  Files          83       82       -1     
  Lines        3853     3918      +65     
  Branches      311      325      +14     
==========================================
+ Hits         3599     3617      +18     
- Misses        197      233      +36     
- Partials       57       68      +11

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

s0nicboOm

Some comments

numalogic/udfs/tools.py

numalogic/udfs/postprocess.py

numalogic/transforms/_stateless.py

numalogic/tools/trainer.py

numalogic/models/threshold/_median.py

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

s0nicboOm

minor comments

numalogic/tools/aggregators.py

numalogic/udfs/postprocess.py

numalogic/config/_config.py

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

qhuai

Share the current set of comments. Still working through files in https://github.com/numaproj/numalogic/blob/main/numalogic/models/vae

numalogic/models/threshold/_mahalanobis.py

numalogic/models/threshold/_median.py

tests/transforms/test_transforms.py

qhuai

Have you evaluated the quality difference between plain VAE v.s. conv1d VAE in anomaly detection?

numalogic/models/vae/variants/conv.py

numalogic/transforms/_stateless.py

s0nicboOm

Some more minor comments.

numalogic/models/threshold/_median.py

numalogic/udfs/factory.py

s0nicboOm

Also, I am not sure how do we test pytest files. can you add a step in README for it too.

Added some minor comments otherwise LGTM.

ab93 · 2024-02-07T22:34:06Z

Have you evaluated the quality difference between plain VAE v.s. conv1d VAE in anomaly detection?

Yes, conv1d has much better quantitative scores.

ab93 · 2024-02-07T22:34:55Z

Also, I am not sure how do we test pytest files. can you add a step in README for it too.

Added some minor comments otherwise LGTM.

Simple make test will work, already in README.

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

Additions: - Beta VAE to balance the KLD and Log likelihood loss - RobustMahalanobis Distance for multivariate thresholding - Percentile based thresholding with aggregation - Allow a tag suffix in docker workflow when using the github UI - Exponential moving average in postprocess output - Introduce pytests instead of unittests for newer test cases - Improve synthetic anomaly generator module - Per feature anomaly score when using certain thresholding models - Score config for different feature and window aggreation methods Removal: - backtesting cli tool (since this will be added in a separate backtesting tool UI) --------- Signed-off-by: Avik Basu <ab93@users.noreply.github.com> Co-authored-by: Kushal Batra <34571348+s0nicboOm@users.noreply.github.com> Signed-off-by: skondakindi <saisharathreddy_kondakindi@intuit.com>

s0nicboOm and others added 18 commits December 13, 2023 12:47

Demo branch (#335)

19ddc8e

Demo branch --------- Signed-off-by: s0nicboOm <i.kushalbatra@gmail.com>

feat: add transforms and robust thresholding

43c1ec8

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

tmp: try exp mov average

3ca8c15

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

add more logs

c0596d7

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

feat: multivariate backtesting

3035407

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

feat: support loading nl conf

7f9317c

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

feat: output threshold + final scores

46363d3

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

feat: difference transform

5ec5883

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

feat: add beta parameter for disentanglement

16c02f1

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

fix: vae nsamples

03faf8f

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

Merge branch 'main' into tmp/mv-demo-exp

c09df5d

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

Merge branch 'main' into tmp/mv-demo-exp

56bea46

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

fix: default ml pipe id in preproc

faf1e32

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

try: better docker workflow

dc35c80

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

fix: pipeline id key error

add642c

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

fix: pipeline id keyerr in tools.py

fbf171f

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

feat: add percentile thresholding

a075761

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

chore: remove backtest cli

fdbc40e

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

ab93 self-assigned this Jan 26, 2024

fix: import

777b5e2

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

ab93 force-pushed the betavae-1 branch from 3112259 to 777b5e2 Compare January 26, 2024 23:43

ab93 added 3 commits January 26, 2024 19:43

tests: median thresh tests

36db5df

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

chore: add transforms tests

79ceb12

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

chore: bump version to v0.7.0

371ebda

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

ab93 added the area/ml Machine Learning based changes label Jan 29, 2024

ab93 added 3 commits January 29, 2024 16:26

fix: factory class

634b2cf

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

chore: more tests for robust maha thresh

ae8b654

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

docs: update docstrings

8902e6f

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

ab93 marked this pull request as ready for review January 29, 2024 23:45

ab93 requested review from qhuai, nkoppisetty and s0nicboOm as code owners January 29, 2024 23:45

ab93 added 2 commits January 30, 2024 09:56

tmp: testing per feature score

cc681a3

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

fix: get item for final score

a913545

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

s0nicboOm reviewed Jan 30, 2024

View reviewed changes

ab93 added 5 commits January 30, 2024 19:54

feat: Score config in postprocess

bc6d08c

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

tmpfix: numalogic opex tags for query filter

c418b5a

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

fix: backtest

9de1e83

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

Merge branch 'main' into betavae-1

114d6e3

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

fix merge issues

780f2e2

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

ab93 changed the title ~~feat: Beta VAE~~ feat: Multivariate Anomaly Detection Feb 5, 2024

s0nicboOm reviewed Feb 6, 2024

View reviewed changes

fix: comments

b41a7f7

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

qhuai reviewed Feb 7, 2024

View reviewed changes

numalogic/models/threshold/_mahalanobis.py Show resolved Hide resolved

numalogic/models/threshold/_median.py Show resolved Hide resolved

numalogic/models/threshold/_median.py Show resolved Hide resolved

tests/transforms/test_transforms.py Show resolved Hide resolved

qhuai reviewed Feb 7, 2024

View reviewed changes

numalogic/models/vae/variants/conv.py Show resolved Hide resolved

numalogic/transforms/_stateless.py Show resolved Hide resolved

s0nicboOm reviewed Feb 7, 2024

View reviewed changes

numalogic/models/threshold/_median.py Outdated Show resolved Hide resolved

numalogic/models/threshold/_median.py Show resolved Hide resolved

numalogic/models/threshold/_median.py Show resolved Hide resolved

numalogic/udfs/factory.py Show resolved Hide resolved

s0nicboOm approved these changes Feb 7, 2024

View reviewed changes

ab93 added 2 commits February 7, 2024 17:39

docs: update docstrings

dd847d5

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

Merge branch 'main' into betavae-1

644875d

Signed-off-by: Avik Basu <ab93@users.noreply.github.com>

ab93 merged commit d8bda0e into main Feb 8, 2024
11 checks passed

ab93 deleted the betavae-1 branch February 8, 2024 18:34

ab93 temporarily deployed to production February 8, 2024 19:10 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Multivariate Anomaly Detection #343

feat: Multivariate Anomaly Detection #343

ab93 commented Jan 26, 2024 •

edited

codecov bot commented Jan 26, 2024 •

edited

s0nicboOm left a comment

s0nicboOm left a comment

qhuai left a comment

qhuai left a comment

s0nicboOm left a comment

s0nicboOm left a comment •

edited

ab93 commented Feb 7, 2024

ab93 commented Feb 7, 2024

feat: Multivariate Anomaly Detection #343

feat: Multivariate Anomaly Detection #343

Conversation

ab93 commented Jan 26, 2024 • edited

codecov bot commented Jan 26, 2024 • edited

Codecov Report

s0nicboOm left a comment

Choose a reason for hiding this comment

s0nicboOm left a comment

Choose a reason for hiding this comment

qhuai left a comment

Choose a reason for hiding this comment

qhuai left a comment

Choose a reason for hiding this comment

s0nicboOm left a comment

Choose a reason for hiding this comment

s0nicboOm left a comment • edited

Choose a reason for hiding this comment

ab93 commented Feb 7, 2024

ab93 commented Feb 7, 2024

ab93 commented Jan 26, 2024 •

edited

codecov bot commented Jan 26, 2024 •

edited

s0nicboOm left a comment •

edited