Som plugin #1269

LuisSanchez25 · 2023-09-24T23:35:10Z

Before you submit this PR: make sure to put all operations-related information in a wiki-note, a PR should be about code and is publicly accessible

What does the code in this PR do / what does it improve?

Creates a new peaklet classification plugin that uses SOM to refine our current peaklet classification algorithm.

Can you briefly describe how it works?

We first classify the data using out traditional peaklet classification algorithm, then all the data that gets classified as either an S1 or an S2 are passed to the SOM where the new data can be reclassified as type 0, 1 or 2, corresponding to the typical designation of junk, S1 and S2.

We also create a new context xenonnt_som which uses this pluggin instead of the usual peaklet classification plugin.

Can you give a minimal working example (or illustrate with a figure)?

Please include the following if applicable:

Update the docstring(s)
Update the documentation
Tests to check the (new) code is working as desired.
Does it solve one of the open issues on github?

Notes on testing

Until the automated tests pass, please mark the PR as a draft.
On the XENONnT fork we test with database access, on private forks there is no database access for security considerations.

All italic comments can be removed from this template.

fix error in data_to_log_deciles

combine with master branch

merge to version 2.1.13

github-actions

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

pep8

straxen/plugins/peaklets/__init__.py

straxen/plugins/peaklets/peaklet_classification_som.py

github-actions · 2023-09-24T23:36:21Z

straxen/plugins/peaklets/peaklet_classification_som.py

+        result['length'] = peaklets['length']
+        result['dt'] = peaklets['dt']
+        result['type'][mask_non_zero] = strax_type
+        #result['som_type'][mask_non_zero] = som_type + 1


[pep8] _{reported by reviewdog 🐶}
E265 block comment should start with '# '

github-actions · 2023-09-24T23:36:21Z

straxen/plugins/peaklets/peaklet_classification_som.py

+        result['dt'] = peaklets['dt']
+        result['type'][mask_non_zero] = strax_type
+        #result['som_type'][mask_non_zero] = som_type + 1
+        return dict(peaklet_classification_som=result, som_peaklet_data=som_info)


[pep8] _{reported by reviewdog 🐶}
C408 Unnecessary dict call - rewrite as a literal.

github-actions · 2023-09-24T23:36:21Z

straxen/plugins/peaklets/peaklet_classification_som.py

+        return dict(peaklet_classification_som=result, som_peaklet_data=som_info)
+
+
+def recall_populations(dataset, weight_cube, SOM_cls_img, norm_factors):


[pep8] _{reported by reviewdog 🐶}
N803 argument name 'SOM_cls_img' should be lowercase

please fix this

github-actions · 2023-09-24T23:36:21Z

straxen/plugins/peaklets/peaklet_classification_som.py

+    dataset:          Data to preform the recall on (Should be peaklet level data)
+    normfactos:       A set of 11 numbers to normalize the data so we can preform a recall
+    """
+    [SOM_xdim, SOM_ydim, SOM_zdim] = weight_cube.shape


[pep8] _{reported by reviewdog 🐶}
WPS359 Found an iterable unpacking to list

Suggested change

[SOM_xdim, SOM_ydim, SOM_zdim] = weight_cube.shape

xdim, ydim, zdim = weight_cube.shape

github-actions · 2023-09-24T23:36:21Z

straxen/plugins/peaklets/peaklet_classification_som.py

+    dataset:          Data to preform the recall on (Should be peaklet level data)
+    normfactos:       A set of 11 numbers to normalize the data so we can preform a recall
+    """
+    [SOM_xdim, SOM_ydim, SOM_zdim] = weight_cube.shape


[pep8] _{reported by reviewdog 🐶}
N806 variable 'SOM_xdim' in function should be lowercase

please remove all these CamelCase variable names. Use snake_case for variable names.

…SOM_plugin

straxen/plugins/peaklets/peaklet_classification_som.py

straxen/plugins/merged_s2s/merged_s2s.py

coveralls · 2023-10-04T13:16:08Z

coverage: 92.659% (-1.0%) from 93.643% when pulling d5cf03f on SOM_plugin into 8c70f8f on master.

Comply with some of the pep8 suggestions

jmosbacher

You are committing to collaborative project. Please conform to existing coding styles so that others can easily read your work.

straxen/contexts.py

straxen/plugins/peaklets/__init__.py

straxen/plugins/peaklets/peaklet_classification_som.py

jmosbacher · 2023-10-06T15:16:03Z

straxen/plugins/peaklets/peaklet_classification_som.py

+    som_files = straxen.URLConfig(default='resource:///stor2/data/LS_data/SOM_data/som_data_v4.1.npz?fmt=npy')
+
+    def infer_dtype(self):
+        dtype = dict()


Suggested change

dtype = dict()

dtype = {}

jmosbacher · 2023-10-06T15:16:51Z

straxen/plugins/peaklets/peaklet_classification_som.py

+        return dict(peaklet_classification_som=result, som_peaklet_data=som_info)
+
+
+def recall_populations(dataset, weight_cube, SOM_cls_img, norm_factors):


please fix this

jmosbacher · 2023-10-06T15:18:06Z

straxen/plugins/peaklets/peaklet_classification_som.py

+    dataset:          Data to preform the recall on (Should be peaklet level data)
+    normfactos:       A set of 11 numbers to normalize the data so we can preform a recall
+    """
+    [SOM_xdim, SOM_ydim, SOM_zdim] = weight_cube.shape


Suggested change

[SOM_xdim, SOM_ydim, SOM_zdim] = weight_cube.shape

xdim, ydim, zdim = weight_cube.shape

jmosbacher · 2023-10-06T15:19:49Z

straxen/plugins/peaklets/peaklet_classification_som.py

+    dataset:          Data to preform the recall on (Should be peaklet level data)
+    normfactos:       A set of 11 numbers to normalize the data so we can preform a recall
+    """
+    [SOM_xdim, SOM_ydim, SOM_zdim] = weight_cube.shape


please remove all these CamelCase variable names. Use snake_case for variable names.

straxen/plugins/peaklets/peaklet_classification_som.py

LuisSanchez25 · 2023-10-06T20:41:31Z

The problem with changing the structure with how you suggested (just importing straxen and using straxen.PeakletClassification for the plugin) is that it creates a circular import error since straxen cant initialize it. This is also part of the reason I removed it from the init.py file since it can just be initialized later by the context, the other reason is that in contexts if you look at line 88, it will register all plugins and since the SOM pluggin also provides peaklet classification I think it will cause problems. I could explicitly state not to fetch that config in the context where its registering everything but I think we probably do not want to hardcode that.

missed a function with upper cases

LuisSanchez25 · 2023-10-06T21:21:34Z

@jmosbacher all test have passed so I think this is ready to merge now

pep8 space

LuisSanchez25 · 2023-10-07T17:27:53Z

I cant merge the pull request so if you could merge it that would be great!

LuisSanchez25 and others added 14 commits July 31, 2023 14:52

som peaklet classification

99d9165

Updating plugin, this needs straxen PR Waveform attributes #745

9bf35fb

update plugin, this works now

280bdc4

remove som type and propagate som classification up to peaks

4949ef6

do not classify type 0

57bd953

Fix issue with array sisize

ed8cfd5

add som_type

303e0c7

Update peaklet_classification_som.py

b8c80c7

fix error in data_to_log_deciles

fix som_type issue

a19ff24

update model + some function details

78afb2e

new file + new type 3 added

202ec92

Merge branch 'master' into SOM_plugin

a8dec9b

combine with master branch

Merge branch 'master' into SOM_plugin

e650cbb

merge to version 2.1.13

separate som_plugin from main pipeline

2ad4902

github-actions bot reviewed Sep 24, 2023

View reviewed changes

LuisSanchez25 and others added 4 commits September 24, 2023 18:37

Merge branch 'master' into SOM_plugin

019a332

Add tests to produce input vectors

2802303

Merge branch 'SOM_plugin' of https://github.com/XENONnT/straxen into …

e59228a

…SOM_plugin

current status of plugin, detach plugin before we are ready fro review

ca7fccc

jmosbacher reviewed Sep 29, 2023

View reviewed changes

straxen/plugins/peaklets/peaklet_classification_som.py Outdated Show resolved Hide resolved

straxen/plugins/merged_s2s/merged_s2s.py Outdated Show resolved Hide resolved

LuisSanchez25 and others added 3 commits October 3, 2023 11:22

start making new context

1ff863c

fix plugin and context

4c0aaab

Merge branch 'master' into SOM_plugin

e563201

LuisSanchez25 and others added 3 commits October 4, 2023 16:29

clean up plugin

162dae4

Merge branch 'master' into SOM_plugin

4dbf283

Update peaklet_classification_som.py

2c8618f

Comply with some of the pep8 suggestions

LuisSanchez25 marked this pull request as ready for review October 4, 2023 22:18

LuisSanchez25 requested a review from jmosbacher October 5, 2023 15:40

jmosbacher requested changes Oct 6, 2023

View reviewed changes

LuisSanchez25 and others added 2 commits October 6, 2023 15:26

conform to pep8 style

2c67dd6

Merge branch 'master' into SOM_plugin

9607781

jmosbacher reviewed Oct 6, 2023

View reviewed changes

straxen/plugins/peaklets/peaklet_classification_som.py Outdated Show resolved Hide resolved

jmosbacher previously approved these changes Oct 6, 2023

View reviewed changes

Update peaklet_classification_som.py

66e392a

missed a function with upper cases

LuisSanchez25 dismissed jmosbacher’s stale review via 66e392a October 6, 2023 20:44

Update peaklet_classification_som.py

d5cf03f

pep8 space

jmosbacher approved these changes Oct 7, 2023

View reviewed changes

ahiguera-mx merged commit 02621e1 into master Oct 9, 2023
8 of 9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Som plugin #1269

Som plugin #1269

LuisSanchez25 commented Sep 24, 2023 •

edited

github-actions bot left a comment

github-actions bot Sep 24, 2023

github-actions bot Sep 24, 2023

github-actions bot Sep 24, 2023

jmosbacher Oct 6, 2023

github-actions bot Sep 24, 2023

jmosbacher Oct 6, 2023

github-actions bot Sep 24, 2023

jmosbacher Oct 6, 2023

coveralls commented Oct 4, 2023 •

edited

jmosbacher left a comment

jmosbacher Oct 6, 2023

jmosbacher Oct 6, 2023

jmosbacher Oct 6, 2023

jmosbacher Oct 6, 2023

LuisSanchez25 commented Oct 6, 2023

LuisSanchez25 commented Oct 6, 2023

LuisSanchez25 commented Oct 7, 2023

		return dict(peaklet_classification_som=result, som_peaklet_data=som_info)


		def recall_populations(dataset, weight_cube, SOM_cls_img, norm_factors):

	[SOM_xdim, SOM_ydim, SOM_zdim] = weight_cube.shape
	xdim, ydim, zdim = weight_cube.shape

Som plugin #1269

Som plugin #1269

Conversation

LuisSanchez25 commented Sep 24, 2023 • edited

What does the code in this PR do / what does it improve?

Can you briefly describe how it works?

Can you give a minimal working example (or illustrate with a figure)?

Notes on testing

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Sep 24, 2023

Choose a reason for hiding this comment

github-actions bot Sep 24, 2023

Choose a reason for hiding this comment

github-actions bot Sep 24, 2023

Choose a reason for hiding this comment

jmosbacher Oct 6, 2023

Choose a reason for hiding this comment

github-actions bot Sep 24, 2023

Choose a reason for hiding this comment

jmosbacher Oct 6, 2023

Choose a reason for hiding this comment

github-actions bot Sep 24, 2023

Choose a reason for hiding this comment

jmosbacher Oct 6, 2023

Choose a reason for hiding this comment

coveralls commented Oct 4, 2023 • edited

jmosbacher left a comment

Choose a reason for hiding this comment

jmosbacher Oct 6, 2023

Choose a reason for hiding this comment

jmosbacher Oct 6, 2023

Choose a reason for hiding this comment

jmosbacher Oct 6, 2023

Choose a reason for hiding this comment

jmosbacher Oct 6, 2023

Choose a reason for hiding this comment

LuisSanchez25 commented Oct 6, 2023

LuisSanchez25 commented Oct 6, 2023

LuisSanchez25 commented Oct 7, 2023

LuisSanchez25 commented Sep 24, 2023 •

edited

coveralls commented Oct 4, 2023 •

edited