New reconstruction and dependent variables #174

coderdj · 2017-12-07T23:38:41Z

This adds the s1_pattern_fit using 3d FDC and neural network posrec. The other version uses the 2D FDC with TPF. This also adds the s1 area fraction top probability variable computed at the 3D-NN position.

Diagnostic plots incoming. Putting this here as placeholder.

pdeperio · 2017-12-11T19:05:48Z

hax/treemakers/posrec.py

+        hax.minitrees.TreeMaker.__init__(self)
+
+        # We need to pull some stuff from the pax config
+        self.pax_config = load_configuration("XENON1T")


do we also need SR-dependent gains from configs in processing repo?

If I understand your question correctly then no. Gains already applied at this stage. This is just to pull some defaults. Unfortunately we wouldn't get override values in case the defaults are overridden in the run doc. I checked that they are not overridden for the values I took but I'm not sure of a better way to generalize.

pdeperio · 2017-12-11T19:11:06Z

hax/treemakers/posrec.py

+        weights_file = utils.data_file_name('tensorflow_nn_pos_weights_XENON1T_20171211.h5')
+        loaded_nn_model.load_weights(weights_file)
+        self.nn_tensorflow = loaded_nn_model
+        self.list_bad_pmts = [1, 2, 12, 26, 34, 62, 65, 79, 86, 88, 102, 118, 130, 134, 135, 139, 148, 150, 152, 162, 178,


can this be taken from the PMT gain list above?

Why needed? Bad pmts are blinded in pax (gain zero) so have no signal in the hit pattern.

@feigaodm says: defines the number of layers in NN.

Hardcoded for SR0 and SR1 now in #184. @weiyuehuan working on including this list with the NN models to replace this.

fixed in c994b08 and c1849fd

pdeperio · 2017-12-11T19:12:03Z

hax/treemakers/posrec.py

+        self.nn_tensorflow = loaded_nn_model
+        self.list_bad_pmts = [1, 2, 12, 26, 34, 62, 65, 79, 86, 88, 102, 118, 130, 134, 135, 139, 148, 150, 152, 162, 178,
+                         183, 190, 198, 206, 213, 214, 234, 239, 244]
+        self.ntop_pmts = 127 # How to get this automatically?


indeed must be automated for SR dependency (following above comments)

Same comment as above. You should just be able to get this from the XENON1T config because XENON1T will always have the same number of PMTs in the top array. Sure, some will be blinded, but they always have zero signal then.

Fixed in d61afe0

pdeperio · 2017-12-11T19:13:33Z

hax/treemakers/posrec.py

+        # Don't yell at me for hardcoding the filename into hax because it was
+        # also hardcoded in pax. Just kicking the can.
+        #aftmap_filename = utils.data_file_name('s1_aft_xyz_XENON1T_06Mar2017.json')
+        aftmap_filename = utils.data_file_name('s1_aft_xyz_XENON1T_20170808.json')


move to hax.ini like other maps in corrections.py?

Fixed in ef6fd7d

pdeperio · 2017-12-11T19:13:47Z

hax/treemakers/posrec.py

+        self.low_pe_threshold=10
+        #hax.minitrees.Treemaker.__init__(self)
+        # load trained NN models
+        nn_model_json = utils.data_file_name('tensorflow_nn_pos_XENON1T_20171211.json')


move to hax.ini like other maps in corrections.py

Fixed in ef6fd7d

pdeperio · 2017-12-11T19:14:42Z

hax/treemakers/posrec.py

+        for i, s2_t in enumerate(s2apc):
+            if i not in self.list_bad_pmts and i < self.ntop_pmts:
+                s2apc.clean.append(s2_t)
+        s2acp_clean.np.assaray(s2acp_clean)


acp -> apc?

Fixed in c8d44aa

pdeperio · 2017-12-14T15:07:53Z

hax/treemakers/posrec.py

@@ -131,7 +130,9 @@ def extract_data(self, event):
        aft = self.aft_map.get_value(self.x[i], self.y[i], self.z[i])
        event_data['s1_area_fraction_top_probability_hax'] = binom_test(
            size_top, size_tot, aft)
-
+        event_data['s1_area_fraction_top_probability_hax2'] = binom_test_pax(


@darrylmasson @coderdj what's the difference with this one? (better name than hax2?)

The binom_test from scipy is only valid for integer inputs of PE. The version I put into pax accepts floating points and is more accurate. I don't think there's a particularly good reason to include the scipy-based calculation (unless we want to compare with the original SR0 codes)

Yuck. OK I had them both for testing. The scipy one fails miserably below ~10pe. The plan is to revert to one before merging. The one will be the pax one I guess.

Fixed in #180

pdeperio · 2017-12-14T15:15:25Z

hax/treemakers/posrec.py

+                                           confused_s1_channels),
+                statistic=self.statistic)
+
+            event_data['s1_pattern_fit_hits_hax'] = self.pattern_fitter.compute_gof(


this (hits) is using the same MC map as for area?

In pax yes but we don't actually use hits so I don't know why I included it.

might be good to keep, since the optical maps are using photons detected i think, not PE

pdeperio · 2017-12-14T15:15:52Z

hax/treemakers/posrec.py

+                hpc[self.tpc_channels],
+                pmt_selection=pmts_bottom,
+                statistic=self.statistic)
+


add also variable for hits as above?

fixed in #180

pdeperio · 2017-12-14T17:25:45Z

@pelssers have you seen this new treemaker yet? wondering if the treemaker you were working on is in a more advanced state (like how you made corrections.py more general for all reconstruction algorithms).

* Fix S1 bottom pattern, docs, and lint * Fix oops * Increment version

pdeperio · 2017-12-14T18:04:29Z

hax/treemakers/posrec.py

+        s2apc = np.array(list(s2.area_per_channel))
+        if(len(s2apc)!=self.ntop_pmts):
+            return event_data
+        s2apc_clean = []


@feigaodm @weiyuehuan: as @coderdj mentioned above, is this needed if s2apc is already 0 from pax for bad PMTs? or does nn_tensorflow.predict actually need to remove these channels from the list?

See response above and #184

pdeperio · 2017-12-14T18:06:38Z

hax/treemakers/posrec.py

+
+        # Position reconstruction based on NN from TensorFlow
+        s2apc = np.array(list(s2.area_per_channel))
+        if(len(s2apc)!=self.ntop_pmts):


@feigaodm @weiyuehuan: in what case does this actually happen? i.e. is it ok to just ignore these events?

coderdj · 2017-12-15T15:06:05Z

I'm starting some cleanup of this but it seems the NN stuff doesn't work at all. Did it ever work?

First it won't even import on midway login nodes due to some glibc error, but we can live with this for now since you can run it on batch nodes fine. Then line https://github.com/XENON1T/hax/blob/newpatternlikelihood/hax/treemakers/posrec.py#L114 causes, by definition, a return before the neural net is ever invoked. After removing that clause I start getting weird tensorflow-related errors.

Can someone say if this is working for them in some setup? I don't mind helping debugging but we should be sure to test before committing.

pdeperio · 2017-12-15T15:11:04Z

Worked for me before. But @feigaodm and I were thinking that posrec stuff should be moved to another treemaker, agree?

coderdj · 2017-12-15T15:12:29Z

Oh I just got it working. We were clearing the keras session in the init, which wiped the network we just loaded. I'll commit the fix.
Your mass production all failed unfortunately.

coderdj · 2017-12-15T16:16:40Z

As for splitting into another minitree. May be useful for testing, but unpacking the peak objects and hit patterns is computationally slow so it might make sense to have everything that needs to do that in a single minitree.

add s1 area fraction near injection points near Rn220 source for anomalous leakage studies. Details can be seen in the following note. https://xe1t-wiki.lngs.infn.it/doku.php?id=xenon:xenon1t:analysis:sciencerun1:anomalous_background#can_we_remove_leakage_events_by_tightening_some_cuts

skazama · 2017-12-16T05:05:36Z

The list of excluded PMTs is different between SR0 and SR1.
For SR0, it is [1, 12, 26, 27, 34, 51, 62, 65, 73, 79, 86, 88, 91, 102, 118, 130, 134, 135, 137, 139, 148, 150, 152, 162, 167, 178, 183, 198, 203, 206, 213, 214, 234, 239, 244].

tunnell

This is nice functionality but it's almost certainly going to break down quickly as it introduces a lot of technical debt (that nobody will ever go back for). More generally, it seems like a rewrite of certain pax plugins. Is it not possible just to initialize those plugins then feed the event in? At very least, there are a few places that redefine things already in pax or messily define the keras models.

tunnell · 2017-12-17T10:45:12Z

hax/treemakers/posrec.py

+
+        # We need to pull some stuff from the pax config
+        self.pax_config = load_configuration("XENON1T")
+        self.tpc_channels = list(range(0, 247 + 1))


You can fetch this from pax config in previous line

Fixed in d61afe0

tunnell · 2017-12-17T10:47:13Z

hax/treemakers/posrec.py

+
+        # load trained NN models
+        nn_model_json = utils.data_file_name(hax.config["neural_network_model"])
+        json_file_nn = open(nn_model_json, 'r')


This is the bad way to load a model. Please do model.save('blah.hdf5'), which will save the model and weights together. There's no reason to keep these apart and it can just lead to problems.

Ah, there's also a keras.utils load_model command to reload it.

tunnell · 2017-12-17T10:48:00Z

hax/treemakers/posrec.py

+        loaded_nn_model = model_from_json(loaded_model_json)
+        weights_file = utils.data_file_name(hax.config['neural_network_weights'])
+        loaded_nn_model.load_weights(weights_file)
+        loaded_nn_model.compile(loss='mean_squared_error', optimizer='adam')


If you load the model as mentioned before, you don't need to recompile. You shouldn't need to recompile anyways since you're not retraining it, right?

tunnell · 2017-12-17T10:48:57Z

hax/treemakers/posrec.py

+                              130, 134, 135, 139, 148, 150, 152, 162, 178, 183,
+                              190, 198, 206, 213, 214, 234, 239, 244, 27, 73,
+                              91, 137, 167, 203]
+        self.ntop_pmts = 127  # How to get this automatically?


https://github.com/XENON1T/pax/blob/master/pax/config/XENON1T.ini#L450
len(pax_config['channels_top'])

Fixed in d61afe0

tunnell · 2017-12-17T10:49:26Z

hax/treemakers/posrec.py

+        self.ntop_pmts = 127  # How to get this automatically?
+
+    def get_data(self, dataset, event_list=None):
+


Docstring? What is this used for?

tunnell · 2017-12-17T10:50:22Z

hax/treemakers/posrec.py

+            "y_observed_nn_tf": None,
+            "s1_area_upper_injection_fraction": None,
+            "s1_area_lower_injection_fraction": None,
+        }


I sort of feel that each category of variable should be its own subfunction. This is super hard to follow.

tunnell · 2017-12-17T10:51:29Z

hax/treemakers/posrec.py

+        confused_s1_channels = []
+        for a, c in enumerate(s1.n_saturated_per_channel):
+            if c > 0:
+                confused_s1_channels.append(a)


Can be one line https://github.com/XENON1T/pax/blob/master/pax/datastructure.py#L249

tunnell · 2017-12-17T10:51:54Z

hax/treemakers/posrec.py

+                statistic=self.statistic)
+
+            pmts_bottom = np.setdiff1d(self.tpc_channels, confused_s1_channels)
+            pmts_bottom[0:127] = 0


Hardcode, though you define before 127.

Comment on how to get this list of PMTs above

Fixed in d61afe0

tunnell · 2017-12-17T10:52:26Z

hax/treemakers/posrec.py

+        event_data['s1_area_fraction_top_probability_hax'] = binom_test_pax(
+            size_top, size_tot, aft)
+
+        # Now do s1_pattern_fit


tunnell · 2017-12-17T11:01:44Z

hax/treemakers/posrec.py

+        s2apc_clean_norm = s2apc_clean_norm.reshape(1, len(s2apc_clean_norm))
+        predicted_xy_tensorflow = self.nn_tensorflow.predict(s2apc_clean_norm)
+        event_data['x_observed_nn_tf'] = predicted_xy_tensorflow[0, 0] / 10.
+        event_data['y_observed_nn_tf'] = predicted_xy_tensorflow[0, 1] / 10.


Can this preprocessing be broken into its own function? If I am to add another model, then it's copy-paste 10 more lines... though the preprocessing is slightly different.

Also, clear is the wrong word. It's 'preprocessed' or 'normalized'.

@tunnell It looks like you have some nice ideas on how to clean up the codes here. Can you modify this PR and test it on a dataset. Maybe you can also fix the issue that sometimes TF fails to be running.

* Fix bugs in event indexing and PMT selection for S1 likelihood calculations * Increment version * Fix version oops 0.2 is actually less than 0.16

* Added new corrections handler class * Made it work * Made posrec treemaker work with new class too * Revert version bump * Oops. Moved instead of git moved. * Implement in CorrectedDoubleS1Scatter and lint fixes

See details here: XENON1T/pax#655

Forgot again...

Name should be 'event_data' instead of 'result'...

adding additional s1 width variables for AC events rejection

pdeperio · 2017-12-21T16:07:25Z

hax/treemakers/posrec.py

@@ -38,11 +38,15 @@ class PositionReconstruction(TreeMaker):

       - s1_area_upper_injection_fraction: s1 area fraction near Rn220 injection points (near PMT 131)
       - s1_area_lower_injection_fraction: s1 area fraction near Rn220 injection points (near PMT 243)
+
+       - s1_range_90p_area: The width of the s1 (ns), duration of region that contains 90% of the area of the peak


@skazama Should we put this in Extended treemaker instead (where s1_range_80p_area already is)?

Fixed in #190

pdeperio · 2017-12-21T22:23:06Z

hax/hax.ini

@@ -135,6 +135,12 @@ corrections_definitions = {
        {"run_min": 10223, "run_max": 12089, "correction": "FDC_SR1_data_driven_time_dependent_3d_correction_part3_v1.json.gz"},
        {"run_min": 12090, "correction": "FDC_SR1_data_driven_time_dependent_3d_correction_part4_v1.json.gz"}
    ],
+    "fdc_3d_tfnn": [


@jingqiangye Can you add SR0 as well? (Otherwise, I think this will crash on SR0 files.)

@/all When pushing code/analyses from now, for completeness, we should ensure SR0+SR1 are both included.

Fixed in f60ce10

* Store additional AFT var using area instead of hits and AFT probability from map * Fix variable name * Store binomial probability instead of aft

Moved to Extended minitree in #190

Get a production going over holiday. I've done my best to beautify while maintaining desired functionality. Can improve on next iteration.

pdeperio · 2018-01-04T16:48:45Z

hax/hax.ini

+    "fdc_3d_tfnn": [
+        {"run_min": 6386, "run_max": 8648, "correction": "FDC_SR1_data_driven_time_dependent_3d_correction_tf_nn_part1_v1.json.gz"},
+        {"run_min": 8649, "run_max": 10976, "correction": "FDC_SR1_data_driven_time_dependent_3d_correction_tf_nn_part2_v1.json.gz"},
+        {"run_min": 10977, "run_max": 13195, "correction": "FDC_SR1_data_driven_time_dependent_3d_correction_tf_nn_part3_v1.json.gz"},


@jingqiangye Why does this have different run binning than FANN 3D FDC?

Because TFNN was using more Kr data for FDC. FANN 3D FDC was using Kr data up to Oct 2017, and TFNN is to Nov 2017. FANN 3D FDC will be updated with latest Kr(Jan. 2, 2018)today.

Daniel Coderre and others added 4 commits December 7, 2017 11:24

Added posrec treemaker

1bf3560

Updates to posrec treemaker

738b5d4

Bug fixes

30cc317

placeholder for tensorflow based pos_reconstruction

eab3d17

pdeperio suggested changes Dec 11, 2017

View reviewed changes

feigaodm and others added 3 commits December 11, 2017 21:20

minor style issue

c8d44aa

working version

6b4b6e4

lint fixes

8a2eea5

pdeperio changed the title ~~Newpatternlikelihood~~ New reconstruction and dependent variables Dec 12, 2017

pdeperio and others added 4 commits December 12, 2017 13:12

Update posrec.py (#177)

a76ff08

Properly declare size of array

13a014d

ignor bad pattern with hits

913f037

Added s1patternfit for bottom. Also split files into ini options

ef6fd7d

pdeperio reviewed Dec 14, 2017

View reviewed changes

Fix S1 bottom pattern, docs, and lint (#180)

576a8fb

* Fix S1 bottom pattern, docs, and lint * Fix oops * Increment version

pdeperio suggested changes Dec 14, 2017

View reviewed changes

Fixed some bugs and made it run

9e1b4a1

skazama added 3 commits December 16, 2017 13:48

Update posrec.py

397b72a

Update posrec.py

1f0de2d

Update posrec.py

e240fe7

tunnell requested changes Dec 17, 2017

View reviewed changes

tunnell previously requested changes Dec 17, 2017

View reviewed changes

pdeperio and others added 12 commits December 19, 2017 10:00

Fix two bugs in new S1 AFT and pattern likelihood calculations (#183)

5bf6e1f

* Fix bugs in event indexing and PMT selection for S1 likelihood calculations * Increment version * Fix version oops 0.2 is actually less than 0.16

Add SR dependent bad PMT list (#184)

eaea8a1

Pull corrections logic out of minitree maker (#185)

52c612b

* Added new corrections handler class * Made it work * Made posrec treemaker work with new class too * Revert version bump * Oops. Moved instead of git moved. * Implement in CorrectedDoubleS1Scatter and lint fixes

Add 3D FDC for TFNN in SR1

fdee4a5

See details here: XENON1T/pax#655

Add 3D FDC for TFNN in SR1

3f1fc7f

See details here: XENON1T/pax#655

Update version number

1255c5b

Forgot again...

fix bug

41d9d3f

Name should be 'event_data' instead of 'result'...

fix bugs

8078ff5

fix bugs 2

7cf6d54

Update posrec.py

c3a8670

Update posrec.py

4a6b223

adding additional s1 width variables for AC events rejection

Update posrec.py

7178200

pdeperio suggested changes Dec 21, 2017

View reviewed changes

pdeperio mentioned this pull request Dec 21, 2017

Corrections refactor #187

Merged

pdeperio suggested changes Dec 21, 2017

View reviewed changes

jingqiangye and others added 8 commits December 21, 2017 15:48

Add 3D FDC for TFNN in SR0

f60ce10

More S1 AFT variables (#188)

ba77d13

* Store additional AFT var using area instead of hits and AFT probability from map * Fix variable name * Store binomial probability instead of aft

Merge branch 'master' into newpatternlikelihood

cefe296

Remove S1 width variables

8b89b09

Moved to Extended minitree in #190

Merge branch 'master' into newpatternlikelihood

bba3d3a

sort the tensorflow files name

9246573

Update posrec.py

c994b08

Make bad PMT list read-in work

c1849fd

pdeperio approved these changes Dec 25, 2017

View reviewed changes

pdeperio merged commit 284a300 into master Dec 25, 2017

pdeperio deleted the newpatternlikelihood branch December 25, 2017 18:16

pdeperio mentioned this pull request Dec 25, 2017

Bugfix for S1_AFT calculation XENON1T/pax#657

Merged

pdeperio reviewed Jan 4, 2018

View reviewed changes

feigaodm mentioned this pull request Jan 4, 2018

update branch for AC analysis, triggered by Shingo's new cuts #192

Merged

		self.ntop_pmts = 127 # How to get this automatically?

		def get_data(self, dataset, event_list=None):

New reconstruction and dependent variables #174

New reconstruction and dependent variables #174

Conversation

coderdj commented Dec 7, 2017

pdeperio Dec 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdeperio Dec 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdeperio Dec 14, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdeperio commented Dec 14, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coderdj commented Dec 15, 2017

pdeperio commented Dec 15, 2017

coderdj commented Dec 15, 2017

coderdj commented Dec 15, 2017

skazama commented Dec 16, 2017

tunnell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdeperio Dec 11, 2017 •

edited

Loading

pdeperio Dec 17, 2017 •

edited

Loading

pdeperio Dec 14, 2017 •

edited

Loading

pdeperio commented Dec 14, 2017 •

edited

Loading