Stricter blinding cut #206

pdeperio · 2018-01-30T19:46:21Z

Implement a more stricter blinding cut (as requested here):

Move definition from hax.ini into separate unblinding.ini file (the new name should be more scary to edit).
Move cut string from hax.config to hax.unblinding_config such that it cannot be modified with hax.init(kwargs) as before.
a. Thus, Warn on blinding cut customization #204 becomes obsolete since it's not allowed at all here. @JelleAalbers what other legitimate uses were you thinking?
Rename blinding_cut to unblinding_selection to break backwards compatibility of any rogue scripts (though should already be taken care of by 2. above)
Automatically unblind MC (previously, manually using kwargs in secret MC notebooks and laxer).
a. Moved mc_data variable from minitrees.py to hax global variable to reduce code duplication.
b. Reduced code duplication for MC checks and e-lifetime.

Henceforth, the only way to unblind is to submit a PR modifying unblinding.ini (or still a few other previous malicious ways, but please don't).

Do not allow blinding cut to be modified by init(kwargs) Rename "blinding_cut" to "unblinding_selection" Automatically unblind MC Make MC flag global (not just in minitrees) Reduce code redundancy in e-lifetime a bit lint fixes

…ind_mc

(though not sure what to do with other eval above)

coderdj

What about support for different SR0/SR1 cuts? Or is this not needed?

Otherwise I like it in general. I hesitate to say this but I really think there's no way to 'accidentally unblind' now. Though I thought that before...

pdeperio · 2018-01-31T15:22:11Z

Added the SR0 plot here. Quite similar to SR1, so I don't think using the same lines for SR0 will bias us.

coderdj · 2018-01-31T15:31:27Z

OK. Might want a second or third review also since important.

JelleAalbers · 2018-01-31T15:43:13Z

hax/__init__.py

+unblinding_config = {}
+
+# Flag for whether you are working with MC (True) or not
+mc_data = False


Could you clarify what this global mc_data is doing? It looks like this is a run-level property, but it's stored as a package global. You might get some interesting behaviour if you're doing real data / MC comparison studies. Could the treemakers that need to know whether they are in MC land (or maybe event the treemaker base class) call hax.runs.is_mc instead?

Remnant from confusion during development. Fixed in 1154d04.

JelleAalbers · 2018-01-31T15:57:01Z

hax/__init__.py

+    unblinding_config = {}
+    section_to_use = 'UNBLINDING'
+    for key, value in unblinding_configp[section_to_use].items():
+        unblinding_config[key] = ast.literal_eval(value)


Since you're only storing two settings in unblinding.ini, and the unblinding.ini filename can't be overriden by the user, would it be easier to make unblinding.ini just unblinding.py? Then add unblinding here https://github.com/XENON1T/hax/blob/master/hax/__init__.py#L7, and you can access the settings as hax.unblinding.unblinding_selection and hax.unblinding.blind_from_run when you need them in hax.runs.

it will soon have many entries after #199 here. but i guess your suggestion applies there too, and would clean up the init function, right?

Yes, it should work just as well. I haven't looked at #199 in detail, but maybe you want to put all settings in a settings dictionary inside unblinding.py if you have to do a lot of dynamic key access (getattr(hax.unblinding, key) isn't so great). You could even move is_blind into the file as well to have all blinding logic together.

Great suggestion! Implemented in 1cc8c65 and 309112f

JelleAalbers · 2018-01-31T16:04:29Z

hax/corrections_handler.py

+
+            except Exception as e:
+                print(e)
+                return 1.


If there is no electron lifetime available, are you sure you want to produce a correction minitree?

The generic except catching with a default of 1 instead of NaN might cause some fun later on. If someone puts a syntax error in get_electron_lifetime, everything will run happily but we'll get no electron lifetime corrections. Hopefully someone is checking the logs and they'll see there was a syntaxerror somewhere (but since it's just a print they also don't know where).

I think @coderdj implemented this at first; @coderdj what was your motivation for this?

I copy-pasted code from @jpienaar13 who had added this condition for MC I think (or at least that's what I thought when I copy-pasted). But we could remove it as far as I'm concerned.

OK raising the exception instead in e3525fd

@coderdj Pretty sure I didnt add the e-lifetime=1 option. in fact i believe git blame says you did.

Yeah now that I look at it I did. I don't know why I did.

But the other part is true. When I did the refactor of the corrections stuff I also noticed it was weird and copy-pasted it thinking you added it for MC stuff (without checking if that was the case).

Anyway, it can be removed.

JelleAalbers · 2018-01-31T16:07:23Z

hax/runs.py

@@ -307,6 +307,16 @@ def get_run_start(run_id):
        # return np.datetime64('2017-06-13T18:17:43.000000000')


+def is_mc(run_id):


Glad you cleaned up this code duplication!

JelleAalbers

Looks good, I can understand having the blinding_cut option so easily overridable in hax.init may be too tempting. Factoring the blinding settings out to a separate file also makes it easier to track the history of the blinding cut.

I made a few suggestions for avoiding the extra config parsing and the new package globals. I think they're easy and might save time in the long run, but see for yourself.

also forgot logger in unblinding.py

JelleAalbers · 2018-01-31T17:54:08Z

hax/__init__.py

@@ -1,11 +1,12 @@
 import logging
 import os
+import ast


This is now unused, though I suppose you could replace the eval of the main config with this (currently it has a "slightly" less secure eval with the os module exposed (which is also no longer used))

Just removed in 6ad82f6 (since I couldn't easily just swap eval with ast.literal_eval).

JelleAalbers · 2018-01-31T17:55:09Z

hax/corrections_handler.py

+            elifetime = self.get_misc_correction("mc_electron_lifetime_liquid", run_number)
+
+        else:
+            try:


You can remove the whole try-except now, it just raises every exception it encounters (which happens anyway)

Fixed in 01169a1

XENON1T/hax#206

(#206)

XENON1T/hax#206

* Sideband unblinding and update blinding doc * More readable blinding, and also unblind S2<200 * Remove #199 comment for High-E below NR (already unblinded before this PR) * Adapt to stricter blinding implementation (#206) * Unblind below S2 = 200 pe (threshold) Following #208 * Remove redundant s2 < 200 * Remove unnecessary s2>200 at high energy unblindin * Remove obsolete comment

pdeperio and others added 9 commits January 30, 2018 09:34

Automatically unblind MC

e302338

Fix oops

db40e39

Move blinding cut into separate .ini file and var

77ffacb

Do not allow blinding cut to be modified by init(kwargs) Rename "blinding_cut" to "unblinding_selection" Automatically unblind MC Make MC flag global (not just in minitrees) Reduce code redundancy in e-lifetime a bit lint fixes

Automatically unblind MC

436f979

Fix oops

56605d4

Move blinding cut into separate .ini file and var

8725860

Do not allow blinding cut to be modified by init(kwargs) Rename "blinding_cut" to "unblinding_selection" Automatically unblind MC Make MC flag global (not just in minitrees) Reduce code redundancy in e-lifetime a bit lint fixes

Remove unnecessary edits

0952629

Merge branch 'unblind_mc' of https://github.com/XENON1T/hax into unbl…

b21125c

…ind_mc

Switch to global mc_data var in posrec.py

34cd166

pdeperio requested review from tunnell, JelleAalbers, coderdj, feigaodm and jpienaar13 January 30, 2018 19:46

pdeperio and others added 6 commits January 30, 2018 14:57

Ignore eval insecurity

1fb4122

Try again ignoring eval()

a8ebc8f

Fine, try to use ast.literal_eval

5237de1

(though not sure what to do with other eval above)

Set MC flag to false if given dataset is not

b33c67a

Reduce more code duplication

2f013cd

More code duplication reduction

7e4d3d4

coderdj reviewed Jan 31, 2018

View reviewed changes

coderdj approved these changes Jan 31, 2018

View reviewed changes

JelleAalbers reviewed Jan 31, 2018

View reviewed changes

Revert mc_data flag from global to Treemaker

1154d04

JelleAalbers reviewed Jan 31, 2018

View reviewed changes

jpienaar13 approved these changes Jan 31, 2018

View reviewed changes

pdeperio added 3 commits January 31, 2018 11:21

Create unblinding.py instead of bulky .ini

1cc8c65

Move is_blind() to unblinding.py

309112f

Raise if elifetime fails instead of pass

e3525fd

also forgot logger in unblinding.py

JelleAalbers reviewed Jan 31, 2018

View reviewed changes

pdeperio added 2 commits January 31, 2018 13:58

Just let elifetime crash if it fails

01169a1

Remove unused ast module

6ad82f6

feigaodm merged commit a1922b0 into master Jan 31, 2018

feigaodm deleted the unblind_mc branch January 31, 2018 20:42

pdeperio added a commit to XENON1T/lax that referenced this pull request Jan 31, 2018

Remove manual unblinding, now automatic from

1eb8c01

XENON1T/hax#206

pdeperio mentioned this pull request Jan 31, 2018

Remove manual unblinding in laxer XENON1T/lax#141

Merged

pdeperio added a commit that referenced this pull request Jan 31, 2018

Adapt to stricter blinding implementation

789a873

(#206)

pdeperio added a commit to XENON1T/lax that referenced this pull request Feb 1, 2018

Remove manual unblinding, now automatic from (#141)

d69131a

XENON1T/hax#206

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stricter blinding cut #206

Stricter blinding cut #206

pdeperio commented Jan 30, 2018 •

edited

Loading

coderdj left a comment

pdeperio commented Jan 31, 2018 •

edited

Loading

coderdj commented Jan 31, 2018

JelleAalbers Jan 31, 2018

pdeperio Jan 31, 2018

JelleAalbers Jan 31, 2018

pdeperio Jan 31, 2018

JelleAalbers Jan 31, 2018

pdeperio Jan 31, 2018

JelleAalbers Jan 31, 2018

pdeperio Jan 31, 2018

coderdj Jan 31, 2018 •

edited by pdeperio

Loading

pdeperio Jan 31, 2018

jpienaar13 Jan 31, 2018

coderdj Feb 1, 2018

JelleAalbers Jan 31, 2018

JelleAalbers left a comment

JelleAalbers Jan 31, 2018

pdeperio Jan 31, 2018

JelleAalbers Jan 31, 2018

pdeperio Jan 31, 2018

		@@ -307,6 +307,16 @@ def get_run_start(run_id):
		# return np.datetime64('2017-06-13T18:17:43.000000000')


		def is_mc(run_id):

Stricter blinding cut #206

Stricter blinding cut #206

Conversation

pdeperio commented Jan 30, 2018 • edited Loading

coderdj left a comment

Choose a reason for hiding this comment

pdeperio commented Jan 31, 2018 • edited Loading

coderdj commented Jan 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coderdj Jan 31, 2018 • edited by pdeperio Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JelleAalbers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdeperio commented Jan 30, 2018 •

edited

Loading

pdeperio commented Jan 31, 2018 •

edited

Loading

coderdj Jan 31, 2018 •

edited by pdeperio

Loading