Python script for t-test

[ypark234]

This adds a Python script that runs two-sample t-test with two data sets.
t-test is a statistical calculation that can be used to determine if there is a statistically significant difference between two sample groups.
In this script, unpaired equal variance two-sample t-test will be conducted for comparison of data that are assumed to be taken from an identical distribution.

Future work:

• Switch to Panda dataframe

[bam241]

Maybe add some kind of readme to explain how to use and ref to the stats

[bam241]

maybe separate snippet and unit_tests

in this PR or a separate PR, maybe setup CI-testing ?

[kkiesling]

Other thoughts to add:

2-D/3-D histogram of the p-values for like a mesh
write out the actual set of p-values (connecting them to the data sets/dicts that already exist)

Suggested structure changes:

1. expect users to provide two lists of data (not necessarily as a dictionary with keys) - perhaps two lists of tuples (data, standard error).
2. calculate t-test and return the data set with all the stats (np structured array? see line 212)
3. provide additional functions for writing that data set to a file and another function that can plot it in a histogram

Add a README

[gonuke]

Are we waiting on CI for this?

[ypark234]

I am adding README and making last few changes, while waiting for CI config to be added by a separate PR.

[gonuke]

Maybe I should clarify my earlier CI questions.... should I wait for CI to be implemented before merging this PR? Or is this something that can be merged prior to CI being implemented?

[ypark234]

CI will be implemented in a separate PR.
I expect this PR to be merged after CI is properly added.

[ypark234]

This PR is ready to be reviewed.
(CI test for the repo still needs to be triggered properly.)

[kkiesling]

This is a very well done and thorough "snippet"! I mostly just have questions.

The README has a lot of theory information in it and I am wondering if it is really necessary? I appreciate all the information, but I at the same time I think that the README for a script should really just include the information needed for the users to run the script. So that would basically be just information on how the input should be structured and how to run and then what the results should look like. I think it's plenty to just refer users to a resource to learn about the t-test if they don't know what it is. Though I also appreciate that you took all the time and effort to put the resource together. Maybe the "usage" and the background information parts of the README just need to be two separate documents?

On a different note, I think it's awesome that you have this structured so that one could just import the modules in a python script or actually run your script with command line arguments. Allows for a lot of different use which is nice.

[kkiesling]

Theory understanding question: If this test is being run on a tally, is the sample size n the number of histories run in the simulation or only the number of histories that contributed to that tally?

[ypark234]

In theory, it should be the number of histories that actually contributed to that tally,
which I assume is hard/impossible to know from MCNP output? (correct me if I'm wrong!)

[gonuke]

I think this should be the total number of histories. The fact that some contribute a score of 0 is part of the distribution.

[kkiesling]

@ypark234 do you want me to review this again or are we waiting on CI first?

[ypark234]

Sorry, let me do a final touch on this sooon and ask for possibly final review.

[ypark234]

It is finally ready for another review & merge.

[kkiesling]

This looks good to me

[kkiesling]

pending approval from @bam241 and this could probably be merged?

[gonuke]

Thanks @ypark234 - This is a great example of well-documented and complete code!

I think there are some design questions worth discussing, however, and might be great for a software meeting. Primarily, I think the twosample_ttest.py should have a minimal set of functions to perform a t-test on data with a well defined structure, or variations for different well-defined structures. The functions to load and format the data into those well-defined structures should live elsewhere like the script example you have provided.

For example, it seems like you have two well-defined structures right now: (1) a vector of data, (2) a grid of data. For each of these, I would anticipate:

• a way to perform an element-wise t-test
• a way to plot the t-value or p-value
• a histogram of t-values or p-values

In addition to a script that demonstrates the basic functionality, we could also have scripts (different PRs) that apply this test to some standard data we use in CNERG, esp. MCNP mesh tallies.

[gonuke]

Here is one little comment.

[gonuke]

Just one small suggestion- thanks @ypark234

[gonuke]

Thanks @ypark234 - these last changes look great!