-
Notifications
You must be signed in to change notification settings - Fork 33
/
chapter1.ipynb
252 lines (252 loc) · 8.05 KB
/
chapter1.ipynb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![SciUnit Logo](https://raw.githubusercontent.com/scidash/assets/master/logos/SciUnit/sci-unit-tag.png)\n",
"\n",
"\n",
"<a href=\"https://colab.research.google.com/github/scidash/sciunit/blob/master/docs/chapter1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
"\n",
"# Chapter 1. What is SciUnit?\n",
"Everyone hopes that their model has some correspondence with reality. Usually, checking whether this is true is done informally.\n",
"### SciUnit makes this formal and transparent.\n",
"SciUnit is a framework for validating scientific models by creating experimental-data-driven unit tests. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### If you are using this file in Google Colab, this block of code can help you install sciunit from PyPI in Colab environment."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-04T17:17:26.797848Z",
"iopub.status.busy": "2021-05-04T17:17:26.796560Z",
"iopub.status.idle": "2021-05-04T17:17:29.165793Z",
"shell.execute_reply": "2021-05-04T17:17:29.167482Z"
}
},
"outputs": [],
"source": [
"!pip install -q sciunit"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After installation, let's begin with importing sciunit."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2021-05-04T17:17:29.175051Z",
"iopub.status.busy": "2021-05-04T17:17:29.174070Z",
"iopub.status.idle": "2021-05-04T17:17:30.011470Z",
"shell.execute_reply": "2021-05-04T17:17:30.009992Z"
}
},
"outputs": [],
"source": [
"import sciunit"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### What does testing look like? \n",
"We'll start with a simple example from the history of cosmology, where we know that better models displaced their predecessors.<br>\n",
"Suppose we have a test suite called \"Saturn Suite\" that aims to test cosmological models for their correspondence to empirical data about the planet Saturn.<br>\n",
"*Everything in this example is hypothetical, but once you understand the basic ideas you can visit the [documentation for NeuronUnit](https://github.com/scidash/neuronunit/blob/master/docs/chapter1.ipynb) to see some working, interactive examples from a different domain (neuron and ion channel models).* "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"from saturnsuite.tests import position_test,velocity_test,eccentricity_test # Examples of test classes. \n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There's nothing specific to Saturn about position, velocity, or eccentricity. They could apply to any cosmological body.\n",
"### SciUnit test classes used with in one scientific domain (like cosmology) are located in a discipline-specific library. \n",
"In this case, the test classes (hypothetically) come from a SciUnit library called CosmoUnit, and are instantiated with data specific to Saturn, in order to create tests specific to a model's predictions about Saturn:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"'''\n",
"saturnsuite/tests.py # Tests for the Saturn suite. \n",
"'''\n",
"from . import saturn_data # Hypothetical library containing Saturn data. \n",
"from cosmounit import PositionTest, VelocityTest, EccentricityyTest # Cosmounit is an external library. \n",
"position_test = PositionTest(observation=saturn_data.position)\n",
"velocity_test = VelocityTest(observation=saturn_data.velocity)\n",
"eccentricity_test = EccentricityTest(observation=saturn_data.eccentricity)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This means the test *classes* are data-agnostic, but the test *instances* encode the data we want a model to recapitulate."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, let's load some models that aim to predict the cosmological features being assessed by the tests above."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"from saturnsuite.models import ptolemy_model, copernicus_model, kepler_model, newton_model # Examples of models. \n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Ptolemy's, Copernicus's, Kepler's, or Newton's models could similarly apply to any cosmological body.\n",
"So these model classes are found in CosmoUnit, and the Saturn Suite contains model instances parameterized to emit predictions about Saturn specifically."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"'''\n",
"saturnsuite/models.py # Models for the Saturn suite. \n",
"'''\n",
"from cosmounit import PtolemyModel, CopernicusModel, KeplerModel, NewtonModel \n",
"ptolemy_model = PtolemyModel(planet='Saturn')\n",
"copernicus_model = CopernicusModel(planet='Saturn')\n",
"kepler_model = KeplerModel(planet='Saturn')\n",
"newton_model = NewtonModel(planet='Saturn')\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the above each model takes a keyword argument 'planet' that determines about what planet the model will make predicitons."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### All of our tests can be organized into a suite to compare results across related tests. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"'''\n",
"saturnsuite/suites.py # Tests for the Saturn suite. \n",
"'''\n",
"import sciunit\n",
"from .tests import position_test, velocity_test, eccentricity_test\n",
"saturn_motion_suite = sciunit.TestSuite([position_test, velocity_test, eccentricity_test])\n",
"suites = (saturn_motion_suite,)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Now we can execute this entire test suite against our models. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```python\n",
"from saturn_suite.suites import saturn_motion_suite\n",
"saturn_motion_suite.judge([ptolemy_model, copernicus_model, kepler_model, newton_model])\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The exact output will depend on your preferences (terminal, HTML, etc.) but the figure below illustrates both the results you get (center table) and the relationship between the components listed here. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![Cosmo Example](https://raw.githubusercontent.com/scidash/assets/master/figures/cosmo_example.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The figure above also refers to SciDash, an in-development portal for accessing public test results, but for the remainder of this tutorial, we will focus on model/test development, execution, and visualization on your own machine. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### In the next section we'll see how to create models and tests from scratch in SciUnit. \n",
"### Onto [Chapter 2](chapter2.ipynb)!"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.4"
}
},
"nbformat": 4,
"nbformat_minor": 4
}