This repository has been archived by the owner on Jun 17, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 0
/
README.md
246 lines (208 loc) · 10.8 KB
/
README.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
[](https://pyscaffold.org/)
[](https://mp-time-split.readthedocs.io/en/stable/)
[](https://pypi.org/project/mp-time-split/)
[](https://anaconda.org/conda-forge/mp_time_split)
[](https://coveralls.io/github/sparks-baird/mp-time-split?branch=main)
<!-- These are examples of badges you might also want to add to your README. Update the URLs accordingly.
[](https://cirrus-ci.com/github/<USER>/mp-time-split)
[](https://pepy.tech/project/mp-time-split)
[](https://twitter.com/mp-time-split)
-->
# mp-time-split
> Use Materials Project time-splits for generative modeling benchmarking.
While methods for cross-validating accuracy of materials informatics models is well
estabilished (see for example [Matbench](https://matbench.materialsproject.org/)),
evaluating the performance of generative models such as
[FTCP](https://github.com/PV-Lab/FTCP) or
[imatgen](https://github.com/kaist-amsg/imatgen), and [many
others](https://github.com/stars/sgbaird/lists/materials-generative-models) is less
straightforward. Recently, [Xie et al.](http://arxiv.org/abs/2110.06197) introduced new
benchmark datasets and metrics in [CDVAE](https://github.com/txie-93/cdvae) for several
state-of-the-art algorithms. This repository acts as a supplement to CDVAE benchmarks,
delivering [a new benchmark dataset](https://figshare.com/articles/dataset/Materials_Project_Time_Split_Data/19991516) (`Materials_Project_Time_Split_52` or **MPTS-52**) with time-based (5 $\times$ train/val)
+train/test splits suitable for cross-validated hyperparameter optimization and
subsequent benchmarking via the test split. **MPTS-52** is most comparable to **MP-20**
from [Xie et al.](http://arxiv.org/abs/2110.06197), with the difference that up to 52
atoms are allowed and possibly a difference in the unique elements, as no elemental
filtering was applied (e.g. removal of radioactive elements).
## Quick Start
### Installation
```bash
conda env create -n mp-time-split -c conda-forge mp-time-split
conda activate mp-time-split
```
### Example
```python
from mp_time_split.core import MPTimeSplit
mpt = MPTimeSplit(target="energy_above_hull")
mpt.load(dummy=False)
for fold in mpt.folds:
train_inputs, val_inputs, train_outputs, val_outputs = mpt.get_train_and_val_data(
fold
)
final_train_inputs, test_inputs, final_train_outputs, test_outputs = mpt.get_test_data()
```
### Output
```python
print(train_inputs.iloc[0], train_outputs)
```
<table>
<tr>
<th> `train_inputs.iloc[0]` </th>
<th> `train_outputs` </th>
</tr>
<tr>
<td>
```python
Structure Summary
Lattice
abc : 2.591619125942699 2.591619125942699 2.591619125942699
angles : 109.47122063449069 109.47122063449069 109.47122063449069
volume : 13.399593956465264
A : -1.496272 1.496272 1.496272
B : 1.496272 -1.496272 1.496272
C : 1.496272 1.496272 -1.496272
PeriodicSite: V (0.0000, 0.0000, 0.0000) [0.0000, 0.0000, 0.0000]
```
</td>
<td>
```python
146 0.000000
925 0.190105
1282 0.087952
1335 0.022710
12778 0.003738
2540 0.000000
316 0.000000
```
</td>
</tr>
</table>
```{note}
Data (e.g. `train_inputs`, `train_outputs`) is sorted by earliest publication year.
The Materials Project ID number is used as the pandas `Series` index. The "mp-" and
"mvc" suffixes are dropped, except that "-" is retained in case of "mvc-" to make it
negative in the index and distinguish between "mp-" and "mvc-" type entries. This is
to make it easy to sort the entries based on the index. See https://matsci.org/t/how-do-i-do-a-time-split-of-materials-project-entries-e-g-pre-2018-vs-post-2018/42584/2?u=sgbaird
for more information. More detailed information (e.g. BibTeX references) can be accessed
via `mpt.data`.
```
For additional examples, see the [notebooks](notebooks) directory.
## Installation
### Anaconda
Create an environment named (`-n`) `mp-time-split` with the Anaconda package `mp-time-split` intalled from the `conda-forge` channel (`-c`).
```bash
conda env create -n mp-time-split -c conda-forge mp-time-split
```
### PyPI
Optionally create and activate a conda environment (recommended to use an isolated environment of some kind):
```bash
conda env create -n mp-time-split python==3.9.* # 3.7.* or 3.8.* also OK
conda activate mp-time-split
```
Install the `mp-time-split` package from PyPI.
```bash
pip install mp-time-split
```
### `environment.yml`
In order to set up the necessary environment:
1. review and uncomment what you need in `environment.yml` and create an environment `mp-time-split` with the help of [conda]:
```
conda env create -f environment.yml
```
2. activate the new environment with:
```
conda activate mp-time-split
```
### Local Installation
> **_NOTE:_** The conda environment will have mp-time-split installed in editable mode.
> Some changes, e.g. in `setup.cfg`, might require you to run `pip install -e .` again.
Optional and needed only once after `git clone`:
3. install several [pre-commit] git hooks with:
```bash
pre-commit install
# You might also want to run `pre-commit autoupdate`
```
and checkout the configuration under `.pre-commit-config.yaml`.
The `-n, --no-verify` flag of `git commit` can be used to deactivate pre-commit hooks temporarily.
4. install [nbstripout] git hooks to remove the output cells of committed notebooks with:
```bash
nbstripout --install --attributes notebooks/.gitattributes
```
This is useful to avoid large diffs due to plots in your notebooks.
A simple `nbstripout --uninstall` will revert these changes.
Then take a look into the `scripts` and `notebooks` folders.
<!-- ## Dependency Management & Reproducibility
1. Always keep your abstract (unpinned) dependencies updated in `environment.yml` and eventually
in `setup.cfg` if you want to ship and install your package via `pip` later on.
2. Create concrete dependencies as `environment.lock.yml` for the exact reproduction of your
environment with:
```bash
conda env export -n mp-time-split -f environment.lock.yml
```
For multi-OS development, consider using `--no-builds` during the export.
3. Update your current environment with respect to a new `environment.lock.yml` using:
```bash
conda env update -f environment.lock.yml --prune
``` -->
## Project Organization
```
├── AUTHORS.md <- List of developers and maintainers.
├── CHANGELOG.md <- Changelog to keep track of new features and fixes.
├── CONTRIBUTING.md <- Guidelines for contributing to this project.
├── Dockerfile <- Build a docker container with `docker build .`.
├── LICENSE.txt <- License as chosen on the command-line.
├── README.md <- The top-level README for developers.
├── configs <- Directory for configurations of model & application.
├── data
│ ├── external <- Data from third party sources.
│ ├── interim <- Intermediate data that has been transformed.
│ ├── processed <- The final, canonical data sets for modeling.
│ └── raw <- The original, immutable data dump.
├── docs <- Directory for Sphinx documentation in rst or md.
├── environment.yml <- The conda environment file for reproducibility.
├── models <- Trained and serialized models, model predictions,
│ or model summaries.
├── notebooks <- Jupyter notebooks. Naming convention is a number (for
│ ordering), the creator's initials and a description,
│ e.g. `1.0-fw-initial-data-exploration`.
├── pyproject.toml <- Build configuration. Don't change! Use `pip install -e .`
│ to install for development or to build `tox -e build`.
├── references <- Data dictionaries, manuals, and all other materials.
├── reports <- Generated analysis as HTML, PDF, LaTeX, etc.
│ └── figures <- Generated plots and figures for reports.
├── scripts <- Analysis and production scripts which import the
│ actual PYTHON_PKG, e.g. train_model.
├── setup.cfg <- Declarative configuration of your project.
├── setup.py <- [DEPRECATED] Use `python setup.py develop` to install for
│ development or `python setup.py bdist_wheel` to build.
├── src
│ └── mp_time_split <- Actual Python package where the main functionality goes.
├── tests <- Unit tests which can be run with `pytest`.
├── .coveragerc <- Configuration for coverage reports of unit tests.
├── .isort.cfg <- Configuration for git hook that sorts imports.
└── .pre-commit-config.yaml <- Configuration of pre-commit git hooks.
```
<!-- pyscaffold-notes -->
## Note
This project has been set up using [PyScaffold] 4.2.2 and the [dsproject extension] 0.7.post1.dev8+g43a905e.
[conda]: https://docs.conda.io/
[pre-commit]: https://pre-commit.com/
[Jupyter]: https://jupyter.org/
[nbstripout]: https://github.com/kynan/nbstripout
[Google style]: http://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings
[PyScaffold]: https://pyscaffold.org/
[dsproject extension]: https://github.com/pyscaffold/pyscaffoldext-dsproject
To create the same starting point for this repository, as of 2022-06-01 on Windows you will need the development versions of PyScaffold and extensions, however this will not be necessary once certain bugfixes have been introduced in the next stable releases:
```bash
pip install git+https://github.com/pyscaffold/pyscaffold.git git+https://github.com/pyscaffold/pyscaffoldext-dsproject.git git+https://github.com/pyscaffold/pyscaffoldext-markdown.git
```
The following `pyscaffold` command creates a starting point for this repository:
```bash
putup xtal2png --github-actions --markdown --dsproj
```
Alternatively, you can edit a file interactively and update and uncomment relevant lines, which saves some of the additional setup:
```bash
putup --interactive xtal2png
```