MolNet_Benchmark_Tutorial.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"accelerator":"GPU","colab":{"name":"MolNet_Benchmark_Tutorial.ipynb","provenance":[{"file_id":"1f9ESZgWtCg0ZH4ljyBszH-ooIT4b0oXs","timestamp":1613682143417}],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.10"}},"cells":[{"cell_type":"markdown","metadata":{"id":"blY1hg-dJRgA"},"source":["# MoleculeNet Benchmarking\n","By [Nathan C. Frey](https://ncfrey.github.io/) | [Twitter](https://twitter.com/nc_frey)\n"," \n","This notebook shows how to benchmark model performance on a dataset from MoleculeNet and collect validation and test set metrics for the [MoleculeNet Leaderboard](https://github.com/deepchem/moleculenet).\n","\n","As an example, we use Bayesian optimization to tune the hyperparameters of a random forest and a GraphConv model to predict solubilities of small molecules from the [Delaney dataset](https://pubs.acs.org/doi/10.1021/ci034243x)."]},{"cell_type":"markdown","metadata":{"id":"wi3tcnn7YGln"},"source":["### Install condacolab\n","`condacolab` installs `mamba` or `miniconda` and automatically restarts the kernel. You'll see a \"session crashed unexpectedly\" message, but you can safely ignore this."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"QH4Xxi_lU_u7","executionInfo":{"status":"ok","timestamp":1619553630079,"user_tz":240,"elapsed":37618,"user":{"displayName":"Nathan Frey","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiCEtTj6AL3entEShxjitkGUQo5YhZ7CJA0917VzA=s64","userId":"14838914823565259795"}},"outputId":"62248f5f-736f-4853-fd57-239bc682bc04"},"source":["!pip install -q condacolab\n","import condacolab\n","condacolab.install_mambaforge()  #miniconda()"],"execution_count":1,"outputs":[{"output_type":"stream","text":["⏬ Downloading https://github.com/jaimergp/miniforge/releases/latest/download/Mambaforge-colab-Linux-x86_64.sh...\n","📦 Installing...\n","📌 Adjusting configuration...\n","🩹 Patching environment...\n","⏲ Done in 0:00:34\n","🔁 Restarting kernel...\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"JrhIJr9Kcptn"},"source":["### Setup Colab environment to run DeepChem and MoleculeNet\n","Install any additional dependencies required by your model or dataset."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"bw_pnMyPVFe4","executionInfo":{"status":"ok","timestamp":1619553657866,"user_tz":240,"elapsed":353,"user":{"displayName":"Nathan Frey","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiCEtTj6AL3entEShxjitkGUQo5YhZ7CJA0917VzA=s64","userId":"14838914823565259795"}},"outputId":"2eec6df7-54ca-45b7-e3d6-2d44a202e3fe"},"source":["import condacolab\n","condacolab.check()"],"execution_count":1,"outputs":[{"output_type":"stream","text":["✨🍰✨ Everything looks OK!\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"nJNAHyqRbI5t","executionInfo":{"status":"ok","timestamp":1619553658624,"user_tz":240,"elapsed":563,"user":{"displayName":"Nathan Frey","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiCEtTj6AL3entEShxjitkGUQo5YhZ7CJA0917VzA=s64","userId":"14838914823565259795"}},"outputId":"9fd392de-4d66-4a51-cf26-62bea7389255"},"source":["!conda --version"],"execution_count":2,"outputs":[{"output_type":"stream","text":["conda 4.9.2\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"JTKfN74566wC","executionInfo":{"status":"ok","timestamp":1619553667558,"user_tz":240,"elapsed":401,"user":{"displayName":"Nathan Frey","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiCEtTj6AL3entEShxjitkGUQo5YhZ7CJA0917VzA=s64","userId":"14838914823565259795"}},"outputId":"67434b79-9055-486f-e79d-144e46900764"},"source":["!nvcc --version"],"execution_count":3,"outputs":[{"output_type":"stream","text":["nvcc: NVIDIA (R) Cuda compiler driver\n","Copyright (c) 2005-2020 NVIDIA Corporation\n","Built on Wed_Jul_22_19:09:09_PDT_2020\n","Cuda compilation tools, release 11.0, V11.0.221\n","Build cuda_11.0_bu.TC445_37.28845127_0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"yujhHnJxfOBy"},"source":["Install DeepChem and dependencies. Add any dependencies that are unique to your model or dataset."]},{"cell_type":"code","metadata":{"id":"3JGFjSan48gU"},"source":["# !mamba install openmm pdbfixer -q  # dependencies for AtomicConvs and PDBbind datasets"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"ddsZrZwTrLtW"},"source":["!conda install -c conda-forge rdkit\n","!pip install tensorflow~=2.4 \n","\n","# Dependencies for benchmarking\n","!conda install -c dglteam dgl-cuda11.0\n","!pip install dgllife\n","!pip install hyperopt torch\n","!pip install --pre deepchem"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"qrW-Bi6vrJQv","executionInfo":{"status":"ok","timestamp":1619553990356,"user_tz":240,"elapsed":7389,"user":{"displayName":"Nathan Frey","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiCEtTj6AL3entEShxjitkGUQo5YhZ7CJA0917VzA=s64","userId":"14838914823565259795"}}},"source":["import rdkit\n","import deepchem as dc"],"execution_count":5,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ysCoe5dpfQEi"},"source":["### Clone the `moleculenet` repo"]},{"cell_type":"markdown","metadata":{"id":"tUzuv7gDfUIR"},"source":["Clone the `moleculenet` repo or your fork and branch of the `moleculenet` repo to access a benchmarking script."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"cVkcJCeaRsUH","executionInfo":{"status":"ok","timestamp":1619554003939,"user_tz":240,"elapsed":1143,"user":{"displayName":"Nathan Frey","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiCEtTj6AL3entEShxjitkGUQo5YhZ7CJA0917VzA=s64","userId":"14838914823565259795"}},"outputId":"0e15f117-ac05-45fc-a1b4-e9a0e22cc5bb"},"source":["!git clone https://github.com/deepchem/moleculenet.git\n","# !git clone --single-branch --branch <branch-name> https://github.com/<username>/moleculenet.git\n","# !(cd moleculenet && git pull)"],"execution_count":7,"outputs":[{"output_type":"stream","text":["Cloning into 'moleculenet'...\n","remote: Enumerating objects: 366, done.\u001b[K\n","remote: Counting objects: 100% (96/96), done.\u001b[K\n","remote: Compressing objects: 100% (76/76), done.\u001b[K\n","remote: Total 366 (delta 47), reused 49 (delta 18), pack-reused 270\u001b[K\n","Receiving objects: 100% (366/366), 59.19 KiB | 11.84 MiB/s, done.\n","Resolving deltas: 100% (189/189), done.\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"O2AwiNJzR9nN","executionInfo":{"status":"ok","timestamp":1619554006498,"user_tz":240,"elapsed":323,"user":{"displayName":"Nathan Frey","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiCEtTj6AL3entEShxjitkGUQo5YhZ7CJA0917VzA=s64","userId":"14838914823565259795"}},"outputId":"0f4f3583-d3e8-4a6b-f3e4-bf0d99bb3aa8"},"source":["%cd /content/moleculenet/examples"],"execution_count":8,"outputs":[{"output_type":"stream","text":["/content/moleculenet/examples\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"9ChnoOEmSSeW"},"source":["### Run benchmarking script with hyperparameter search\n","The `--help` option displays the possible arguments and default values. For a benchmarking run, you should run a hyperparameter search with Bayesian optimization by specifying `-hs` and setting the relevant parameters like `-r` and `-nt`. The moleculenet scripts use the [hyperopt library](http://hyperopt.github.io/hyperopt/) for hyperparameter optimization. The hyperparameter search space can be modified in the molnet python script (*e.g.*, `fingerprint.py`, `gnn.py`, etc.) if desired.\n","\n","If you don't specify the result path argument (`-p`), the results will be saved in `moleculenet/examples/results`. A folder is created for each trial of the hyperparameter search, and inside you will see `configure.json` (with the hyperparameters for that run) and `eval.txt` with the validation and test set metrics."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"6fluprLahdbU","executionInfo":{"status":"ok","timestamp":1619551484844,"user_tz":240,"elapsed":4187,"user":{"displayName":"Nathan Frey","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiCEtTj6AL3entEShxjitkGUQo5YhZ7CJA0917VzA=s64","userId":"14838914823565259795"}},"outputId":"0dae48e6-97a4-4a05-b554-ab5a234cea2c"},"source":["!python fingerprint.py --help"],"execution_count":9,"outputs":[{"output_type":"stream","text":["2021-04-27 19:24:42.042724: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0\n","usage: Examples for MoleculeNet with fingerprint [-h]\n","                                                 [-d {BACE_classification,BACE_regression,BBBP,ClinTox,Delaney,HOPV,SIDER,Lipo}]\n","                                                 [-m {RF}] [-f {ECFP}]\n","                                                 [-p RESULT_PATH]\n","                                                 [-r NUM_RUNS] [-hs]\n","                                                 [-nt NUM_TRIALS]\n","\n","optional arguments:\n","  -h, --help            show this help message and exit\n","  -d {BACE_classification,BACE_regression,BBBP,ClinTox,Delaney,HOPV,SIDER,Lipo}, --dataset {BACE_classification,BACE_regression,BBBP,ClinTox,Delaney,HOPV,SIDER,Lipo}\n","                        Dataset to use\n","  -m {RF}, --model {RF}\n","                        Options include 1) random forest (RF) (default: RF)\n","  -f {ECFP}, --featurizer {ECFP}\n","                        Options include 1) ECFP (default: ECFP)\n","  -p RESULT_PATH, --result-path RESULT_PATH\n","                        Path to save training results (default: results)\n","  -r NUM_RUNS, --num-runs NUM_RUNS\n","                        Number of runs for each hyperparameter configuration\n","                        (default: 3)\n","  -hs, --hyper-search   Whether to perform hyperparameter search or use the\n","                        default configuration. (default: False)\n","  -nt NUM_TRIALS, --num-trials NUM_TRIALS\n","                        Number of trials for hyperparameter search (default:\n","                        16)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Qjoy_hv6ehMk"},"source":["Let's run some simple benchmarks on the [Delaney drug solubility dataset](https://pubs.acs.org/doi/10.1021/ci034243x) of predicted log solubility in $mol/L$ and see how two different methods perform: 1) a random forest trained on 1024-bit circular fingerprints, and 2) a [GraphConv](https://arxiv.org/abs/1609.02907)."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"l1ZXZwvNW3EY","executionInfo":{"status":"ok","timestamp":1619552898681,"user_tz":240,"elapsed":1314089,"user":{"displayName":"Nathan Frey","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiCEtTj6AL3entEShxjitkGUQo5YhZ7CJA0917VzA=s64","userId":"14838914823565259795"}},"outputId":"5351af49-b4c0-4321-d57f-d9a248ecaed6"},"source":["!python fingerprint.py -d Delaney -hs\n","!cat results/eval.txt"],"execution_count":10,"outputs":[{"output_type":"stream","text":["2021-04-27 19:26:25.286420: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0\n","Created directory results\n","Start hyperparameter search with Bayesian optimization for 16 trials\n","Created directory results/1\n","Created directory results/2\n","Created directory results/3\n","Created directory results/4\n","Created directory results/5\n","Created directory results/6\n","Created directory results/7\n","Created directory results/8\n","Created directory results/9\n","Created directory results/10\n","Created directory results/11\n","Created directory results/12\n","Created directory results/13\n","Created directory results/14\n","Created directory results/15\n","Created directory results/16\n","100% 16/16 [21:50<00:00, 81.92s/trial, best loss: 1.7806047351134193]\n","Best val rmse: 1.7806 +- 0.0080\n","Test rmse: 1.6901 +- 0.0010\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"yQaWu2g50wMo"},"source":["After training the random forest model with 16 different hyperparameter configurations (and repeating each 3 times to collect statistics for the validation and test errors) and using Bayesian optimization, we find an optimal set of model hyperparameters (saved in `examples/configure.json`). We have a validation RMSE of 1.78 $\\pm$ 0.008 and a test RMSE of 1.69 $\\pm$ 0.001. "]},{"cell_type":"markdown","metadata":{"id":"EZFmAMJF5KBp"},"source":["Next, let's look at the same dataset with GraphConv models."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"-ZI2DGMP0sKU","executionInfo":{"status":"ok","timestamp":1619553179526,"user_tz":240,"elapsed":3044,"user":{"displayName":"Nathan Frey","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiCEtTj6AL3entEShxjitkGUQo5YhZ7CJA0917VzA=s64","userId":"14838914823565259795"}},"outputId":"e36ba06d-5391-49ad-f0db-d6448ce533f7"},"source":["!python gnn.py --help"],"execution_count":11,"outputs":[{"output_type":"stream","text":["2021-04-27 19:52:57.121764: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0\n","usage: Examples for MoleculeNet with GNN [-h]\n","                                         [-d {BACE_classification,BACE_regression,BBBP,ClinTox,Delaney,HOPV,SIDER,Lipo}]\n","                                         [-m {GCN}] [-f {GC}] [-p RESULT_PATH]\n","                                         [-r NUM_RUNS] [-pa PATIENCE] [-hs]\n","                                         [-nt NUM_TRIALS]\n","\n","optional arguments:\n","  -h, --help            show this help message and exit\n","  -d {BACE_classification,BACE_regression,BBBP,ClinTox,Delaney,HOPV,SIDER,Lipo}, --dataset {BACE_classification,BACE_regression,BBBP,ClinTox,Delaney,HOPV,SIDER,Lipo}\n","                        Dataset to use\n","  -m {GCN}, --model {GCN}\n","                        Options include 1) Graph Convolutional Network (GCN)\n","                        (default: GCN)\n","  -f {GC}, --featurizer {GC}\n","                        Options include 1) Graph Convolution (GC) (default:\n","                        GC)\n","  -p RESULT_PATH, --result-path RESULT_PATH\n","                        Path to save training results (default: results)\n","  -r NUM_RUNS, --num-runs NUM_RUNS\n","                        Number of runs for each hyperparameter configuration\n","                        (default: 3)\n","  -pa PATIENCE, --patience PATIENCE\n","                        Number of epochs to wait before early stop if\n","                        validation performance stops getting improved\n","                        (default: 30)\n","  -hs, --hyper-search   Whether to perform hyperparameter search or use the\n","                        default configuration. (default: False)\n","  -nt NUM_TRIALS, --num-trials NUM_TRIALS\n","                        Number of trials for hyperparameter search (default:\n","                        16)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"RHF-XeyP8Y80","executionInfo":{"status":"ok","timestamp":1619554048950,"user_tz":240,"elapsed":323,"user":{"displayName":"Nathan Frey","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiCEtTj6AL3entEShxjitkGUQo5YhZ7CJA0917VzA=s64","userId":"14838914823565259795"}}},"source":["!rm -rf results"],"execution_count":10,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"-kLRSiyK5Iwo","executionInfo":{"status":"ok","timestamp":1619556551795,"user_tz":240,"elapsed":2499489,"user":{"displayName":"Nathan Frey","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiCEtTj6AL3entEShxjitkGUQo5YhZ7CJA0917VzA=s64","userId":"14838914823565259795"}},"outputId":"8cc9450a-724f-4862-a845-59b819962dff"},"source":["!python gnn.py -d Delaney -hs\n","!cat results/eval.txt"],"execution_count":11,"outputs":[{"output_type":"stream","text":["2021-04-27 20:07:32.956194: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0\n","Created directory results\n","Start hyperparameter search with Bayesian optimization for 16 trials\n","Created directory results/1\n","  0% 0/16 [00:00<?, ?trial/s, best loss=?]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","DGL backend not selected or invalid.  Assuming PyTorch for now.\n","Setting the default backend to \"pytorch\". You can change it in the ~/.dgl/config.json file or export the DGLBACKEND environment variable.  Valid options are: pytorch, mxnet, tensorflow (all lowercase)\n","  0% 0/16 [00:05<?, ?trial/s, best loss=?]Using backend: pytorch\n","Created directory results/2\n","  6% 1/16 [01:49<27:27, 109.81s/trial, best loss: 1.5391837572656486]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","Created directory results/3\n"," 12% 2/16 [05:05<37:24, 160.30s/trial, best loss: 0.9818914359019653]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","Created directory results/4\n"," 19% 3/16 [08:38<39:54, 184.16s/trial, best loss: 0.9818914359019653]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","Created directory results/5\n"," 25% 4/16 [10:41<32:04, 160.36s/trial, best loss: 0.9818914359019653]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","Created directory results/6\n"," 31% 5/16 [13:06<28:21, 154.70s/trial, best loss: 0.9818914359019653]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","Created directory results/7\n"," 38% 6/16 [16:25<28:17, 169.78s/trial, best loss: 0.9818914359019653]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","Created directory results/8\n"," 44% 7/16 [19:45<26:57, 179.77s/trial, best loss: 0.9818914359019653]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","Created directory results/9\n"," 50% 8/16 [21:53<21:45, 163.17s/trial, best loss: 0.9818914359019653]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","Created directory results/10\n"," 56% 9/16 [24:13<18:11, 155.93s/trial, best loss: 0.9818914359019653]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","Created directory results/11\n"," 62% 10/16 [27:29<16:49, 168.31s/trial, best loss: 0.9818914359019653]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","Created directory results/12\n"," 69% 11/16 [30:32<14:23, 172.71s/trial, best loss: 0.9818914359019653]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","Created directory results/13\n"," 75% 12/16 [32:16<10:07, 151.93s/trial, best loss: 0.9818914359019653]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","Created directory results/14\n"," 81% 13/16 [34:43<07:31, 150.52s/trial, best loss: 0.9818914359019653]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","Created directory results/15\n"," 88% 14/16 [37:09<04:58, 149.14s/trial, best loss: 0.9818914359019653]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","Created directory results/16\n"," 94% 15/16 [39:14<02:21, 141.84s/trial, best loss: 0.9818914359019653]Failed to featurize datapoint 934, C. Appending empty array\n","\n","Exception message: zero-size array to reduction operation maximum which has no identity\n","\n","/usr/local/lib/python3.7/site-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n","  return array(a, dtype, copy=False, order=order)\n","\n","100% 16/16 [41:35<00:00, 155.97s/trial, best loss: 0.9818914359019653]\n","Best val rmse: 0.9819 +- 0.0253\n","Test rmse: 0.9253 +- 0.0182\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"n3JwQpz5HFid"},"source":["The GraphConv model is quite a bit more sophisticated than a random forest, so it's good to see that it performs much better in predicting solubility. And unlike random forests, which have comparatively simple hyperparameters to tune, here we tuned the dropout, learning rate, number of hidden layers and features within each layer, so Bayesian optimization is critical to really evaluate the model performance."]},{"cell_type":"markdown","metadata":{"id":"om-IzfmSgAZI"},"source":["### Run benchmark with `configure.json`\n","Alternatively, you can simply add a `<DATASET_NAME>.json` file in a folder called `'configures/<MODEL_NAME>_<FEATURIZER_NAME>.json` that specifies the hyperparameters and runs the benchmark without a hyperparameter search."]},{"cell_type":"code","metadata":{"id":"NYHk3KKggC9w","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1619557197779,"user_tz":240,"elapsed":69638,"user":{"displayName":"Nathan Frey","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiCEtTj6AL3entEShxjitkGUQo5YhZ7CJA0917VzA=s64","userId":"14838914823565259795"}},"outputId":"16a9a125-b4cc-4bf1-dff2-8768cfd8c4f4"},"source":["!python fingerprint.py -d Delaney "],"execution_count":12,"outputs":[{"output_type":"stream","text":["2021-04-27 20:58:48.902857: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0\n","Directory results already exists.\n","Use the manually specified hyperparameters\n","Val metric for 3 runs: 1.7685 +- 0.0115\n","Test metric for 3 runs: 1.7255 +- 0.0266\n"],"name":"stdout"}]}]}