train_model_Main_file.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "438fe4bb",
   "metadata": {},
   "source": [
    "# Note on feature engineering & techniques\n",
    "- volume/g is **WORKING**\n",
    "- n_atoms and atoms_mass_average **WORKING** (but need to use both, not just 1)\n",
    "- atoms_vol **WORKING**\n",
    "- weight / n_atoms **WORKING**\n",
    "- GSA from Monte Carlo **NOT WORKING**\n",
    "- surface_area / (void_vol * 100) is **NOT WORKING** (for prediction, but quite good for ratio to fill surface)\n",
    "- void_vol / volume/g is **NOT WORKING**\n",
    "- Abs(CO2_working_capacity [mL/g]) **NOT WORKING** (that means the negative values in label are valid)\n",
    "- All cell_length and cell_angle **NOT WORKING**\n",
    "- Sum_cell_length and Sum_cell_angle **NOT WORKING**\n",
    "- void_fraction * volume [A^3] **NOT WORKING**\n",
    "- void_fraction * weight [u] **NOT WORKING**\n",
    "- surface_area / n_atoms **NOT WORKING**\n",
    "- mol_total_mass **NOT WORKING** (mol_avg_mass is wayyy better)\n",
    "- pseudo labeling **NOT WORKING**\n",
    "- classify void_fraction to porosity_level (from 1 to 5) **NOT WORKING**\n",
    "- classify mol_total_mass to mol_size (from 1 to 5) **NOT WORKING**\n",
    "- preprocess train and pretest data in the different way result in worse score! (even only change the way to fill 1 variable) **DON'T DO THIS**\n",
    "- After all this time, including functional_groups still doesn't work and leads to overfitting, functional_groups **NOT WORKING**\n",
    "- MultiLabelBinarizer on organic_linker **NOT WORKING**\n",
    "- hba1, hba2, hbd, nF of MOF **NOT WORKING**\n",
    "- hba1, hba2, hbd, nF, linker_weight, linker_atoms **NOT WORKING**\n",
    "- heat / n_atoms **NOT WORKING**\n",
    "- CO2/N2_selectivity / n_atoms **NOT WORKING**\n",
    "- charges / n_atoms **NOT WORKING**\n",
    "- void_vol cm3 **NOT WORKING**\n",
    "- density / n_atoms **NOT WORKING**\n",
    "- atoms_vol / volume & volume / atoms_vol & void_vol / (atoms_vol / volume) **NOT WORKING**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d86f116",
   "metadata": {},
   "source": [
    "## Notes feats from di_cnn\n",
    "- All 11 feats work best for gb and lgbm\n",
    "- Less feats give a bit better result for TabNet\n",
    "- NOT use feats from di_cnn at all is best for mlp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "af41608b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:40.429721Z",
     "start_time": "2021-10-17T09:25:40.421762Z"
    }
   },
   "outputs": [],
   "source": [
    "import joblib\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.preprocessing import LabelBinarizer, MinMaxScaler, RobustScaler, StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4f89876d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:40.942721Z",
     "start_time": "2021-10-17T09:25:40.431723Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MOFname</th>\n",
       "      <th>volume [A^3]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>functional_groups</th>\n",
       "      <th>metal_linker</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>organic_linker2</th>\n",
       "      <th>...</th>\n",
       "      <th>_cell_length_c</th>\n",
       "      <th>_cell_angle_alpha</th>\n",
       "      <th>_cell_angle_beta</th>\n",
       "      <th>_cell_angle_gamma</th>\n",
       "      <th>n_atoms</th>\n",
       "      <th>mol_avg_mass</th>\n",
       "      <th>charges</th>\n",
       "      <th>mol_avg_radius</th>\n",
       "      <th>atoms_volume</th>\n",
       "      <th>atoms_area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mof_unit_1</td>\n",
       "      <td>1116.667429</td>\n",
       "      <td>875.240600</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.07899</td>\n",
       "      <td>0.0607</td>\n",
       "      <td>COOH-OEt</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>9.890832</td>\n",
       "      <td>1.569125</td>\n",
       "      <td>1.592480</td>\n",
       "      <td>1.575368</td>\n",
       "      <td>75</td>\n",
       "      <td>11.669907</td>\n",
       "      <td>92</td>\n",
       "      <td>59.653333</td>\n",
       "      <td>85.488522</td>\n",
       "      <td>359.334171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mof_unit_2</td>\n",
       "      <td>2769.503842</td>\n",
       "      <td>2211.697211</td>\n",
       "      <td>603.61</td>\n",
       "      <td>0.13794</td>\n",
       "      <td>0.1040</td>\n",
       "      <td>F-OMe</td>\n",
       "      <td>10</td>\n",
       "      <td>44</td>\n",
       "      <td>57</td>\n",
       "      <td>...</td>\n",
       "      <td>18.960098</td>\n",
       "      <td>1.746437</td>\n",
       "      <td>1.602488</td>\n",
       "      <td>1.691961</td>\n",
       "      <td>194</td>\n",
       "      <td>11.400559</td>\n",
       "      <td>250</td>\n",
       "      <td>60.515464</td>\n",
       "      <td>318.194213</td>\n",
       "      <td>1020.568686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mof_unit_3</td>\n",
       "      <td>1089.818728</td>\n",
       "      <td>773.687960</td>\n",
       "      <td>788.50</td>\n",
       "      <td>0.14874</td>\n",
       "      <td>0.1262</td>\n",
       "      <td>OMe-COOH</td>\n",
       "      <td>2</td>\n",
       "      <td>22</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>10.631996</td>\n",
       "      <td>1.556872</td>\n",
       "      <td>1.569806</td>\n",
       "      <td>1.577559</td>\n",
       "      <td>82</td>\n",
       "      <td>9.435293</td>\n",
       "      <td>120</td>\n",
       "      <td>59.585366</td>\n",
       "      <td>92.531908</td>\n",
       "      <td>390.662171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mof_unit_4</td>\n",
       "      <td>2205.198301</td>\n",
       "      <td>1304.638720</td>\n",
       "      <td>1441.53</td>\n",
       "      <td>0.21814</td>\n",
       "      <td>0.2220</td>\n",
       "      <td>H-SO3H</td>\n",
       "      <td>9</td>\n",
       "      <td>17</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>19.274980</td>\n",
       "      <td>1.911789</td>\n",
       "      <td>1.574891</td>\n",
       "      <td>1.580099</td>\n",
       "      <td>112</td>\n",
       "      <td>11.648598</td>\n",
       "      <td>204</td>\n",
       "      <td>63.500000</td>\n",
       "      <td>185.375000</td>\n",
       "      <td>638.296686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mof_unit_5</td>\n",
       "      <td>1137.800963</td>\n",
       "      <td>901.736120</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.07778</td>\n",
       "      <td>0.0591</td>\n",
       "      <td>NHMe-OH</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "      <td>...</td>\n",
       "      <td>10.853274</td>\n",
       "      <td>1.565467</td>\n",
       "      <td>1.622999</td>\n",
       "      <td>1.595312</td>\n",
       "      <td>94</td>\n",
       "      <td>9.593000</td>\n",
       "      <td>90</td>\n",
       "      <td>58.425532</td>\n",
       "      <td>98.775910</td>\n",
       "      <td>428.741029</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68608</th>\n",
       "      <td>mof_unit_68609</td>\n",
       "      <td>1188.302573</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>Pr-F</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>10.193870</td>\n",
       "      <td>1.585497</td>\n",
       "      <td>1.609910</td>\n",
       "      <td>1.583947</td>\n",
       "      <td>119</td>\n",
       "      <td>8.417773</td>\n",
       "      <td>216</td>\n",
       "      <td>59.277311</td>\n",
       "      <td>124.277288</td>\n",
       "      <td>552.300571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68609</th>\n",
       "      <td>mof_unit_68610</td>\n",
       "      <td>1506.660363</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01108</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>SO3H</td>\n",
       "      <td>10</td>\n",
       "      <td>42</td>\n",
       "      <td>46</td>\n",
       "      <td>...</td>\n",
       "      <td>15.033794</td>\n",
       "      <td>1.661287</td>\n",
       "      <td>1.730445</td>\n",
       "      <td>1.700483</td>\n",
       "      <td>126</td>\n",
       "      <td>11.851548</td>\n",
       "      <td>126</td>\n",
       "      <td>62.460317</td>\n",
       "      <td>261.730396</td>\n",
       "      <td>736.477029</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68610</th>\n",
       "      <td>mof_unit_68611</td>\n",
       "      <td>2035.532738</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>OPr</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>22</td>\n",
       "      <td>...</td>\n",
       "      <td>18.608120</td>\n",
       "      <td>1.574297</td>\n",
       "      <td>1.572863</td>\n",
       "      <td>1.034849</td>\n",
       "      <td>204</td>\n",
       "      <td>9.605572</td>\n",
       "      <td>366</td>\n",
       "      <td>61.578431</td>\n",
       "      <td>289.412179</td>\n",
       "      <td>1072.591771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68611</th>\n",
       "      <td>mof_unit_68612</td>\n",
       "      <td>3985.426053</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>OPr-Me</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>...</td>\n",
       "      <td>18.544746</td>\n",
       "      <td>1.578949</td>\n",
       "      <td>1.585477</td>\n",
       "      <td>1.569257</td>\n",
       "      <td>364</td>\n",
       "      <td>9.996454</td>\n",
       "      <td>652</td>\n",
       "      <td>62.054945</td>\n",
       "      <td>541.832410</td>\n",
       "      <td>1958.553143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68612</th>\n",
       "      <td>mof_unit_68613</td>\n",
       "      <td>1591.009408</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01609</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>I-OEt</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>13.706241</td>\n",
       "      <td>2.068063</td>\n",
       "      <td>1.280517</td>\n",
       "      <td>1.711999</td>\n",
       "      <td>116</td>\n",
       "      <td>17.855463</td>\n",
       "      <td>222</td>\n",
       "      <td>65.155172</td>\n",
       "      <td>176.607141</td>\n",
       "      <td>671.663771</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68613 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              MOFname  volume [A^3]   weight [u]  surface_area [m^2/g]  \\\n",
       "0          mof_unit_1   1116.667429   875.240600                  0.00   \n",
       "1          mof_unit_2   2769.503842  2211.697211                603.61   \n",
       "2          mof_unit_3   1089.818728   773.687960                788.50   \n",
       "3          mof_unit_4   2205.198301  1304.638720               1441.53   \n",
       "4          mof_unit_5   1137.800963   901.736120                  0.00   \n",
       "...               ...           ...          ...                   ...   \n",
       "68608  mof_unit_68609   1188.302573  1001.700216                  0.00   \n",
       "68609  mof_unit_68610   1506.660363  1493.296496                  0.00   \n",
       "68610  mof_unit_68611   2035.532738  1959.518320                  0.00   \n",
       "68611  mof_unit_68612   3985.426053  3638.677280                  0.00   \n",
       "68612  mof_unit_68613   1591.009408  2071.219000                  0.00   \n",
       "\n",
       "       void_fraction  void_volume [cm^3/g] functional_groups  metal_linker  \\\n",
       "0            0.07899                0.0607          COOH-OEt             3   \n",
       "1            0.13794                0.1040             F-OMe            10   \n",
       "2            0.14874                0.1262          OMe-COOH             2   \n",
       "3            0.21814                0.2220            H-SO3H             9   \n",
       "4            0.07778                0.0591           NHMe-OH             2   \n",
       "...              ...                   ...               ...           ...   \n",
       "68608        0.00000                0.0000              Pr-F             3   \n",
       "68609        0.01108                0.0000              SO3H            10   \n",
       "68610        0.00000                0.0000               OPr             4   \n",
       "68611        0.00000                0.0000            OPr-Me             4   \n",
       "68612        0.01609                0.0000             I-OEt             2   \n",
       "\n",
       "       organic_linker1  organic_linker2  ... _cell_length_c  \\\n",
       "0                    4               11  ...       9.890832   \n",
       "1                   44               57  ...      18.960098   \n",
       "2                   22               24  ...      10.631996   \n",
       "3                   17               24  ...      19.274980   \n",
       "4                    1               22  ...      10.853274   \n",
       "...                ...              ...  ...            ...   \n",
       "68608                4               24  ...      10.193870   \n",
       "68609               42               46  ...      15.033794   \n",
       "68610               14               22  ...      18.608120   \n",
       "68611                4               15  ...      18.544746   \n",
       "68612                9               16  ...      13.706241   \n",
       "\n",
       "       _cell_angle_alpha  _cell_angle_beta  _cell_angle_gamma  n_atoms  \\\n",
       "0               1.569125          1.592480           1.575368       75   \n",
       "1               1.746437          1.602488           1.691961      194   \n",
       "2               1.556872          1.569806           1.577559       82   \n",
       "3               1.911789          1.574891           1.580099      112   \n",
       "4               1.565467          1.622999           1.595312       94   \n",
       "...                  ...               ...                ...      ...   \n",
       "68608           1.585497          1.609910           1.583947      119   \n",
       "68609           1.661287          1.730445           1.700483      126   \n",
       "68610           1.574297          1.572863           1.034849      204   \n",
       "68611           1.578949          1.585477           1.569257      364   \n",
       "68612           2.068063          1.280517           1.711999      116   \n",
       "\n",
       "       mol_avg_mass  charges  mol_avg_radius  atoms_volume   atoms_area  \n",
       "0         11.669907       92       59.653333     85.488522   359.334171  \n",
       "1         11.400559      250       60.515464    318.194213  1020.568686  \n",
       "2          9.435293      120       59.585366     92.531908   390.662171  \n",
       "3         11.648598      204       63.500000    185.375000   638.296686  \n",
       "4          9.593000       90       58.425532     98.775910   428.741029  \n",
       "...             ...      ...             ...           ...          ...  \n",
       "68608      8.417773      216       59.277311    124.277288   552.300571  \n",
       "68609     11.851548      126       62.460317    261.730396   736.477029  \n",
       "68610      9.605572      366       61.578431    289.412179  1072.591771  \n",
       "68611      9.996454      652       62.054945    541.832410  1958.553143  \n",
       "68612     17.855463      222       65.155172    176.607141   671.663771  \n",
       "\n",
       "[68613 rows x 26 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('train_extra_2.csv')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4708b1a2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:40.958720Z",
     "start_time": "2021-10-17T09:25:40.945724Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['MOFname', 'volume [A^3]', 'weight [u]', 'surface_area [m^2/g]',\n",
       "       'void_fraction', 'void_volume [cm^3/g]', 'functional_groups',\n",
       "       'metal_linker', 'organic_linker1', 'organic_linker2', 'topology',\n",
       "       'CO2/N2_selectivity', 'heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]',\n",
       "       'CO2_working_capacity [mL/g]', '_cell_length_a', '_cell_length_b',\n",
       "       '_cell_length_c', '_cell_angle_alpha', '_cell_angle_beta',\n",
       "       '_cell_angle_gamma', 'n_atoms', 'mol_avg_mass', 'charges',\n",
       "       'mol_avg_radius', 'atoms_volume', 'atoms_area'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6cd9600e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:40.989721Z",
     "start_time": "2021-10-17T09:25:40.960722Z"
    }
   },
   "outputs": [],
   "source": [
    "# let's arrange and select only good features\n",
    "## features for MLP and NN\n",
    "# df = df[['MOFname', 'volume [A^3]', 'weight [u]', 'surface_area [m^2/g]',\n",
    "#        'void_fraction', 'void_volume [cm^3/g]', 'functional_groups',\n",
    "#        'metal_linker', 'organic_linker1', 'organic_linker2', 'topology',\n",
    "#        'CO2/N2_selectivity', 'heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]',\n",
    "#        'n_atoms', 'mol_avg_mass', 'charges', 'CO2_working_capacity [mL/g]']]\n",
    "\n",
    "df = df[['MOFname', 'volume [A^3]', 'weight [u]', 'surface_area [m^2/g]',\n",
    "       'void_fraction', 'void_volume [cm^3/g]', 'functional_groups',\n",
    "       'metal_linker', 'organic_linker1', 'organic_linker2', 'topology',\n",
    "       'CO2/N2_selectivity', 'heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]',\n",
    "       'n_atoms', 'mol_avg_mass', 'charges', 'mol_avg_radius', 'atoms_volume', \n",
    "       'atoms_area', 'CO2_working_capacity [mL/g]']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b7ee4a75",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:41.020722Z",
     "start_time": "2021-10-17T09:25:40.991721Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 68613 entries, 0 to 68612\n",
      "Data columns (total 20 columns):\n",
      " #   Column                                         Non-Null Count  Dtype  \n",
      "---  ------                                         --------------  -----  \n",
      " 0   MOFname                                        68613 non-null  object \n",
      " 1   volume [A^3]                                   68613 non-null  float64\n",
      " 2   weight [u]                                     68613 non-null  float64\n",
      " 3   surface_area [m^2/g]                           68613 non-null  float64\n",
      " 4   void_fraction                                  68613 non-null  float64\n",
      " 5   void_volume [cm^3/g]                           68613 non-null  float64\n",
      " 6   functional_groups                              68290 non-null  object \n",
      " 7   metal_linker                                   68613 non-null  int64  \n",
      " 8   organic_linker1                                68613 non-null  int64  \n",
      " 9   organic_linker2                                68613 non-null  int64  \n",
      " 10  topology                                       68613 non-null  object \n",
      " 11  CO2/N2_selectivity                             68613 non-null  float64\n",
      " 12  heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]  66526 non-null  float64\n",
      " 13  n_atoms                                        68613 non-null  int64  \n",
      " 14  mol_avg_mass                                   68613 non-null  float64\n",
      " 15  charges                                        68613 non-null  int64  \n",
      " 16  mol_avg_radius                                 68613 non-null  float64\n",
      " 17  atoms_volume                                   68613 non-null  float64\n",
      " 18  atoms_area                                     68613 non-null  float64\n",
      " 19  CO2_working_capacity [mL/g]                    68613 non-null  float64\n",
      "dtypes: float64(12), int64(5), object(3)\n",
      "memory usage: 10.5+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4706d5b6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:41.143724Z",
     "start_time": "2021-10-17T09:25:41.022723Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>volume [A^3]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>metal_linker</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>organic_linker2</th>\n",
       "      <th>CO2/N2_selectivity</th>\n",
       "      <th>heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]</th>\n",
       "      <th>n_atoms</th>\n",
       "      <th>mol_avg_mass</th>\n",
       "      <th>charges</th>\n",
       "      <th>mol_avg_radius</th>\n",
       "      <th>atoms_volume</th>\n",
       "      <th>atoms_area</th>\n",
       "      <th>CO2_working_capacity [mL/g]</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>68613.000000</td>\n",
       "      <td>68613.000000</td>\n",
       "      <td>68613.000000</td>\n",
       "      <td>68613.000000</td>\n",
       "      <td>68613.000000</td>\n",
       "      <td>68613.000000</td>\n",
       "      <td>68613.000000</td>\n",
       "      <td>68613.000000</td>\n",
       "      <td>68613.000000</td>\n",
       "      <td>6.652600e+04</td>\n",
       "      <td>68613.000000</td>\n",
       "      <td>68613.000000</td>\n",
       "      <td>68613.000000</td>\n",
       "      <td>68613.000000</td>\n",
       "      <td>68613.000000</td>\n",
       "      <td>68613.000000</td>\n",
       "      <td>68613.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3447.363207</td>\n",
       "      <td>1656.761858</td>\n",
       "      <td>1666.766690</td>\n",
       "      <td>0.259164</td>\n",
       "      <td>0.376052</td>\n",
       "      <td>4.203271</td>\n",
       "      <td>11.919257</td>\n",
       "      <td>20.583592</td>\n",
       "      <td>28.599681</td>\n",
       "      <td>inf</td>\n",
       "      <td>140.665253</td>\n",
       "      <td>11.990179</td>\n",
       "      <td>220.989055</td>\n",
       "      <td>61.988885</td>\n",
       "      <td>193.150420</td>\n",
       "      <td>739.177366</td>\n",
       "      <td>120.002797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>4840.665782</td>\n",
       "      <td>1259.086320</td>\n",
       "      <td>1366.317223</td>\n",
       "      <td>0.164758</td>\n",
       "      <td>0.476452</td>\n",
       "      <td>3.144905</td>\n",
       "      <td>10.783136</td>\n",
       "      <td>10.100870</td>\n",
       "      <td>153.806887</td>\n",
       "      <td>NaN</td>\n",
       "      <td>101.953227</td>\n",
       "      <td>3.010884</td>\n",
       "      <td>178.118021</td>\n",
       "      <td>2.017855</td>\n",
       "      <td>141.562607</td>\n",
       "      <td>532.277329</td>\n",
       "      <td>89.573112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>606.576038</td>\n",
       "      <td>439.281220</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.612299e+00</td>\n",
       "      <td>26.000000</td>\n",
       "      <td>6.507769</td>\n",
       "      <td>-344.000000</td>\n",
       "      <td>55.209302</td>\n",
       "      <td>42.584658</td>\n",
       "      <td>146.371657</td>\n",
       "      <td>-44.285746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1556.075767</td>\n",
       "      <td>893.662700</td>\n",
       "      <td>511.930000</td>\n",
       "      <td>0.142580</td>\n",
       "      <td>0.119000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>12.818366</td>\n",
       "      <td>5.267536e+00</td>\n",
       "      <td>78.000000</td>\n",
       "      <td>10.153402</td>\n",
       "      <td>114.000000</td>\n",
       "      <td>60.666667</td>\n",
       "      <td>96.954142</td>\n",
       "      <td>399.807886</td>\n",
       "      <td>65.537205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>2190.442847</td>\n",
       "      <td>1259.699253</td>\n",
       "      <td>1542.830000</td>\n",
       "      <td>0.242620</td>\n",
       "      <td>0.248400</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>19.689890</td>\n",
       "      <td>5.899089e+00</td>\n",
       "      <td>110.000000</td>\n",
       "      <td>11.369379</td>\n",
       "      <td>174.000000</td>\n",
       "      <td>61.720430</td>\n",
       "      <td>141.888015</td>\n",
       "      <td>569.976000</td>\n",
       "      <td>98.552185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>3605.836441</td>\n",
       "      <td>1897.163660</td>\n",
       "      <td>2517.960000</td>\n",
       "      <td>0.345120</td>\n",
       "      <td>0.443800</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>16.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>32.954388</td>\n",
       "      <td>6.768365e+00</td>\n",
       "      <td>164.000000</td>\n",
       "      <td>12.979880</td>\n",
       "      <td>266.000000</td>\n",
       "      <td>62.956522</td>\n",
       "      <td>238.156101</td>\n",
       "      <td>871.466514</td>\n",
       "      <td>163.139540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>223964.854408</td>\n",
       "      <td>22595.928960</td>\n",
       "      <td>7083.530000</td>\n",
       "      <td>0.872060</td>\n",
       "      <td>6.610100</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>29369.777780</td>\n",
       "      <td>inf</td>\n",
       "      <td>1776.000000</td>\n",
       "      <td>49.464101</td>\n",
       "      <td>4128.000000</td>\n",
       "      <td>80.238095</td>\n",
       "      <td>2471.014455</td>\n",
       "      <td>9893.392457</td>\n",
       "      <td>736.061636</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        volume [A^3]    weight [u]  surface_area [m^2/g]  void_fraction  \\\n",
       "count   68613.000000  68613.000000          68613.000000   68613.000000   \n",
       "mean     3447.363207   1656.761858           1666.766690       0.259164   \n",
       "std      4840.665782   1259.086320           1366.317223       0.164758   \n",
       "min       606.576038    439.281220             -1.000000      -1.000000   \n",
       "25%      1556.075767    893.662700            511.930000       0.142580   \n",
       "50%      2190.442847   1259.699253           1542.830000       0.242620   \n",
       "75%      3605.836441   1897.163660           2517.960000       0.345120   \n",
       "max    223964.854408  22595.928960           7083.530000       0.872060   \n",
       "\n",
       "       void_volume [cm^3/g]  metal_linker  organic_linker1  organic_linker2  \\\n",
       "count          68613.000000  68613.000000     68613.000000     68613.000000   \n",
       "mean               0.376052      4.203271        11.919257        20.583592   \n",
       "std                0.476452      3.144905        10.783136        10.100870   \n",
       "min                0.000000      1.000000         1.000000         1.000000   \n",
       "25%                0.119000      2.000000         4.000000        14.000000   \n",
       "50%                0.248400      3.000000        10.000000        20.000000   \n",
       "75%                0.443800      4.000000        16.000000        25.000000   \n",
       "max                6.610100     12.000000        59.000000        59.000000   \n",
       "\n",
       "       CO2/N2_selectivity  heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]  \\\n",
       "count        68613.000000                                   6.652600e+04   \n",
       "mean            28.599681                                            inf   \n",
       "std            153.806887                                            NaN   \n",
       "min              0.000000                                   1.612299e+00   \n",
       "25%             12.818366                                   5.267536e+00   \n",
       "50%             19.689890                                   5.899089e+00   \n",
       "75%             32.954388                                   6.768365e+00   \n",
       "max          29369.777780                                            inf   \n",
       "\n",
       "            n_atoms  mol_avg_mass       charges  mol_avg_radius  atoms_volume  \\\n",
       "count  68613.000000  68613.000000  68613.000000    68613.000000  68613.000000   \n",
       "mean     140.665253     11.990179    220.989055       61.988885    193.150420   \n",
       "std      101.953227      3.010884    178.118021        2.017855    141.562607   \n",
       "min       26.000000      6.507769   -344.000000       55.209302     42.584658   \n",
       "25%       78.000000     10.153402    114.000000       60.666667     96.954142   \n",
       "50%      110.000000     11.369379    174.000000       61.720430    141.888015   \n",
       "75%      164.000000     12.979880    266.000000       62.956522    238.156101   \n",
       "max     1776.000000     49.464101   4128.000000       80.238095   2471.014455   \n",
       "\n",
       "         atoms_area  CO2_working_capacity [mL/g]  \n",
       "count  68613.000000                 68613.000000  \n",
       "mean     739.177366                   120.002797  \n",
       "std      532.277329                    89.573112  \n",
       "min      146.371657                   -44.285746  \n",
       "25%      399.807886                    65.537205  \n",
       "50%      569.976000                    98.552185  \n",
       "75%      871.466514                   163.139540  \n",
       "max     9893.392457                   736.061636  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95031f5f",
   "metadata": {},
   "source": [
    "## Clean Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ae032e7",
   "metadata": {},
   "source": [
    "### Engineer new features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f4d1df95",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:41.159719Z",
     "start_time": "2021-10-17T09:25:41.145724Z"
    }
   },
   "outputs": [],
   "source": [
    "def check_all_problems(df):\n",
    "    if 'functional_groups' in df.columns:\n",
    "        print('Missing values in Functional_groups:', df['functional_groups'].isnull().sum())\n",
    "    print(f\"Missing values in Heat adsorption: {df['heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]'].isnull().sum()}, Inf value in Heat Adsorption: {len(df[df['heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]'] == np.inf])}\")\n",
    "    print(f\"0 values in void_volume: {len(df[df['void_volume [cm^3/g]']==0])}\")\n",
    "    print(f\"0 values in void_fraction: {len(df[df['void_fraction']==0])}, -1 value: {len(df[df['void_fraction']==-1])}\")\n",
    "    print(f\"0 values in surface_area: {len(df[df['surface_area [m^2/g]']==0])}, -1 value: {len(df[df['surface_area [m^2/g]']==-1])}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "611c08e3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:41.221721Z",
     "start_time": "2021-10-17T09:25:41.161722Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing values in Functional_groups: 323\n",
      "Missing values in Heat adsorption: 2087, Inf value in Heat Adsorption: 2\n",
      "0 values in void_volume: 3130\n",
      "0 values in void_fraction: 2874, -1 value: 56\n",
      "0 values in surface_area: 14483, -1 value: 56\n"
     ]
    }
   ],
   "source": [
    "check_all_problems(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fa5aac01",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:41.267721Z",
     "start_time": "2021-10-17T09:25:41.223723Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MOFname</th>\n",
       "      <th>volume [A^3]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>metal_linker</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>organic_linker2</th>\n",
       "      <th>topology</th>\n",
       "      <th>CO2/N2_selectivity</th>\n",
       "      <th>heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]</th>\n",
       "      <th>n_atoms</th>\n",
       "      <th>mol_avg_mass</th>\n",
       "      <th>charges</th>\n",
       "      <th>mol_avg_radius</th>\n",
       "      <th>atoms_volume</th>\n",
       "      <th>atoms_area</th>\n",
       "      <th>CO2_working_capacity [mL/g]</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mof_unit_1</td>\n",
       "      <td>1116.667429</td>\n",
       "      <td>875.240600</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.07899</td>\n",
       "      <td>0.0607</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>pcu</td>\n",
       "      <td>22.864166</td>\n",
       "      <td>6.786041</td>\n",
       "      <td>75</td>\n",
       "      <td>11.669907</td>\n",
       "      <td>92</td>\n",
       "      <td>59.653333</td>\n",
       "      <td>85.488522</td>\n",
       "      <td>359.334171</td>\n",
       "      <td>105.284502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mof_unit_2</td>\n",
       "      <td>2769.503842</td>\n",
       "      <td>2211.697211</td>\n",
       "      <td>603.61</td>\n",
       "      <td>0.13794</td>\n",
       "      <td>0.1040</td>\n",
       "      <td>10</td>\n",
       "      <td>44</td>\n",
       "      <td>57</td>\n",
       "      <td>etb</td>\n",
       "      <td>33.616780</td>\n",
       "      <td>7.147286</td>\n",
       "      <td>194</td>\n",
       "      <td>11.400559</td>\n",
       "      <td>250</td>\n",
       "      <td>60.515464</td>\n",
       "      <td>318.194213</td>\n",
       "      <td>1020.568686</td>\n",
       "      <td>101.224774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mof_unit_3</td>\n",
       "      <td>1089.818728</td>\n",
       "      <td>773.687960</td>\n",
       "      <td>788.50</td>\n",
       "      <td>0.14874</td>\n",
       "      <td>0.1262</td>\n",
       "      <td>2</td>\n",
       "      <td>22</td>\n",
       "      <td>24</td>\n",
       "      <td>pcu</td>\n",
       "      <td>19.263726</td>\n",
       "      <td>6.347967</td>\n",
       "      <td>82</td>\n",
       "      <td>9.435293</td>\n",
       "      <td>120</td>\n",
       "      <td>59.585366</td>\n",
       "      <td>92.531908</td>\n",
       "      <td>390.662171</td>\n",
       "      <td>118.987011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mof_unit_4</td>\n",
       "      <td>2205.198301</td>\n",
       "      <td>1304.638720</td>\n",
       "      <td>1441.53</td>\n",
       "      <td>0.21814</td>\n",
       "      <td>0.2220</td>\n",
       "      <td>9</td>\n",
       "      <td>17</td>\n",
       "      <td>24</td>\n",
       "      <td>sra</td>\n",
       "      <td>25.701377</td>\n",
       "      <td>6.190085</td>\n",
       "      <td>112</td>\n",
       "      <td>11.648598</td>\n",
       "      <td>204</td>\n",
       "      <td>63.500000</td>\n",
       "      <td>185.375000</td>\n",
       "      <td>638.296686</td>\n",
       "      <td>187.626004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mof_unit_5</td>\n",
       "      <td>1137.800963</td>\n",
       "      <td>901.736120</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.07778</td>\n",
       "      <td>0.0591</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "      <td>pcu</td>\n",
       "      <td>30.001838</td>\n",
       "      <td>6.478063</td>\n",
       "      <td>94</td>\n",
       "      <td>9.593000</td>\n",
       "      <td>90</td>\n",
       "      <td>58.425532</td>\n",
       "      <td>98.775910</td>\n",
       "      <td>428.741029</td>\n",
       "      <td>79.210001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68608</th>\n",
       "      <td>mof_unit_68609</td>\n",
       "      <td>1188.302573</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>24</td>\n",
       "      <td>pcu</td>\n",
       "      <td>24.131770</td>\n",
       "      <td>NaN</td>\n",
       "      <td>119</td>\n",
       "      <td>8.417773</td>\n",
       "      <td>216</td>\n",
       "      <td>59.277311</td>\n",
       "      <td>124.277288</td>\n",
       "      <td>552.300571</td>\n",
       "      <td>-12.943652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68609</th>\n",
       "      <td>mof_unit_68610</td>\n",
       "      <td>1506.660363</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01108</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>10</td>\n",
       "      <td>42</td>\n",
       "      <td>46</td>\n",
       "      <td>etb</td>\n",
       "      <td>6.071818</td>\n",
       "      <td>NaN</td>\n",
       "      <td>126</td>\n",
       "      <td>11.851548</td>\n",
       "      <td>126</td>\n",
       "      <td>62.460317</td>\n",
       "      <td>261.730396</td>\n",
       "      <td>736.477029</td>\n",
       "      <td>-12.985582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68610</th>\n",
       "      <td>mof_unit_68611</td>\n",
       "      <td>2035.532738</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>22</td>\n",
       "      <td>acs</td>\n",
       "      <td>9.876134</td>\n",
       "      <td>NaN</td>\n",
       "      <td>204</td>\n",
       "      <td>9.605572</td>\n",
       "      <td>366</td>\n",
       "      <td>61.578431</td>\n",
       "      <td>289.412179</td>\n",
       "      <td>1072.591771</td>\n",
       "      <td>-13.187635</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68611</th>\n",
       "      <td>mof_unit_68612</td>\n",
       "      <td>3985.426053</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>acs</td>\n",
       "      <td>5.285051</td>\n",
       "      <td>inf</td>\n",
       "      <td>364</td>\n",
       "      <td>9.996454</td>\n",
       "      <td>652</td>\n",
       "      <td>62.054945</td>\n",
       "      <td>541.832410</td>\n",
       "      <td>1958.553143</td>\n",
       "      <td>15.672698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68612</th>\n",
       "      <td>mof_unit_68613</td>\n",
       "      <td>1591.009408</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01609</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>pcu</td>\n",
       "      <td>2.621272</td>\n",
       "      <td>inf</td>\n",
       "      <td>116</td>\n",
       "      <td>17.855463</td>\n",
       "      <td>222</td>\n",
       "      <td>65.155172</td>\n",
       "      <td>176.607141</td>\n",
       "      <td>671.663771</td>\n",
       "      <td>3.144708</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68613 rows × 19 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              MOFname  volume [A^3]   weight [u]  surface_area [m^2/g]  \\\n",
       "0          mof_unit_1   1116.667429   875.240600                  0.00   \n",
       "1          mof_unit_2   2769.503842  2211.697211                603.61   \n",
       "2          mof_unit_3   1089.818728   773.687960                788.50   \n",
       "3          mof_unit_4   2205.198301  1304.638720               1441.53   \n",
       "4          mof_unit_5   1137.800963   901.736120                  0.00   \n",
       "...               ...           ...          ...                   ...   \n",
       "68608  mof_unit_68609   1188.302573  1001.700216                  0.00   \n",
       "68609  mof_unit_68610   1506.660363  1493.296496                  0.00   \n",
       "68610  mof_unit_68611   2035.532738  1959.518320                  0.00   \n",
       "68611  mof_unit_68612   3985.426053  3638.677280                  0.00   \n",
       "68612  mof_unit_68613   1591.009408  2071.219000                  0.00   \n",
       "\n",
       "       void_fraction  void_volume [cm^3/g]  metal_linker  organic_linker1  \\\n",
       "0            0.07899                0.0607             3                4   \n",
       "1            0.13794                0.1040            10               44   \n",
       "2            0.14874                0.1262             2               22   \n",
       "3            0.21814                0.2220             9               17   \n",
       "4            0.07778                0.0591             2                1   \n",
       "...              ...                   ...           ...              ...   \n",
       "68608        0.00000                0.0000             3                4   \n",
       "68609        0.01108                0.0000            10               42   \n",
       "68610        0.00000                0.0000             4               14   \n",
       "68611        0.00000                0.0000             4                4   \n",
       "68612        0.01609                0.0000             2                9   \n",
       "\n",
       "       organic_linker2 topology  CO2/N2_selectivity  \\\n",
       "0                   11      pcu           22.864166   \n",
       "1                   57      etb           33.616780   \n",
       "2                   24      pcu           19.263726   \n",
       "3                   24      sra           25.701377   \n",
       "4                   22      pcu           30.001838   \n",
       "...                ...      ...                 ...   \n",
       "68608               24      pcu           24.131770   \n",
       "68609               46      etb            6.071818   \n",
       "68610               22      acs            9.876134   \n",
       "68611               15      acs            5.285051   \n",
       "68612               16      pcu            2.621272   \n",
       "\n",
       "       heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]  n_atoms  mol_avg_mass  \\\n",
       "0                                           6.786041       75     11.669907   \n",
       "1                                           7.147286      194     11.400559   \n",
       "2                                           6.347967       82      9.435293   \n",
       "3                                           6.190085      112     11.648598   \n",
       "4                                           6.478063       94      9.593000   \n",
       "...                                              ...      ...           ...   \n",
       "68608                                            NaN      119      8.417773   \n",
       "68609                                            NaN      126     11.851548   \n",
       "68610                                            NaN      204      9.605572   \n",
       "68611                                            inf      364      9.996454   \n",
       "68612                                            inf      116     17.855463   \n",
       "\n",
       "       charges  mol_avg_radius  atoms_volume   atoms_area  \\\n",
       "0           92       59.653333     85.488522   359.334171   \n",
       "1          250       60.515464    318.194213  1020.568686   \n",
       "2          120       59.585366     92.531908   390.662171   \n",
       "3          204       63.500000    185.375000   638.296686   \n",
       "4           90       58.425532     98.775910   428.741029   \n",
       "...        ...             ...           ...          ...   \n",
       "68608      216       59.277311    124.277288   552.300571   \n",
       "68609      126       62.460317    261.730396   736.477029   \n",
       "68610      366       61.578431    289.412179  1072.591771   \n",
       "68611      652       62.054945    541.832410  1958.553143   \n",
       "68612      222       65.155172    176.607141   671.663771   \n",
       "\n",
       "       CO2_working_capacity [mL/g]  \n",
       "0                       105.284502  \n",
       "1                       101.224774  \n",
       "2                       118.987011  \n",
       "3                       187.626004  \n",
       "4                        79.210001  \n",
       "...                            ...  \n",
       "68608                   -12.943652  \n",
       "68609                   -12.985582  \n",
       "68610                   -13.187635  \n",
       "68611                    15.672698  \n",
       "68612                     3.144708  \n",
       "\n",
       "[68613 rows x 19 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# from a lot of model fitting, functional groups column is barely important\n",
    "# so, let's drop it\n",
    "df = df.drop(['functional_groups'], axis=1)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5ab1ee4a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:41.436723Z",
     "start_time": "2021-10-17T09:25:41.270723Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LabelBinarizer()"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We need to fit the binarizer before we dropping rows with invalid void_volume\n",
    "# since we are risking dropping some label in these columns and it will raise an error\n",
    "# if the test set contain some labels which our encoders have never seen before\n",
    "## ['functional_groups', 'metal_linker', 'organic_linker1', 'organic_linker2', 'topology']\n",
    "\n",
    "# fg_lb = LabelBinarizer()\n",
    "# fg_lb.fit(df['functional_groups'])\n",
    "\n",
    "metal_lb = LabelBinarizer()\n",
    "metal_lb.fit(df['metal_linker'])\n",
    "\n",
    "# organic_linker 1 and 2 contain ALL exactly the same label, so fit just 1 of them is OK\n",
    "organic_lb = LabelBinarizer()\n",
    "organic_lb.fit(df['organic_linker1'])\n",
    "\n",
    "topology_lb = LabelBinarizer()\n",
    "topology_lb.fit(df['topology'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "80697f88",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:41.467722Z",
     "start_time": "2021-10-17T09:25:41.439724Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 68613 entries, 0 to 68612\n",
      "Data columns (total 19 columns):\n",
      " #   Column                                         Non-Null Count  Dtype  \n",
      "---  ------                                         --------------  -----  \n",
      " 0   MOFname                                        68613 non-null  object \n",
      " 1   volume [A^3]                                   68613 non-null  float64\n",
      " 2   weight [u]                                     68613 non-null  float64\n",
      " 3   surface_area [m^2/g]                           68613 non-null  float64\n",
      " 4   void_fraction                                  68613 non-null  float64\n",
      " 5   void_volume [cm^3/g]                           68613 non-null  float64\n",
      " 6   metal_linker                                   68613 non-null  int64  \n",
      " 7   organic_linker1                                68613 non-null  int64  \n",
      " 8   organic_linker2                                68613 non-null  int64  \n",
      " 9   topology                                       68613 non-null  object \n",
      " 10  CO2/N2_selectivity                             68613 non-null  float64\n",
      " 11  heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]  66526 non-null  float64\n",
      " 12  n_atoms                                        68613 non-null  int64  \n",
      " 13  mol_avg_mass                                   68613 non-null  float64\n",
      " 14  charges                                        68613 non-null  int64  \n",
      " 15  mol_avg_radius                                 68613 non-null  float64\n",
      " 16  atoms_volume                                   68613 non-null  float64\n",
      " 17  atoms_area                                     68613 non-null  float64\n",
      " 18  CO2_working_capacity [mL/g]                    68613 non-null  float64\n",
      "dtypes: float64(12), int64(5), object(2)\n",
      "memory usage: 9.9+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23dd01a3",
   "metadata": {},
   "source": [
    "## Drop some serious outliers\n",
    "Dropping outliers leads to better result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "36212795",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:41.515721Z",
     "start_time": "2021-10-17T09:25:41.469725Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# all these cutting values, we look at it from EDA\n",
    "df = df.drop(df[df['volume [A^3]'] > 100000].index, axis=0)\n",
    "df = df.drop(df[df['CO2/N2_selectivity'] > 10000].index, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "734e017a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:41.545723Z",
     "start_time": "2021-10-17T09:25:41.518725Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing values in Heat adsorption: 2083, Inf value in Heat Adsorption: 2\n",
      "0 values in void_volume: 3126\n",
      "0 values in void_fraction: 2871, -1 value: 56\n",
      "0 values in surface_area: 14479, -1 value: 56\n"
     ]
    }
   ],
   "source": [
    "check_all_problems(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6a902f46",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:41.591722Z",
     "start_time": "2021-10-17T09:25:41.546722Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MOFname</th>\n",
       "      <th>volume [A^3]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>metal_linker</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>organic_linker2</th>\n",
       "      <th>topology</th>\n",
       "      <th>CO2/N2_selectivity</th>\n",
       "      <th>heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]</th>\n",
       "      <th>n_atoms</th>\n",
       "      <th>mol_avg_mass</th>\n",
       "      <th>charges</th>\n",
       "      <th>mol_avg_radius</th>\n",
       "      <th>atoms_volume</th>\n",
       "      <th>atoms_area</th>\n",
       "      <th>CO2_working_capacity [mL/g]</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mof_unit_1</td>\n",
       "      <td>1116.667429</td>\n",
       "      <td>875.240600</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.07899</td>\n",
       "      <td>0.0607</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>pcu</td>\n",
       "      <td>22.864166</td>\n",
       "      <td>6.786041</td>\n",
       "      <td>75</td>\n",
       "      <td>11.669907</td>\n",
       "      <td>92</td>\n",
       "      <td>59.653333</td>\n",
       "      <td>85.488522</td>\n",
       "      <td>359.334171</td>\n",
       "      <td>105.284502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mof_unit_2</td>\n",
       "      <td>2769.503842</td>\n",
       "      <td>2211.697211</td>\n",
       "      <td>603.61</td>\n",
       "      <td>0.13794</td>\n",
       "      <td>0.1040</td>\n",
       "      <td>10</td>\n",
       "      <td>44</td>\n",
       "      <td>57</td>\n",
       "      <td>etb</td>\n",
       "      <td>33.616780</td>\n",
       "      <td>7.147286</td>\n",
       "      <td>194</td>\n",
       "      <td>11.400559</td>\n",
       "      <td>250</td>\n",
       "      <td>60.515464</td>\n",
       "      <td>318.194213</td>\n",
       "      <td>1020.568686</td>\n",
       "      <td>101.224774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mof_unit_3</td>\n",
       "      <td>1089.818728</td>\n",
       "      <td>773.687960</td>\n",
       "      <td>788.50</td>\n",
       "      <td>0.14874</td>\n",
       "      <td>0.1262</td>\n",
       "      <td>2</td>\n",
       "      <td>22</td>\n",
       "      <td>24</td>\n",
       "      <td>pcu</td>\n",
       "      <td>19.263726</td>\n",
       "      <td>6.347967</td>\n",
       "      <td>82</td>\n",
       "      <td>9.435293</td>\n",
       "      <td>120</td>\n",
       "      <td>59.585366</td>\n",
       "      <td>92.531908</td>\n",
       "      <td>390.662171</td>\n",
       "      <td>118.987011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mof_unit_4</td>\n",
       "      <td>2205.198301</td>\n",
       "      <td>1304.638720</td>\n",
       "      <td>1441.53</td>\n",
       "      <td>0.21814</td>\n",
       "      <td>0.2220</td>\n",
       "      <td>9</td>\n",
       "      <td>17</td>\n",
       "      <td>24</td>\n",
       "      <td>sra</td>\n",
       "      <td>25.701377</td>\n",
       "      <td>6.190085</td>\n",
       "      <td>112</td>\n",
       "      <td>11.648598</td>\n",
       "      <td>204</td>\n",
       "      <td>63.500000</td>\n",
       "      <td>185.375000</td>\n",
       "      <td>638.296686</td>\n",
       "      <td>187.626004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mof_unit_5</td>\n",
       "      <td>1137.800963</td>\n",
       "      <td>901.736120</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.07778</td>\n",
       "      <td>0.0591</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "      <td>pcu</td>\n",
       "      <td>30.001838</td>\n",
       "      <td>6.478063</td>\n",
       "      <td>94</td>\n",
       "      <td>9.593000</td>\n",
       "      <td>90</td>\n",
       "      <td>58.425532</td>\n",
       "      <td>98.775910</td>\n",
       "      <td>428.741029</td>\n",
       "      <td>79.210001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68608</th>\n",
       "      <td>mof_unit_68609</td>\n",
       "      <td>1188.302573</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>24</td>\n",
       "      <td>pcu</td>\n",
       "      <td>24.131770</td>\n",
       "      <td>NaN</td>\n",
       "      <td>119</td>\n",
       "      <td>8.417773</td>\n",
       "      <td>216</td>\n",
       "      <td>59.277311</td>\n",
       "      <td>124.277288</td>\n",
       "      <td>552.300571</td>\n",
       "      <td>-12.943652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68609</th>\n",
       "      <td>mof_unit_68610</td>\n",
       "      <td>1506.660363</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01108</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>10</td>\n",
       "      <td>42</td>\n",
       "      <td>46</td>\n",
       "      <td>etb</td>\n",
       "      <td>6.071818</td>\n",
       "      <td>NaN</td>\n",
       "      <td>126</td>\n",
       "      <td>11.851548</td>\n",
       "      <td>126</td>\n",
       "      <td>62.460317</td>\n",
       "      <td>261.730396</td>\n",
       "      <td>736.477029</td>\n",
       "      <td>-12.985582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68610</th>\n",
       "      <td>mof_unit_68611</td>\n",
       "      <td>2035.532738</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>22</td>\n",
       "      <td>acs</td>\n",
       "      <td>9.876134</td>\n",
       "      <td>NaN</td>\n",
       "      <td>204</td>\n",
       "      <td>9.605572</td>\n",
       "      <td>366</td>\n",
       "      <td>61.578431</td>\n",
       "      <td>289.412179</td>\n",
       "      <td>1072.591771</td>\n",
       "      <td>-13.187635</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68611</th>\n",
       "      <td>mof_unit_68612</td>\n",
       "      <td>3985.426053</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>acs</td>\n",
       "      <td>5.285051</td>\n",
       "      <td>inf</td>\n",
       "      <td>364</td>\n",
       "      <td>9.996454</td>\n",
       "      <td>652</td>\n",
       "      <td>62.054945</td>\n",
       "      <td>541.832410</td>\n",
       "      <td>1958.553143</td>\n",
       "      <td>15.672698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68612</th>\n",
       "      <td>mof_unit_68613</td>\n",
       "      <td>1591.009408</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01609</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>pcu</td>\n",
       "      <td>2.621272</td>\n",
       "      <td>inf</td>\n",
       "      <td>116</td>\n",
       "      <td>17.855463</td>\n",
       "      <td>222</td>\n",
       "      <td>65.155172</td>\n",
       "      <td>176.607141</td>\n",
       "      <td>671.663771</td>\n",
       "      <td>3.144708</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 19 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              MOFname  volume [A^3]   weight [u]  surface_area [m^2/g]  \\\n",
       "0          mof_unit_1   1116.667429   875.240600                  0.00   \n",
       "1          mof_unit_2   2769.503842  2211.697211                603.61   \n",
       "2          mof_unit_3   1089.818728   773.687960                788.50   \n",
       "3          mof_unit_4   2205.198301  1304.638720               1441.53   \n",
       "4          mof_unit_5   1137.800963   901.736120                  0.00   \n",
       "...               ...           ...          ...                   ...   \n",
       "68608  mof_unit_68609   1188.302573  1001.700216                  0.00   \n",
       "68609  mof_unit_68610   1506.660363  1493.296496                  0.00   \n",
       "68610  mof_unit_68611   2035.532738  1959.518320                  0.00   \n",
       "68611  mof_unit_68612   3985.426053  3638.677280                  0.00   \n",
       "68612  mof_unit_68613   1591.009408  2071.219000                  0.00   \n",
       "\n",
       "       void_fraction  void_volume [cm^3/g]  metal_linker  organic_linker1  \\\n",
       "0            0.07899                0.0607             3                4   \n",
       "1            0.13794                0.1040            10               44   \n",
       "2            0.14874                0.1262             2               22   \n",
       "3            0.21814                0.2220             9               17   \n",
       "4            0.07778                0.0591             2                1   \n",
       "...              ...                   ...           ...              ...   \n",
       "68608        0.00000                0.0000             3                4   \n",
       "68609        0.01108                0.0000            10               42   \n",
       "68610        0.00000                0.0000             4               14   \n",
       "68611        0.00000                0.0000             4                4   \n",
       "68612        0.01609                0.0000             2                9   \n",
       "\n",
       "       organic_linker2 topology  CO2/N2_selectivity  \\\n",
       "0                   11      pcu           22.864166   \n",
       "1                   57      etb           33.616780   \n",
       "2                   24      pcu           19.263726   \n",
       "3                   24      sra           25.701377   \n",
       "4                   22      pcu           30.001838   \n",
       "...                ...      ...                 ...   \n",
       "68608               24      pcu           24.131770   \n",
       "68609               46      etb            6.071818   \n",
       "68610               22      acs            9.876134   \n",
       "68611               15      acs            5.285051   \n",
       "68612               16      pcu            2.621272   \n",
       "\n",
       "       heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]  n_atoms  mol_avg_mass  \\\n",
       "0                                           6.786041       75     11.669907   \n",
       "1                                           7.147286      194     11.400559   \n",
       "2                                           6.347967       82      9.435293   \n",
       "3                                           6.190085      112     11.648598   \n",
       "4                                           6.478063       94      9.593000   \n",
       "...                                              ...      ...           ...   \n",
       "68608                                            NaN      119      8.417773   \n",
       "68609                                            NaN      126     11.851548   \n",
       "68610                                            NaN      204      9.605572   \n",
       "68611                                            inf      364      9.996454   \n",
       "68612                                            inf      116     17.855463   \n",
       "\n",
       "       charges  mol_avg_radius  atoms_volume   atoms_area  \\\n",
       "0           92       59.653333     85.488522   359.334171   \n",
       "1          250       60.515464    318.194213  1020.568686   \n",
       "2          120       59.585366     92.531908   390.662171   \n",
       "3          204       63.500000    185.375000   638.296686   \n",
       "4           90       58.425532     98.775910   428.741029   \n",
       "...        ...             ...           ...          ...   \n",
       "68608      216       59.277311    124.277288   552.300571   \n",
       "68609      126       62.460317    261.730396   736.477029   \n",
       "68610      366       61.578431    289.412179  1072.591771   \n",
       "68611      652       62.054945    541.832410  1958.553143   \n",
       "68612      222       65.155172    176.607141   671.663771   \n",
       "\n",
       "       CO2_working_capacity [mL/g]  \n",
       "0                       105.284502  \n",
       "1                       101.224774  \n",
       "2                       118.987011  \n",
       "3                       187.626004  \n",
       "4                        79.210001  \n",
       "...                            ...  \n",
       "68608                   -12.943652  \n",
       "68609                   -12.985582  \n",
       "68610                   -13.187635  \n",
       "68611                    15.672698  \n",
       "68612                     3.144708  \n",
       "\n",
       "[68602 rows x 19 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bb32329",
   "metadata": {},
   "source": [
    "### Add new feats from DeepInsight CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8dd61e14",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:41.729720Z",
     "start_time": "2021-10-17T09:25:41.593722Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>di_cnn_1</th>\n",
       "      <th>di_cnn_2</th>\n",
       "      <th>di_cnn_4</th>\n",
       "      <th>di_cnn_5</th>\n",
       "      <th>di_cnn_6</th>\n",
       "      <th>di_cnn_8</th>\n",
       "      <th>di_cnn_10</th>\n",
       "      <th>di_cnn_12</th>\n",
       "      <th>di_cnn_13</th>\n",
       "      <th>di_cnn_14</th>\n",
       "      <th>di_cnn_15</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>21.167830</td>\n",
       "      <td>30.934912</td>\n",
       "      <td>0.0</td>\n",
       "      <td>21.876890</td>\n",
       "      <td>30.938911</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.025457</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>23.965372</td>\n",
       "      <td>35.462387</td>\n",
       "      <td>0.0</td>\n",
       "      <td>25.400830</td>\n",
       "      <td>35.742040</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.279602</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>25.874134</td>\n",
       "      <td>39.247130</td>\n",
       "      <td>0.0</td>\n",
       "      <td>27.537067</td>\n",
       "      <td>39.353320</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.338663</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.913395</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>28.681208</td>\n",
       "      <td>44.552822</td>\n",
       "      <td>0.0</td>\n",
       "      <td>31.782713</td>\n",
       "      <td>44.529415</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>20.764702</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.767157</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>21.825914</td>\n",
       "      <td>31.975190</td>\n",
       "      <td>0.0</td>\n",
       "      <td>22.386007</td>\n",
       "      <td>31.988407</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.455626</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.536157</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        di_cnn_1   di_cnn_2  di_cnn_4   di_cnn_5   di_cnn_6  di_cnn_8  \\\n",
       "0      21.167830  30.934912       0.0  21.876890  30.938911  0.000000   \n",
       "1      23.965372  35.462387       0.0  25.400830  35.742040  0.000000   \n",
       "2      25.874134  39.247130       0.0  27.537067  39.353320  0.000000   \n",
       "3      28.681208  44.552822       0.0  31.782713  44.529415  0.000000   \n",
       "4      21.825914  31.975190       0.0  22.386007  31.988407  0.000000   \n",
       "...          ...        ...       ...        ...        ...       ...   \n",
       "68597   0.000000   0.000000       0.0   0.000000   0.000000  2.536157   \n",
       "68598   0.000000   0.000000       0.0   0.000000   0.000000  0.000000   \n",
       "68599   0.000000   0.000000       0.0   0.000000   0.000000  0.000000   \n",
       "68600   0.000000   0.000000       0.0   0.000000   0.000000  0.000000   \n",
       "68601   0.000000   0.000000       0.0   0.000000   0.000000  0.000000   \n",
       "\n",
       "       di_cnn_10  di_cnn_12  di_cnn_13  di_cnn_14  di_cnn_15  \n",
       "0       0.000000        0.0        0.0  24.025457        0.0  \n",
       "1       0.000000        0.0        0.0  26.279602        0.0  \n",
       "2       5.338663        0.0        0.0  26.913395        0.0  \n",
       "3      20.764702        0.0        0.0  29.767157        0.0  \n",
       "4       0.000000        0.0        0.0  24.455626        0.0  \n",
       "...          ...        ...        ...        ...        ...  \n",
       "68597   0.000000        0.0        0.0   0.000000        0.0  \n",
       "68598   0.000000        0.0        0.0   0.000000        0.0  \n",
       "68599   0.000000        0.0        0.0   0.000000        0.0  \n",
       "68600   0.000000        0.0        0.0   0.000000        0.0  \n",
       "68601   0.000000        0.0        0.0   0.000000        0.0  \n",
       "\n",
       "[68602 rows x 11 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "di_cnn_feats = pd.read_csv('train_feats_from_di_cnn.csv')\n",
    "di_cnn_feats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "4afe7c54",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:41.791722Z",
     "start_time": "2021-10-17T09:25:41.731722Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>comb_cnn_0</th>\n",
       "      <th>comb_cnn_1</th>\n",
       "      <th>comb_cnn_2</th>\n",
       "      <th>comb_cnn_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>58.904250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>69.381874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.915159</td>\n",
       "      <td>82.243230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>117.579170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>61.207540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>0.716291</td>\n",
       "      <td>7.768239</td>\n",
       "      <td>5.032241</td>\n",
       "      <td>8.482945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>0.974198</td>\n",
       "      <td>8.142911</td>\n",
       "      <td>5.304706</td>\n",
       "      <td>7.986442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>0.778554</td>\n",
       "      <td>7.931119</td>\n",
       "      <td>5.137025</td>\n",
       "      <td>8.463214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>0.799744</td>\n",
       "      <td>7.906666</td>\n",
       "      <td>5.131299</td>\n",
       "      <td>8.344973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>1.005248</td>\n",
       "      <td>6.179784</td>\n",
       "      <td>3.833471</td>\n",
       "      <td>5.429558</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       comb_cnn_0  comb_cnn_1  comb_cnn_2  comb_cnn_3\n",
       "0        0.000000    0.000000    0.000000   58.904250\n",
       "1        0.000000    0.000000    0.000000   69.381874\n",
       "2        0.000000    0.000000    7.915159   82.243230\n",
       "3        0.000000    0.000000    0.000000  117.579170\n",
       "4        0.000000    0.000000    0.000000   61.207540\n",
       "...           ...         ...         ...         ...\n",
       "68597    0.716291    7.768239    5.032241    8.482945\n",
       "68598    0.974198    8.142911    5.304706    7.986442\n",
       "68599    0.778554    7.931119    5.137025    8.463214\n",
       "68600    0.799744    7.906666    5.131299    8.344973\n",
       "68601    1.005248    6.179784    3.833471    5.429558\n",
       "\n",
       "[68602 rows x 4 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comb_cnn_feats = pd.read_csv('train_feats_from_comb_cnn.csv')\n",
    "comb_cnn_feats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5788f9a1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:41.806727Z",
     "start_time": "2021-10-17T09:25:41.794727Z"
    }
   },
   "outputs": [],
   "source": [
    "df = df.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d7b8de38",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:41.929721Z",
     "start_time": "2021-10-17T09:25:41.808724Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MOFname</th>\n",
       "      <th>volume [A^3]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>metal_linker</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>organic_linker2</th>\n",
       "      <th>topology</th>\n",
       "      <th>...</th>\n",
       "      <th>di_cnn_8</th>\n",
       "      <th>di_cnn_10</th>\n",
       "      <th>di_cnn_12</th>\n",
       "      <th>di_cnn_13</th>\n",
       "      <th>di_cnn_14</th>\n",
       "      <th>di_cnn_15</th>\n",
       "      <th>comb_cnn_0</th>\n",
       "      <th>comb_cnn_1</th>\n",
       "      <th>comb_cnn_2</th>\n",
       "      <th>comb_cnn_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mof_unit_1</td>\n",
       "      <td>1116.667429</td>\n",
       "      <td>875.240600</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.07899</td>\n",
       "      <td>0.0607</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>pcu</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.025457</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>58.904250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mof_unit_2</td>\n",
       "      <td>2769.503842</td>\n",
       "      <td>2211.697211</td>\n",
       "      <td>603.61</td>\n",
       "      <td>0.13794</td>\n",
       "      <td>0.1040</td>\n",
       "      <td>10</td>\n",
       "      <td>44</td>\n",
       "      <td>57</td>\n",
       "      <td>etb</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.279602</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>69.381874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mof_unit_3</td>\n",
       "      <td>1089.818728</td>\n",
       "      <td>773.687960</td>\n",
       "      <td>788.50</td>\n",
       "      <td>0.14874</td>\n",
       "      <td>0.1262</td>\n",
       "      <td>2</td>\n",
       "      <td>22</td>\n",
       "      <td>24</td>\n",
       "      <td>pcu</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.338663</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.913395</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.915159</td>\n",
       "      <td>82.243230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mof_unit_4</td>\n",
       "      <td>2205.198301</td>\n",
       "      <td>1304.638720</td>\n",
       "      <td>1441.53</td>\n",
       "      <td>0.21814</td>\n",
       "      <td>0.2220</td>\n",
       "      <td>9</td>\n",
       "      <td>17</td>\n",
       "      <td>24</td>\n",
       "      <td>sra</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>20.764702</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.767157</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>117.579170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mof_unit_5</td>\n",
       "      <td>1137.800963</td>\n",
       "      <td>901.736120</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.07778</td>\n",
       "      <td>0.0591</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "      <td>pcu</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.455626</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>61.207540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>mof_unit_68609</td>\n",
       "      <td>1188.302573</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>24</td>\n",
       "      <td>pcu</td>\n",
       "      <td>...</td>\n",
       "      <td>2.536157</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.716291</td>\n",
       "      <td>7.768239</td>\n",
       "      <td>5.032241</td>\n",
       "      <td>8.482945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>mof_unit_68610</td>\n",
       "      <td>1506.660363</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01108</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>10</td>\n",
       "      <td>42</td>\n",
       "      <td>46</td>\n",
       "      <td>etb</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.974198</td>\n",
       "      <td>8.142911</td>\n",
       "      <td>5.304706</td>\n",
       "      <td>7.986442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>mof_unit_68611</td>\n",
       "      <td>2035.532738</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>22</td>\n",
       "      <td>acs</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.778554</td>\n",
       "      <td>7.931119</td>\n",
       "      <td>5.137025</td>\n",
       "      <td>8.463214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>mof_unit_68612</td>\n",
       "      <td>3985.426053</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>acs</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.799744</td>\n",
       "      <td>7.906666</td>\n",
       "      <td>5.131299</td>\n",
       "      <td>8.344973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>mof_unit_68613</td>\n",
       "      <td>1591.009408</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01609</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>pcu</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.005248</td>\n",
       "      <td>6.179784</td>\n",
       "      <td>3.833471</td>\n",
       "      <td>5.429558</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 34 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              MOFname  volume [A^3]   weight [u]  surface_area [m^2/g]  \\\n",
       "0          mof_unit_1   1116.667429   875.240600                  0.00   \n",
       "1          mof_unit_2   2769.503842  2211.697211                603.61   \n",
       "2          mof_unit_3   1089.818728   773.687960                788.50   \n",
       "3          mof_unit_4   2205.198301  1304.638720               1441.53   \n",
       "4          mof_unit_5   1137.800963   901.736120                  0.00   \n",
       "...               ...           ...          ...                   ...   \n",
       "68597  mof_unit_68609   1188.302573  1001.700216                  0.00   \n",
       "68598  mof_unit_68610   1506.660363  1493.296496                  0.00   \n",
       "68599  mof_unit_68611   2035.532738  1959.518320                  0.00   \n",
       "68600  mof_unit_68612   3985.426053  3638.677280                  0.00   \n",
       "68601  mof_unit_68613   1591.009408  2071.219000                  0.00   \n",
       "\n",
       "       void_fraction  void_volume [cm^3/g]  metal_linker  organic_linker1  \\\n",
       "0            0.07899                0.0607             3                4   \n",
       "1            0.13794                0.1040            10               44   \n",
       "2            0.14874                0.1262             2               22   \n",
       "3            0.21814                0.2220             9               17   \n",
       "4            0.07778                0.0591             2                1   \n",
       "...              ...                   ...           ...              ...   \n",
       "68597        0.00000                0.0000             3                4   \n",
       "68598        0.01108                0.0000            10               42   \n",
       "68599        0.00000                0.0000             4               14   \n",
       "68600        0.00000                0.0000             4                4   \n",
       "68601        0.01609                0.0000             2                9   \n",
       "\n",
       "       organic_linker2 topology  ...  di_cnn_8  di_cnn_10  di_cnn_12  \\\n",
       "0                   11      pcu  ...  0.000000   0.000000        0.0   \n",
       "1                   57      etb  ...  0.000000   0.000000        0.0   \n",
       "2                   24      pcu  ...  0.000000   5.338663        0.0   \n",
       "3                   24      sra  ...  0.000000  20.764702        0.0   \n",
       "4                   22      pcu  ...  0.000000   0.000000        0.0   \n",
       "...                ...      ...  ...       ...        ...        ...   \n",
       "68597               24      pcu  ...  2.536157   0.000000        0.0   \n",
       "68598               46      etb  ...  0.000000   0.000000        0.0   \n",
       "68599               22      acs  ...  0.000000   0.000000        0.0   \n",
       "68600               15      acs  ...  0.000000   0.000000        0.0   \n",
       "68601               16      pcu  ...  0.000000   0.000000        0.0   \n",
       "\n",
       "       di_cnn_13  di_cnn_14  di_cnn_15  comb_cnn_0  comb_cnn_1  comb_cnn_2  \\\n",
       "0            0.0  24.025457        0.0    0.000000    0.000000    0.000000   \n",
       "1            0.0  26.279602        0.0    0.000000    0.000000    0.000000   \n",
       "2            0.0  26.913395        0.0    0.000000    0.000000    7.915159   \n",
       "3            0.0  29.767157        0.0    0.000000    0.000000    0.000000   \n",
       "4            0.0  24.455626        0.0    0.000000    0.000000    0.000000   \n",
       "...          ...        ...        ...         ...         ...         ...   \n",
       "68597        0.0   0.000000        0.0    0.716291    7.768239    5.032241   \n",
       "68598        0.0   0.000000        0.0    0.974198    8.142911    5.304706   \n",
       "68599        0.0   0.000000        0.0    0.778554    7.931119    5.137025   \n",
       "68600        0.0   0.000000        0.0    0.799744    7.906666    5.131299   \n",
       "68601        0.0   0.000000        0.0    1.005248    6.179784    3.833471   \n",
       "\n",
       "       comb_cnn_3  \n",
       "0       58.904250  \n",
       "1       69.381874  \n",
       "2       82.243230  \n",
       "3      117.579170  \n",
       "4       61.207540  \n",
       "...           ...  \n",
       "68597    8.482945  \n",
       "68598    7.986442  \n",
       "68599    8.463214  \n",
       "68600    8.344973  \n",
       "68601    5.429558  \n",
       "\n",
       "[68602 rows x 34 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.concat([df, di_cnn_feats, comb_cnn_feats], axis=1)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d483067",
   "metadata": {},
   "source": [
    "### Density"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a3518346",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.006720Z",
     "start_time": "2021-10-17T09:25:41.932723Z"
    }
   },
   "outputs": [],
   "source": [
    "df.insert(\n",
    "    loc=2,\n",
    "    column=\"density [g/cm^3]\",\n",
    "    value=(df[\"weight [u]\"] / df[\"volume [A^3]\"]) * 1.66054,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ceccff75",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.098725Z",
     "start_time": "2021-10-17T09:25:42.008724Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MOFname</th>\n",
       "      <th>volume [A^3]</th>\n",
       "      <th>density [g/cm^3]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>metal_linker</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>organic_linker2</th>\n",
       "      <th>...</th>\n",
       "      <th>di_cnn_8</th>\n",
       "      <th>di_cnn_10</th>\n",
       "      <th>di_cnn_12</th>\n",
       "      <th>di_cnn_13</th>\n",
       "      <th>di_cnn_14</th>\n",
       "      <th>di_cnn_15</th>\n",
       "      <th>comb_cnn_0</th>\n",
       "      <th>comb_cnn_1</th>\n",
       "      <th>comb_cnn_2</th>\n",
       "      <th>comb_cnn_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mof_unit_1</td>\n",
       "      <td>1116.667429</td>\n",
       "      <td>1.301526</td>\n",
       "      <td>875.240600</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.07899</td>\n",
       "      <td>0.0607</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.025457</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>58.904250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mof_unit_2</td>\n",
       "      <td>2769.503842</td>\n",
       "      <td>1.326090</td>\n",
       "      <td>2211.697211</td>\n",
       "      <td>603.61</td>\n",
       "      <td>0.13794</td>\n",
       "      <td>0.1040</td>\n",
       "      <td>10</td>\n",
       "      <td>44</td>\n",
       "      <td>57</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.279602</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>69.381874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mof_unit_3</td>\n",
       "      <td>1089.818728</td>\n",
       "      <td>1.178856</td>\n",
       "      <td>773.687960</td>\n",
       "      <td>788.50</td>\n",
       "      <td>0.14874</td>\n",
       "      <td>0.1262</td>\n",
       "      <td>2</td>\n",
       "      <td>22</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.338663</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.913395</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.915159</td>\n",
       "      <td>82.243230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mof_unit_4</td>\n",
       "      <td>2205.198301</td>\n",
       "      <td>0.982408</td>\n",
       "      <td>1304.638720</td>\n",
       "      <td>1441.53</td>\n",
       "      <td>0.21814</td>\n",
       "      <td>0.2220</td>\n",
       "      <td>9</td>\n",
       "      <td>17</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>20.764702</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.767157</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>117.579170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mof_unit_5</td>\n",
       "      <td>1137.800963</td>\n",
       "      <td>1.316020</td>\n",
       "      <td>901.736120</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.07778</td>\n",
       "      <td>0.0591</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.455626</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>61.207540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>mof_unit_68609</td>\n",
       "      <td>1188.302573</td>\n",
       "      <td>1.399781</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>2.536157</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.716291</td>\n",
       "      <td>7.768239</td>\n",
       "      <td>5.032241</td>\n",
       "      <td>8.482945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>mof_unit_68610</td>\n",
       "      <td>1506.660363</td>\n",
       "      <td>1.645811</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01108</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>10</td>\n",
       "      <td>42</td>\n",
       "      <td>46</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.974198</td>\n",
       "      <td>8.142911</td>\n",
       "      <td>5.304706</td>\n",
       "      <td>7.986442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>mof_unit_68611</td>\n",
       "      <td>2035.532738</td>\n",
       "      <td>1.598529</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>22</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.778554</td>\n",
       "      <td>7.931119</td>\n",
       "      <td>5.137025</td>\n",
       "      <td>8.463214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>mof_unit_68612</td>\n",
       "      <td>3985.426053</td>\n",
       "      <td>1.516066</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.799744</td>\n",
       "      <td>7.906666</td>\n",
       "      <td>5.131299</td>\n",
       "      <td>8.344973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>mof_unit_68613</td>\n",
       "      <td>1591.009408</td>\n",
       "      <td>2.161736</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01609</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.005248</td>\n",
       "      <td>6.179784</td>\n",
       "      <td>3.833471</td>\n",
       "      <td>5.429558</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              MOFname  volume [A^3]  density [g/cm^3]   weight [u]  \\\n",
       "0          mof_unit_1   1116.667429          1.301526   875.240600   \n",
       "1          mof_unit_2   2769.503842          1.326090  2211.697211   \n",
       "2          mof_unit_3   1089.818728          1.178856   773.687960   \n",
       "3          mof_unit_4   2205.198301          0.982408  1304.638720   \n",
       "4          mof_unit_5   1137.800963          1.316020   901.736120   \n",
       "...               ...           ...               ...          ...   \n",
       "68597  mof_unit_68609   1188.302573          1.399781  1001.700216   \n",
       "68598  mof_unit_68610   1506.660363          1.645811  1493.296496   \n",
       "68599  mof_unit_68611   2035.532738          1.598529  1959.518320   \n",
       "68600  mof_unit_68612   3985.426053          1.516066  3638.677280   \n",
       "68601  mof_unit_68613   1591.009408          2.161736  2071.219000   \n",
       "\n",
       "       surface_area [m^2/g]  void_fraction  void_volume [cm^3/g]  \\\n",
       "0                      0.00        0.07899                0.0607   \n",
       "1                    603.61        0.13794                0.1040   \n",
       "2                    788.50        0.14874                0.1262   \n",
       "3                   1441.53        0.21814                0.2220   \n",
       "4                      0.00        0.07778                0.0591   \n",
       "...                     ...            ...                   ...   \n",
       "68597                  0.00        0.00000                0.0000   \n",
       "68598                  0.00        0.01108                0.0000   \n",
       "68599                  0.00        0.00000                0.0000   \n",
       "68600                  0.00        0.00000                0.0000   \n",
       "68601                  0.00        0.01609                0.0000   \n",
       "\n",
       "       metal_linker  organic_linker1  organic_linker2  ...  di_cnn_8  \\\n",
       "0                 3                4               11  ...  0.000000   \n",
       "1                10               44               57  ...  0.000000   \n",
       "2                 2               22               24  ...  0.000000   \n",
       "3                 9               17               24  ...  0.000000   \n",
       "4                 2                1               22  ...  0.000000   \n",
       "...             ...              ...              ...  ...       ...   \n",
       "68597             3                4               24  ...  2.536157   \n",
       "68598            10               42               46  ...  0.000000   \n",
       "68599             4               14               22  ...  0.000000   \n",
       "68600             4                4               15  ...  0.000000   \n",
       "68601             2                9               16  ...  0.000000   \n",
       "\n",
       "       di_cnn_10  di_cnn_12  di_cnn_13  di_cnn_14  di_cnn_15  comb_cnn_0  \\\n",
       "0       0.000000        0.0        0.0  24.025457        0.0    0.000000   \n",
       "1       0.000000        0.0        0.0  26.279602        0.0    0.000000   \n",
       "2       5.338663        0.0        0.0  26.913395        0.0    0.000000   \n",
       "3      20.764702        0.0        0.0  29.767157        0.0    0.000000   \n",
       "4       0.000000        0.0        0.0  24.455626        0.0    0.000000   \n",
       "...          ...        ...        ...        ...        ...         ...   \n",
       "68597   0.000000        0.0        0.0   0.000000        0.0    0.716291   \n",
       "68598   0.000000        0.0        0.0   0.000000        0.0    0.974198   \n",
       "68599   0.000000        0.0        0.0   0.000000        0.0    0.778554   \n",
       "68600   0.000000        0.0        0.0   0.000000        0.0    0.799744   \n",
       "68601   0.000000        0.0        0.0   0.000000        0.0    1.005248   \n",
       "\n",
       "       comb_cnn_1  comb_cnn_2  comb_cnn_3  \n",
       "0        0.000000    0.000000   58.904250  \n",
       "1        0.000000    0.000000   69.381874  \n",
       "2        0.000000    7.915159   82.243230  \n",
       "3        0.000000    0.000000  117.579170  \n",
       "4        0.000000    0.000000   61.207540  \n",
       "...           ...         ...         ...  \n",
       "68597    7.768239    5.032241    8.482945  \n",
       "68598    8.142911    5.304706    7.986442  \n",
       "68599    7.931119    5.137025    8.463214  \n",
       "68600    7.906666    5.131299    8.344973  \n",
       "68601    6.179784    3.833471    5.429558  \n",
       "\n",
       "[68602 rows x 35 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eefe97ce",
   "metadata": {},
   "source": [
    "###  Fill 200 rows that void_volume is 0 and void_fraction is not 0 or -1 with formula \" void_vol = void_fraction / density \""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3d2ed9b4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.160723Z",
     "start_time": "2021-10-17T09:25:42.101724Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MOFname</th>\n",
       "      <th>volume [A^3]</th>\n",
       "      <th>density [g/cm^3]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>metal_linker</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>organic_linker2</th>\n",
       "      <th>...</th>\n",
       "      <th>di_cnn_8</th>\n",
       "      <th>di_cnn_10</th>\n",
       "      <th>di_cnn_12</th>\n",
       "      <th>di_cnn_13</th>\n",
       "      <th>di_cnn_14</th>\n",
       "      <th>di_cnn_15</th>\n",
       "      <th>comb_cnn_0</th>\n",
       "      <th>comb_cnn_1</th>\n",
       "      <th>comb_cnn_2</th>\n",
       "      <th>comb_cnn_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>615</th>\n",
       "      <td>mof_unit_616</td>\n",
       "      <td>2986.257706</td>\n",
       "      <td>1.291033</td>\n",
       "      <td>2321.749040</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00286</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9</td>\n",
       "      <td>18</td>\n",
       "      <td>29</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>28.976114</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.749188</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>119.355064</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5772</th>\n",
       "      <td>mof_unit_5773</td>\n",
       "      <td>2208.884643</td>\n",
       "      <td>1.701815</td>\n",
       "      <td>2263.789760</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00815</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>18</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>28.084759</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.553674</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>85.663025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6631</th>\n",
       "      <td>mof_unit_6632</td>\n",
       "      <td>1595.944227</td>\n",
       "      <td>1.502962</td>\n",
       "      <td>1444.496040</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.01300</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9</td>\n",
       "      <td>22</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>27.241444</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.601681</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>70.222820</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8227</th>\n",
       "      <td>mof_unit_8228</td>\n",
       "      <td>1686.566488</td>\n",
       "      <td>1.321990</td>\n",
       "      <td>1342.710140</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00789</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>27.770300</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.410116</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>78.291700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9625</th>\n",
       "      <td>mof_unit_9626</td>\n",
       "      <td>3951.837174</td>\n",
       "      <td>1.621170</td>\n",
       "      <td>3858.143280</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.01559</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.016468</td>\n",
       "      <td>1.369474</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.356497</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>56.455738</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68578</th>\n",
       "      <td>mof_unit_68590</td>\n",
       "      <td>2981.283155</td>\n",
       "      <td>1.464359</td>\n",
       "      <td>2629.066240</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00486</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9</td>\n",
       "      <td>18</td>\n",
       "      <td>29</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.337637</td>\n",
       "      <td>9.405468</td>\n",
       "      <td>6.199437</td>\n",
       "      <td>11.978576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68582</th>\n",
       "      <td>mof_unit_68594</td>\n",
       "      <td>1524.505449</td>\n",
       "      <td>1.835401</td>\n",
       "      <td>1685.041936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00699</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10</td>\n",
       "      <td>46</td>\n",
       "      <td>48</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.903412</td>\n",
       "      <td>8.181130</td>\n",
       "      <td>5.331585</td>\n",
       "      <td>8.300731</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68584</th>\n",
       "      <td>mof_unit_68596</td>\n",
       "      <td>3760.489888</td>\n",
       "      <td>1.476275</td>\n",
       "      <td>3343.200672</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.01221</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10</td>\n",
       "      <td>42</td>\n",
       "      <td>42</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.951341</td>\n",
       "      <td>7.939940</td>\n",
       "      <td>5.150652</td>\n",
       "      <td>7.821176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>mof_unit_68610</td>\n",
       "      <td>1506.660363</td>\n",
       "      <td>1.645811</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.01108</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10</td>\n",
       "      <td>42</td>\n",
       "      <td>46</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.974198</td>\n",
       "      <td>8.142911</td>\n",
       "      <td>5.304706</td>\n",
       "      <td>7.986442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>mof_unit_68613</td>\n",
       "      <td>1591.009408</td>\n",
       "      <td>2.161736</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.01609</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.005248</td>\n",
       "      <td>6.179784</td>\n",
       "      <td>3.833471</td>\n",
       "      <td>5.429558</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>199 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              MOFname  volume [A^3]  density [g/cm^3]   weight [u]  \\\n",
       "615      mof_unit_616   2986.257706          1.291033  2321.749040   \n",
       "5772    mof_unit_5773   2208.884643          1.701815  2263.789760   \n",
       "6631    mof_unit_6632   1595.944227          1.502962  1444.496040   \n",
       "8227    mof_unit_8228   1686.566488          1.321990  1342.710140   \n",
       "9625    mof_unit_9626   3951.837174          1.621170  3858.143280   \n",
       "...               ...           ...               ...          ...   \n",
       "68578  mof_unit_68590   2981.283155          1.464359  2629.066240   \n",
       "68582  mof_unit_68594   1524.505449          1.835401  1685.041936   \n",
       "68584  mof_unit_68596   3760.489888          1.476275  3343.200672   \n",
       "68598  mof_unit_68610   1506.660363          1.645811  1493.296496   \n",
       "68601  mof_unit_68613   1591.009408          2.161736  2071.219000   \n",
       "\n",
       "       surface_area [m^2/g]  void_fraction  void_volume [cm^3/g]  \\\n",
       "615                     0.0        0.00286                   0.0   \n",
       "5772                    0.0        0.00815                   0.0   \n",
       "6631                    0.0        0.01300                   0.0   \n",
       "8227                    0.0        0.00789                   0.0   \n",
       "9625                    0.0        0.01559                   0.0   \n",
       "...                     ...            ...                   ...   \n",
       "68578                   0.0        0.00486                   0.0   \n",
       "68582                   0.0        0.00699                   0.0   \n",
       "68584                   0.0        0.01221                   0.0   \n",
       "68598                   0.0        0.01108                   0.0   \n",
       "68601                   0.0        0.01609                   0.0   \n",
       "\n",
       "       metal_linker  organic_linker1  organic_linker2  ... di_cnn_8  \\\n",
       "615               9               18               29  ...      0.0   \n",
       "5772              9               16               18  ...      0.0   \n",
       "6631              9               22               24  ...      0.0   \n",
       "8227              9               11                2  ...      0.0   \n",
       "9625              4                1               24  ...      0.0   \n",
       "...             ...              ...              ...  ...      ...   \n",
       "68578             9               18               29  ...      0.0   \n",
       "68582            10               46               48  ...      0.0   \n",
       "68584            10               42               42  ...      0.0   \n",
       "68598            10               42               46  ...      0.0   \n",
       "68601             2                9               16  ...      0.0   \n",
       "\n",
       "       di_cnn_10  di_cnn_12  di_cnn_13  di_cnn_14  di_cnn_15  comb_cnn_0  \\\n",
       "615          0.0        0.0        0.0  28.976114   0.000000    0.000000   \n",
       "5772         0.0        0.0        0.0  28.084759   0.000000    0.000000   \n",
       "6631         0.0        0.0        0.0  27.241444   0.000000    0.000000   \n",
       "8227         0.0        0.0        0.0  27.770300   0.000000    0.000000   \n",
       "9625         0.0        0.0        0.0  26.016468   1.369474    0.000000   \n",
       "...          ...        ...        ...        ...        ...         ...   \n",
       "68578        0.0        0.0        0.0   0.000000   0.000000    0.337637   \n",
       "68582        0.0        0.0        0.0   0.000000   0.000000    0.903412   \n",
       "68584        0.0        0.0        0.0   0.000000   0.000000    0.951341   \n",
       "68598        0.0        0.0        0.0   0.000000   0.000000    0.974198   \n",
       "68601        0.0        0.0        0.0   0.000000   0.000000    1.005248   \n",
       "\n",
       "       comb_cnn_1  comb_cnn_2  comb_cnn_3  \n",
       "615      4.749188    0.000000  119.355064  \n",
       "5772     5.553674    0.000000   85.663025  \n",
       "6631     3.601681    0.000000   70.222820  \n",
       "8227     2.410116    0.000000   78.291700  \n",
       "9625     1.356497    0.000000   56.455738  \n",
       "...           ...         ...         ...  \n",
       "68578    9.405468    6.199437   11.978576  \n",
       "68582    8.181130    5.331585    8.300731  \n",
       "68584    7.939940    5.150652    7.821176  \n",
       "68598    8.142911    5.304706    7.986442  \n",
       "68601    6.179784    3.833471    5.429558  \n",
       "\n",
       "[199 rows x 35 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fill_void_vol_df = df[(df['void_volume [cm^3/g]']==0) & ((df['void_fraction']!=0) & (df['void_fraction']!=-1))]\n",
    "fill_void_vol_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "bf5c0c1d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.175722Z",
     "start_time": "2021-10-17T09:25:42.163723Z"
    }
   },
   "outputs": [],
   "source": [
    "df.loc[((df['void_volume [cm^3/g]']==0) & ((df['void_fraction']!=0) & (df['void_fraction']!=-1))), 'void_volume [cm^3/g]'] = fill_void_vol_df['void_fraction'] / fill_void_vol_df['density [g/cm^3]']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6b8e8dba",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.205720Z",
     "start_time": "2021-10-17T09:25:42.177724Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MOFname</th>\n",
       "      <th>volume [A^3]</th>\n",
       "      <th>density [g/cm^3]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>metal_linker</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>organic_linker2</th>\n",
       "      <th>...</th>\n",
       "      <th>di_cnn_8</th>\n",
       "      <th>di_cnn_10</th>\n",
       "      <th>di_cnn_12</th>\n",
       "      <th>di_cnn_13</th>\n",
       "      <th>di_cnn_14</th>\n",
       "      <th>di_cnn_15</th>\n",
       "      <th>comb_cnn_0</th>\n",
       "      <th>comb_cnn_1</th>\n",
       "      <th>comb_cnn_2</th>\n",
       "      <th>comb_cnn_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>0 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [MOFname, volume [A^3], density [g/cm^3], weight [u], surface_area [m^2/g], void_fraction, void_volume [cm^3/g], metal_linker, organic_linker1, organic_linker2, topology, CO2/N2_selectivity, heat_adsorption_CO2_P0.15bar_T298K [kcal/mol], n_atoms, mol_avg_mass, charges, mol_avg_radius, atoms_volume, atoms_area, CO2_working_capacity [mL/g], di_cnn_1, di_cnn_2, di_cnn_4, di_cnn_5, di_cnn_6, di_cnn_8, di_cnn_10, di_cnn_12, di_cnn_13, di_cnn_14, di_cnn_15, comb_cnn_0, comb_cnn_1, comb_cnn_2, comb_cnn_3]\n",
       "Index: []\n",
       "\n",
       "[0 rows x 35 columns]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# verify\n",
    "df[(df['void_volume [cm^3/g]']==0) & ((df['void_fraction']!=0) & (df['void_fraction']!=-1))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "824f1ea2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.236723Z",
     "start_time": "2021-10-17T09:25:42.209724Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing values in Heat adsorption: 2083, Inf value in Heat Adsorption: 2\n",
      "0 values in void_volume: 2927\n",
      "0 values in void_fraction: 2871, -1 value: 56\n",
      "0 values in surface_area: 14479, -1 value: 56\n"
     ]
    }
   ],
   "source": [
    "check_all_problems(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f4fbe30",
   "metadata": {},
   "source": [
    "### Important feature errors note:\n",
    "From here, All rows that void_fraction are invalid (0 and -1), all void_volume are also 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "3bb0f405",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.251721Z",
     "start_time": "2021-10-17T09:25:42.238725Z"
    }
   },
   "outputs": [],
   "source": [
    "df.insert(\n",
    "    loc=2,\n",
    "    column=\"volume [cm^3/g]\",\n",
    "    value=df['volume [A^3]'] / (df['weight [u]'] * 1.66054),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "394704fe",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.358721Z",
     "start_time": "2021-10-17T09:25:42.253722Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MOFname</th>\n",
       "      <th>volume [A^3]</th>\n",
       "      <th>volume [cm^3/g]</th>\n",
       "      <th>density [g/cm^3]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>metal_linker</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>...</th>\n",
       "      <th>di_cnn_8</th>\n",
       "      <th>di_cnn_10</th>\n",
       "      <th>di_cnn_12</th>\n",
       "      <th>di_cnn_13</th>\n",
       "      <th>di_cnn_14</th>\n",
       "      <th>di_cnn_15</th>\n",
       "      <th>comb_cnn_0</th>\n",
       "      <th>comb_cnn_1</th>\n",
       "      <th>comb_cnn_2</th>\n",
       "      <th>comb_cnn_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mof_unit_1</td>\n",
       "      <td>1116.667429</td>\n",
       "      <td>0.768329</td>\n",
       "      <td>1.301526</td>\n",
       "      <td>875.240600</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.07899</td>\n",
       "      <td>0.060700</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.025457</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>58.904250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mof_unit_2</td>\n",
       "      <td>2769.503842</td>\n",
       "      <td>0.754097</td>\n",
       "      <td>1.326090</td>\n",
       "      <td>2211.697211</td>\n",
       "      <td>603.61</td>\n",
       "      <td>0.13794</td>\n",
       "      <td>0.104000</td>\n",
       "      <td>10</td>\n",
       "      <td>44</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.279602</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>69.381874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mof_unit_3</td>\n",
       "      <td>1089.818728</td>\n",
       "      <td>0.848280</td>\n",
       "      <td>1.178856</td>\n",
       "      <td>773.687960</td>\n",
       "      <td>788.50</td>\n",
       "      <td>0.14874</td>\n",
       "      <td>0.126200</td>\n",
       "      <td>2</td>\n",
       "      <td>22</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.338663</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.913395</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.915159</td>\n",
       "      <td>82.243230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mof_unit_4</td>\n",
       "      <td>2205.198301</td>\n",
       "      <td>1.017907</td>\n",
       "      <td>0.982408</td>\n",
       "      <td>1304.638720</td>\n",
       "      <td>1441.53</td>\n",
       "      <td>0.21814</td>\n",
       "      <td>0.222000</td>\n",
       "      <td>9</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>20.764702</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.767157</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>117.579170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mof_unit_5</td>\n",
       "      <td>1137.800963</td>\n",
       "      <td>0.759867</td>\n",
       "      <td>1.316020</td>\n",
       "      <td>901.736120</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.07778</td>\n",
       "      <td>0.059100</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.455626</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>61.207540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>mof_unit_68609</td>\n",
       "      <td>1188.302573</td>\n",
       "      <td>0.714398</td>\n",
       "      <td>1.399781</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>2.536157</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.716291</td>\n",
       "      <td>7.768239</td>\n",
       "      <td>5.032241</td>\n",
       "      <td>8.482945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>mof_unit_68610</td>\n",
       "      <td>1506.660363</td>\n",
       "      <td>0.607603</td>\n",
       "      <td>1.645811</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01108</td>\n",
       "      <td>0.006732</td>\n",
       "      <td>10</td>\n",
       "      <td>42</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.974198</td>\n",
       "      <td>8.142911</td>\n",
       "      <td>5.304706</td>\n",
       "      <td>7.986442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>mof_unit_68611</td>\n",
       "      <td>2035.532738</td>\n",
       "      <td>0.625575</td>\n",
       "      <td>1.598529</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.778554</td>\n",
       "      <td>7.931119</td>\n",
       "      <td>5.137025</td>\n",
       "      <td>8.463214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>mof_unit_68612</td>\n",
       "      <td>3985.426053</td>\n",
       "      <td>0.659602</td>\n",
       "      <td>1.516066</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.799744</td>\n",
       "      <td>7.906666</td>\n",
       "      <td>5.131299</td>\n",
       "      <td>8.344973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>mof_unit_68613</td>\n",
       "      <td>1591.009408</td>\n",
       "      <td>0.462591</td>\n",
       "      <td>2.161736</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.01609</td>\n",
       "      <td>0.007443</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.005248</td>\n",
       "      <td>6.179784</td>\n",
       "      <td>3.833471</td>\n",
       "      <td>5.429558</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              MOFname  volume [A^3]  volume [cm^3/g]  density [g/cm^3]  \\\n",
       "0          mof_unit_1   1116.667429         0.768329          1.301526   \n",
       "1          mof_unit_2   2769.503842         0.754097          1.326090   \n",
       "2          mof_unit_3   1089.818728         0.848280          1.178856   \n",
       "3          mof_unit_4   2205.198301         1.017907          0.982408   \n",
       "4          mof_unit_5   1137.800963         0.759867          1.316020   \n",
       "...               ...           ...              ...               ...   \n",
       "68597  mof_unit_68609   1188.302573         0.714398          1.399781   \n",
       "68598  mof_unit_68610   1506.660363         0.607603          1.645811   \n",
       "68599  mof_unit_68611   2035.532738         0.625575          1.598529   \n",
       "68600  mof_unit_68612   3985.426053         0.659602          1.516066   \n",
       "68601  mof_unit_68613   1591.009408         0.462591          2.161736   \n",
       "\n",
       "        weight [u]  surface_area [m^2/g]  void_fraction  void_volume [cm^3/g]  \\\n",
       "0       875.240600                  0.00        0.07899              0.060700   \n",
       "1      2211.697211                603.61        0.13794              0.104000   \n",
       "2       773.687960                788.50        0.14874              0.126200   \n",
       "3      1304.638720               1441.53        0.21814              0.222000   \n",
       "4       901.736120                  0.00        0.07778              0.059100   \n",
       "...            ...                   ...            ...                   ...   \n",
       "68597  1001.700216                  0.00        0.00000              0.000000   \n",
       "68598  1493.296496                  0.00        0.01108              0.006732   \n",
       "68599  1959.518320                  0.00        0.00000              0.000000   \n",
       "68600  3638.677280                  0.00        0.00000              0.000000   \n",
       "68601  2071.219000                  0.00        0.01609              0.007443   \n",
       "\n",
       "       metal_linker  organic_linker1  ...  di_cnn_8  di_cnn_10  di_cnn_12  \\\n",
       "0                 3                4  ...  0.000000   0.000000        0.0   \n",
       "1                10               44  ...  0.000000   0.000000        0.0   \n",
       "2                 2               22  ...  0.000000   5.338663        0.0   \n",
       "3                 9               17  ...  0.000000  20.764702        0.0   \n",
       "4                 2                1  ...  0.000000   0.000000        0.0   \n",
       "...             ...              ...  ...       ...        ...        ...   \n",
       "68597             3                4  ...  2.536157   0.000000        0.0   \n",
       "68598            10               42  ...  0.000000   0.000000        0.0   \n",
       "68599             4               14  ...  0.000000   0.000000        0.0   \n",
       "68600             4                4  ...  0.000000   0.000000        0.0   \n",
       "68601             2                9  ...  0.000000   0.000000        0.0   \n",
       "\n",
       "       di_cnn_13  di_cnn_14  di_cnn_15  comb_cnn_0  comb_cnn_1  comb_cnn_2  \\\n",
       "0            0.0  24.025457        0.0    0.000000    0.000000    0.000000   \n",
       "1            0.0  26.279602        0.0    0.000000    0.000000    0.000000   \n",
       "2            0.0  26.913395        0.0    0.000000    0.000000    7.915159   \n",
       "3            0.0  29.767157        0.0    0.000000    0.000000    0.000000   \n",
       "4            0.0  24.455626        0.0    0.000000    0.000000    0.000000   \n",
       "...          ...        ...        ...         ...         ...         ...   \n",
       "68597        0.0   0.000000        0.0    0.716291    7.768239    5.032241   \n",
       "68598        0.0   0.000000        0.0    0.974198    8.142911    5.304706   \n",
       "68599        0.0   0.000000        0.0    0.778554    7.931119    5.137025   \n",
       "68600        0.0   0.000000        0.0    0.799744    7.906666    5.131299   \n",
       "68601        0.0   0.000000        0.0    1.005248    6.179784    3.833471   \n",
       "\n",
       "       comb_cnn_3  \n",
       "0       58.904250  \n",
       "1       69.381874  \n",
       "2       82.243230  \n",
       "3      117.579170  \n",
       "4       61.207540  \n",
       "...           ...  \n",
       "68597    8.482945  \n",
       "68598    7.986442  \n",
       "68599    8.463214  \n",
       "68600    8.344973  \n",
       "68601    5.429558  \n",
       "\n",
       "[68602 rows x 36 columns]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd76e590",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-12T01:36:29.964035Z",
     "start_time": "2021-10-12T01:36:29.947067Z"
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5593287",
   "metadata": {},
   "source": [
    "## Fill void_vol"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6320f7e",
   "metadata": {},
   "source": [
    "**Phase 2**: Now q765 is better than q80, q70, q74, q75, q76, q763, q764, q7645, q7649, q76497, q76504, q76505, q7651, q7655, q766, q768, q77"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f374ea4",
   "metadata": {},
   "source": [
    "**Note:** q765 makes MLP better but, for gb, a little bit worse than q75."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "93f0fcb2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.405723Z",
     "start_time": "2021-10-17T09:25:42.359721Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4733999999999999"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# q80 variable here should be named to q765, we forgot it\n",
    "voidvol_q80 = df[df['void_volume [cm^3/g]']!=0]['void_volume [cm^3/g]'].quantile(0.765)\n",
    "voidvol_q80"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "77021e02",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.421723Z",
     "start_time": "2021-10-17T09:25:42.407722Z"
    }
   },
   "outputs": [],
   "source": [
    "df.loc[df['void_volume [cm^3/g]']==0, 'void_volume [cm^3/g]'] = voidvol_q80"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "774827c2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.452723Z",
     "start_time": "2021-10-17T09:25:42.427721Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing values in Heat adsorption: 2083, Inf value in Heat Adsorption: 2\n",
      "0 values in void_volume: 0\n",
      "0 values in void_fraction: 2871, -1 value: 56\n",
      "0 values in surface_area: 14479, -1 value: 56\n"
     ]
    }
   ],
   "source": [
    "check_all_problems(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99dca411",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1beef41",
   "metadata": {},
   "source": [
    "### Fill void_fraction with formula \" void_fraction = density * void_vol \""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "50619e20",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.513721Z",
     "start_time": "2021-10-17T09:25:42.454723Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MOFname</th>\n",
       "      <th>volume [A^3]</th>\n",
       "      <th>volume [cm^3/g]</th>\n",
       "      <th>density [g/cm^3]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>metal_linker</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>...</th>\n",
       "      <th>di_cnn_8</th>\n",
       "      <th>di_cnn_10</th>\n",
       "      <th>di_cnn_12</th>\n",
       "      <th>di_cnn_13</th>\n",
       "      <th>di_cnn_14</th>\n",
       "      <th>di_cnn_15</th>\n",
       "      <th>comb_cnn_0</th>\n",
       "      <th>comb_cnn_1</th>\n",
       "      <th>comb_cnn_2</th>\n",
       "      <th>comb_cnn_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>mof_unit_38</td>\n",
       "      <td>810.133031</td>\n",
       "      <td>0.705430</td>\n",
       "      <td>1.417576</td>\n",
       "      <td>691.597320</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4734</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.861506</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>31.085001</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6.205471</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>105.866270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>mof_unit_54</td>\n",
       "      <td>1136.093302</td>\n",
       "      <td>0.717497</td>\n",
       "      <td>1.393734</td>\n",
       "      <td>953.552560</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4734</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>22.974613</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30.284027</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6.287744</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>110.978970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>mof_unit_63</td>\n",
       "      <td>1105.832336</td>\n",
       "      <td>0.818345</td>\n",
       "      <td>1.221978</td>\n",
       "      <td>813.773143</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4734</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.013565</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.850498</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>91.242500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>152</th>\n",
       "      <td>mof_unit_153</td>\n",
       "      <td>1683.048333</td>\n",
       "      <td>0.783010</td>\n",
       "      <td>1.277123</td>\n",
       "      <td>1294.434240</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4734</td>\n",
       "      <td>2</td>\n",
       "      <td>14</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.279400</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.596128</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>64.816280</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>243</th>\n",
       "      <td>mof_unit_244</td>\n",
       "      <td>1840.691522</td>\n",
       "      <td>0.789265</td>\n",
       "      <td>1.267001</td>\n",
       "      <td>1404.457740</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4734</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>44.439384</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>33.793285</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.907324</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>167.302950</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68595</th>\n",
       "      <td>mof_unit_68607</td>\n",
       "      <td>949.067112</td>\n",
       "      <td>0.758226</td>\n",
       "      <td>1.318868</td>\n",
       "      <td>753.787520</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4734</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.941383</td>\n",
       "      <td>7.906218</td>\n",
       "      <td>5.176683</td>\n",
       "      <td>7.782713</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68596</th>\n",
       "      <td>mof_unit_68608</td>\n",
       "      <td>1202.182553</td>\n",
       "      <td>0.694431</td>\n",
       "      <td>1.440028</td>\n",
       "      <td>1042.538240</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4734</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.927152</td>\n",
       "      <td>7.960196</td>\n",
       "      <td>5.177389</td>\n",
       "      <td>7.929484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>mof_unit_68609</td>\n",
       "      <td>1188.302573</td>\n",
       "      <td>0.714398</td>\n",
       "      <td>1.399781</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4734</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>2.536157</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.716291</td>\n",
       "      <td>7.768239</td>\n",
       "      <td>5.032241</td>\n",
       "      <td>8.482945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>mof_unit_68611</td>\n",
       "      <td>2035.532738</td>\n",
       "      <td>0.625575</td>\n",
       "      <td>1.598529</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4734</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.778554</td>\n",
       "      <td>7.931119</td>\n",
       "      <td>5.137025</td>\n",
       "      <td>8.463214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>mof_unit_68612</td>\n",
       "      <td>3985.426053</td>\n",
       "      <td>0.659602</td>\n",
       "      <td>1.516066</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4734</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.799744</td>\n",
       "      <td>7.906666</td>\n",
       "      <td>5.131299</td>\n",
       "      <td>8.344973</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2927 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              MOFname  volume [A^3]  volume [cm^3/g]  density [g/cm^3]  \\\n",
       "37        mof_unit_38    810.133031         0.705430          1.417576   \n",
       "53        mof_unit_54   1136.093302         0.717497          1.393734   \n",
       "62        mof_unit_63   1105.832336         0.818345          1.221978   \n",
       "152      mof_unit_153   1683.048333         0.783010          1.277123   \n",
       "243      mof_unit_244   1840.691522         0.789265          1.267001   \n",
       "...               ...           ...              ...               ...   \n",
       "68595  mof_unit_68607    949.067112         0.758226          1.318868   \n",
       "68596  mof_unit_68608   1202.182553         0.694431          1.440028   \n",
       "68597  mof_unit_68609   1188.302573         0.714398          1.399781   \n",
       "68599  mof_unit_68611   2035.532738         0.625575          1.598529   \n",
       "68600  mof_unit_68612   3985.426053         0.659602          1.516066   \n",
       "\n",
       "        weight [u]  surface_area [m^2/g]  void_fraction  void_volume [cm^3/g]  \\\n",
       "37      691.597320                   0.0            0.0                0.4734   \n",
       "53      953.552560                   0.0            0.0                0.4734   \n",
       "62      813.773143                   0.0            0.0                0.4734   \n",
       "152    1294.434240                   0.0            0.0                0.4734   \n",
       "243    1404.457740                   0.0            0.0                0.4734   \n",
       "...            ...                   ...            ...                   ...   \n",
       "68595   753.787520                   0.0            0.0                0.4734   \n",
       "68596  1042.538240                   0.0            0.0                0.4734   \n",
       "68597  1001.700216                   0.0            0.0                0.4734   \n",
       "68599  1959.518320                   0.0            0.0                0.4734   \n",
       "68600  3638.677280                   0.0            0.0                0.4734   \n",
       "\n",
       "       metal_linker  organic_linker1  ...  di_cnn_8  di_cnn_10  di_cnn_12  \\\n",
       "37                2               10  ...  0.000000   1.861506        0.0   \n",
       "53                2                9  ...  0.000000  22.974613        0.0   \n",
       "62                2                4  ...  0.000000   0.000000        0.0   \n",
       "152               2               14  ...  0.000000   0.000000        0.0   \n",
       "243               9               16  ...  0.000000  44.439384        0.0   \n",
       "...             ...              ...  ...       ...        ...        ...   \n",
       "68595             2               10  ...  0.000000   0.000000        0.0   \n",
       "68596             3               12  ...  0.000000   0.000000        0.0   \n",
       "68597             3                4  ...  2.536157   0.000000        0.0   \n",
       "68599             4               14  ...  0.000000   0.000000        0.0   \n",
       "68600             4                4  ...  0.000000   0.000000        0.0   \n",
       "\n",
       "       di_cnn_13  di_cnn_14  di_cnn_15  comb_cnn_0  comb_cnn_1  comb_cnn_2  \\\n",
       "37           0.0  31.085001        0.0    0.000000    6.205471    0.000000   \n",
       "53           0.0  30.284027        0.0    0.000000    6.287744    0.000000   \n",
       "62           0.0  29.013565        0.0    0.000000    3.850498    0.000000   \n",
       "152          0.0  26.279400        0.0    0.000000    0.596128    0.000000   \n",
       "243          0.0  33.793285        0.0    0.000000    4.907324    0.000000   \n",
       "...          ...        ...        ...         ...         ...         ...   \n",
       "68595        0.0   0.000000        0.0    0.941383    7.906218    5.176683   \n",
       "68596        0.0   0.000000        0.0    0.927152    7.960196    5.177389   \n",
       "68597        0.0   0.000000        0.0    0.716291    7.768239    5.032241   \n",
       "68599        0.0   0.000000        0.0    0.778554    7.931119    5.137025   \n",
       "68600        0.0   0.000000        0.0    0.799744    7.906666    5.131299   \n",
       "\n",
       "       comb_cnn_3  \n",
       "37     105.866270  \n",
       "53     110.978970  \n",
       "62      91.242500  \n",
       "152     64.816280  \n",
       "243    167.302950  \n",
       "...           ...  \n",
       "68595    7.782713  \n",
       "68596    7.929484  \n",
       "68597    8.482945  \n",
       "68599    8.463214  \n",
       "68600    8.344973  \n",
       "\n",
       "[2927 rows x 36 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "temp_df = df[df['void_fraction'].isin([0, -1])]\n",
    "temp_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "5a7f9f4c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.529722Z",
     "start_time": "2021-10-17T09:25:42.515722Z"
    }
   },
   "outputs": [],
   "source": [
    "df.loc[df['void_fraction'].isin([0, -1]), 'void_fraction'] = temp_df['density [g/cm^3]'] * temp_df['void_volume [cm^3/g]']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "55294f8e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.560721Z",
     "start_time": "2021-10-17T09:25:42.534727Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing values in Heat adsorption: 2083, Inf value in Heat Adsorption: 2\n",
      "0 values in void_volume: 0\n",
      "0 values in void_fraction: 0, -1 value: 0\n",
      "0 values in surface_area: 14479, -1 value: 56\n"
     ]
    }
   ],
   "source": [
    "check_all_problems(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a957e9c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-12T01:36:29.964035Z",
     "start_time": "2021-10-12T01:36:29.947067Z"
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d846588a",
   "metadata": {},
   "source": [
    "## Impute surface_area"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "2c63cee3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.576725Z",
     "start_time": "2021-10-17T09:25:42.562723Z"
    }
   },
   "outputs": [],
   "source": [
    "df_imp = df.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "48e713cf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.653722Z",
     "start_time": "2021-10-17T09:25:42.578723Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MOFname</th>\n",
       "      <th>volume [A^3]</th>\n",
       "      <th>volume [cm^3/g]</th>\n",
       "      <th>density [g/cm^3]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>metal_linker</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>...</th>\n",
       "      <th>di_cnn_8</th>\n",
       "      <th>di_cnn_10</th>\n",
       "      <th>di_cnn_12</th>\n",
       "      <th>di_cnn_13</th>\n",
       "      <th>di_cnn_14</th>\n",
       "      <th>di_cnn_15</th>\n",
       "      <th>comb_cnn_0</th>\n",
       "      <th>comb_cnn_1</th>\n",
       "      <th>comb_cnn_2</th>\n",
       "      <th>comb_cnn_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mof_unit_1</td>\n",
       "      <td>1116.667429</td>\n",
       "      <td>0.768329</td>\n",
       "      <td>1.301526</td>\n",
       "      <td>875.240600</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.078990</td>\n",
       "      <td>0.060700</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.025457</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>58.904250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mof_unit_2</td>\n",
       "      <td>2769.503842</td>\n",
       "      <td>0.754097</td>\n",
       "      <td>1.326090</td>\n",
       "      <td>2211.697211</td>\n",
       "      <td>603.61</td>\n",
       "      <td>0.137940</td>\n",
       "      <td>0.104000</td>\n",
       "      <td>10</td>\n",
       "      <td>44</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.279602</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>69.381874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mof_unit_3</td>\n",
       "      <td>1089.818728</td>\n",
       "      <td>0.848280</td>\n",
       "      <td>1.178856</td>\n",
       "      <td>773.687960</td>\n",
       "      <td>788.50</td>\n",
       "      <td>0.148740</td>\n",
       "      <td>0.126200</td>\n",
       "      <td>2</td>\n",
       "      <td>22</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.338663</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.913395</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.915159</td>\n",
       "      <td>82.243230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mof_unit_4</td>\n",
       "      <td>2205.198301</td>\n",
       "      <td>1.017907</td>\n",
       "      <td>0.982408</td>\n",
       "      <td>1304.638720</td>\n",
       "      <td>1441.53</td>\n",
       "      <td>0.218140</td>\n",
       "      <td>0.222000</td>\n",
       "      <td>9</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>20.764702</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.767157</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>117.579170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mof_unit_5</td>\n",
       "      <td>1137.800963</td>\n",
       "      <td>0.759867</td>\n",
       "      <td>1.316020</td>\n",
       "      <td>901.736120</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.077780</td>\n",
       "      <td>0.059100</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.455626</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>61.207540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>mof_unit_68609</td>\n",
       "      <td>1188.302573</td>\n",
       "      <td>0.714398</td>\n",
       "      <td>1.399781</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.662656</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>2.536157</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.716291</td>\n",
       "      <td>7.768239</td>\n",
       "      <td>5.032241</td>\n",
       "      <td>8.482945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>mof_unit_68610</td>\n",
       "      <td>1506.660363</td>\n",
       "      <td>0.607603</td>\n",
       "      <td>1.645811</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.011080</td>\n",
       "      <td>0.006732</td>\n",
       "      <td>10</td>\n",
       "      <td>42</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.974198</td>\n",
       "      <td>8.142911</td>\n",
       "      <td>5.304706</td>\n",
       "      <td>7.986442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>mof_unit_68611</td>\n",
       "      <td>2035.532738</td>\n",
       "      <td>0.625575</td>\n",
       "      <td>1.598529</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.756744</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.778554</td>\n",
       "      <td>7.931119</td>\n",
       "      <td>5.137025</td>\n",
       "      <td>8.463214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>mof_unit_68612</td>\n",
       "      <td>3985.426053</td>\n",
       "      <td>0.659602</td>\n",
       "      <td>1.516066</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.717706</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.799744</td>\n",
       "      <td>7.906666</td>\n",
       "      <td>5.131299</td>\n",
       "      <td>8.344973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>mof_unit_68613</td>\n",
       "      <td>1591.009408</td>\n",
       "      <td>0.462591</td>\n",
       "      <td>2.161736</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.016090</td>\n",
       "      <td>0.007443</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.005248</td>\n",
       "      <td>6.179784</td>\n",
       "      <td>3.833471</td>\n",
       "      <td>5.429558</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              MOFname  volume [A^3]  volume [cm^3/g]  density [g/cm^3]  \\\n",
       "0          mof_unit_1   1116.667429         0.768329          1.301526   \n",
       "1          mof_unit_2   2769.503842         0.754097          1.326090   \n",
       "2          mof_unit_3   1089.818728         0.848280          1.178856   \n",
       "3          mof_unit_4   2205.198301         1.017907          0.982408   \n",
       "4          mof_unit_5   1137.800963         0.759867          1.316020   \n",
       "...               ...           ...              ...               ...   \n",
       "68597  mof_unit_68609   1188.302573         0.714398          1.399781   \n",
       "68598  mof_unit_68610   1506.660363         0.607603          1.645811   \n",
       "68599  mof_unit_68611   2035.532738         0.625575          1.598529   \n",
       "68600  mof_unit_68612   3985.426053         0.659602          1.516066   \n",
       "68601  mof_unit_68613   1591.009408         0.462591          2.161736   \n",
       "\n",
       "        weight [u]  surface_area [m^2/g]  void_fraction  void_volume [cm^3/g]  \\\n",
       "0       875.240600                   NaN       0.078990              0.060700   \n",
       "1      2211.697211                603.61       0.137940              0.104000   \n",
       "2       773.687960                788.50       0.148740              0.126200   \n",
       "3      1304.638720               1441.53       0.218140              0.222000   \n",
       "4       901.736120                   NaN       0.077780              0.059100   \n",
       "...            ...                   ...            ...                   ...   \n",
       "68597  1001.700216                   NaN       0.662656              0.473400   \n",
       "68598  1493.296496                   NaN       0.011080              0.006732   \n",
       "68599  1959.518320                   NaN       0.756744              0.473400   \n",
       "68600  3638.677280                   NaN       0.717706              0.473400   \n",
       "68601  2071.219000                   NaN       0.016090              0.007443   \n",
       "\n",
       "       metal_linker  organic_linker1  ...  di_cnn_8  di_cnn_10  di_cnn_12  \\\n",
       "0                 3                4  ...  0.000000   0.000000        0.0   \n",
       "1                10               44  ...  0.000000   0.000000        0.0   \n",
       "2                 2               22  ...  0.000000   5.338663        0.0   \n",
       "3                 9               17  ...  0.000000  20.764702        0.0   \n",
       "4                 2                1  ...  0.000000   0.000000        0.0   \n",
       "...             ...              ...  ...       ...        ...        ...   \n",
       "68597             3                4  ...  2.536157   0.000000        0.0   \n",
       "68598            10               42  ...  0.000000   0.000000        0.0   \n",
       "68599             4               14  ...  0.000000   0.000000        0.0   \n",
       "68600             4                4  ...  0.000000   0.000000        0.0   \n",
       "68601             2                9  ...  0.000000   0.000000        0.0   \n",
       "\n",
       "       di_cnn_13  di_cnn_14  di_cnn_15  comb_cnn_0  comb_cnn_1  comb_cnn_2  \\\n",
       "0            0.0  24.025457        0.0    0.000000    0.000000    0.000000   \n",
       "1            0.0  26.279602        0.0    0.000000    0.000000    0.000000   \n",
       "2            0.0  26.913395        0.0    0.000000    0.000000    7.915159   \n",
       "3            0.0  29.767157        0.0    0.000000    0.000000    0.000000   \n",
       "4            0.0  24.455626        0.0    0.000000    0.000000    0.000000   \n",
       "...          ...        ...        ...         ...         ...         ...   \n",
       "68597        0.0   0.000000        0.0    0.716291    7.768239    5.032241   \n",
       "68598        0.0   0.000000        0.0    0.974198    8.142911    5.304706   \n",
       "68599        0.0   0.000000        0.0    0.778554    7.931119    5.137025   \n",
       "68600        0.0   0.000000        0.0    0.799744    7.906666    5.131299   \n",
       "68601        0.0   0.000000        0.0    1.005248    6.179784    3.833471   \n",
       "\n",
       "       comb_cnn_3  \n",
       "0       58.904250  \n",
       "1       69.381874  \n",
       "2       82.243230  \n",
       "3      117.579170  \n",
       "4       61.207540  \n",
       "...           ...  \n",
       "68597    8.482945  \n",
       "68598    7.986442  \n",
       "68599    8.463214  \n",
       "68600    8.344973  \n",
       "68601    5.429558  \n",
       "\n",
       "[68602 rows x 36 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_imp.loc[df_imp['surface_area [m^2/g]']==0, 'surface_area [m^2/g]'] = np.NaN\n",
    "df_imp.loc[df_imp['surface_area [m^2/g]']==-1, 'surface_area [m^2/g]'] = np.NaN\n",
    "df_imp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "3914d053",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.685723Z",
     "start_time": "2021-10-17T09:25:42.655722Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 68602 entries, 0 to 68601\n",
      "Data columns (total 36 columns):\n",
      " #   Column                                         Non-Null Count  Dtype  \n",
      "---  ------                                         --------------  -----  \n",
      " 0   MOFname                                        68602 non-null  object \n",
      " 1   volume [A^3]                                   68602 non-null  float64\n",
      " 2   volume [cm^3/g]                                68602 non-null  float64\n",
      " 3   density [g/cm^3]                               68602 non-null  float64\n",
      " 4   weight [u]                                     68602 non-null  float64\n",
      " 5   surface_area [m^2/g]                           54067 non-null  float64\n",
      " 6   void_fraction                                  68602 non-null  float64\n",
      " 7   void_volume [cm^3/g]                           68602 non-null  float64\n",
      " 8   metal_linker                                   68602 non-null  int64  \n",
      " 9   organic_linker1                                68602 non-null  int64  \n",
      " 10  organic_linker2                                68602 non-null  int64  \n",
      " 11  topology                                       68602 non-null  object \n",
      " 12  CO2/N2_selectivity                             68602 non-null  float64\n",
      " 13  heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]  66519 non-null  float64\n",
      " 14  n_atoms                                        68602 non-null  int64  \n",
      " 15  mol_avg_mass                                   68602 non-null  float64\n",
      " 16  charges                                        68602 non-null  int64  \n",
      " 17  mol_avg_radius                                 68602 non-null  float64\n",
      " 18  atoms_volume                                   68602 non-null  float64\n",
      " 19  atoms_area                                     68602 non-null  float64\n",
      " 20  CO2_working_capacity [mL/g]                    68602 non-null  float64\n",
      " 21  di_cnn_1                                       68602 non-null  float64\n",
      " 22  di_cnn_2                                       68602 non-null  float64\n",
      " 23  di_cnn_4                                       68602 non-null  float64\n",
      " 24  di_cnn_5                                       68602 non-null  float64\n",
      " 25  di_cnn_6                                       68602 non-null  float64\n",
      " 26  di_cnn_8                                       68602 non-null  float64\n",
      " 27  di_cnn_10                                      68602 non-null  float64\n",
      " 28  di_cnn_12                                      68602 non-null  float64\n",
      " 29  di_cnn_13                                      68602 non-null  float64\n",
      " 30  di_cnn_14                                      68602 non-null  float64\n",
      " 31  di_cnn_15                                      68602 non-null  float64\n",
      " 32  comb_cnn_0                                     68602 non-null  float64\n",
      " 33  comb_cnn_1                                     68602 non-null  float64\n",
      " 34  comb_cnn_2                                     68602 non-null  float64\n",
      " 35  comb_cnn_3                                     68602 non-null  float64\n",
      "dtypes: float64(29), int64(5), object(2)\n",
      "memory usage: 18.8+ MB\n"
     ]
    }
   ],
   "source": [
    "df_imp.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "95106c61",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.715721Z",
     "start_time": "2021-10-17T09:25:42.687723Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>volume [cm^3/g]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>CO2/N2_selectivity</th>\n",
       "      <th>n_atoms</th>\n",
       "      <th>charges</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.768329</td>\n",
       "      <td>875.240600</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.060700</td>\n",
       "      <td>22.864166</td>\n",
       "      <td>75</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.754097</td>\n",
       "      <td>2211.697211</td>\n",
       "      <td>603.61</td>\n",
       "      <td>0.104000</td>\n",
       "      <td>33.616780</td>\n",
       "      <td>194</td>\n",
       "      <td>250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.848280</td>\n",
       "      <td>773.687960</td>\n",
       "      <td>788.50</td>\n",
       "      <td>0.126200</td>\n",
       "      <td>19.263726</td>\n",
       "      <td>82</td>\n",
       "      <td>120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.017907</td>\n",
       "      <td>1304.638720</td>\n",
       "      <td>1441.53</td>\n",
       "      <td>0.222000</td>\n",
       "      <td>25.701377</td>\n",
       "      <td>112</td>\n",
       "      <td>204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.759867</td>\n",
       "      <td>901.736120</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.059100</td>\n",
       "      <td>30.001838</td>\n",
       "      <td>94</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>0.714398</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>24.131770</td>\n",
       "      <td>119</td>\n",
       "      <td>216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>0.607603</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.006732</td>\n",
       "      <td>6.071818</td>\n",
       "      <td>126</td>\n",
       "      <td>126</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>0.625575</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>9.876134</td>\n",
       "      <td>204</td>\n",
       "      <td>366</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>0.659602</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>5.285051</td>\n",
       "      <td>364</td>\n",
       "      <td>652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>0.462591</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.007443</td>\n",
       "      <td>2.621272</td>\n",
       "      <td>116</td>\n",
       "      <td>222</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       volume [cm^3/g]   weight [u]  surface_area [m^2/g]  \\\n",
       "0             0.768329   875.240600                   NaN   \n",
       "1             0.754097  2211.697211                603.61   \n",
       "2             0.848280   773.687960                788.50   \n",
       "3             1.017907  1304.638720               1441.53   \n",
       "4             0.759867   901.736120                   NaN   \n",
       "...                ...          ...                   ...   \n",
       "68597         0.714398  1001.700216                   NaN   \n",
       "68598         0.607603  1493.296496                   NaN   \n",
       "68599         0.625575  1959.518320                   NaN   \n",
       "68600         0.659602  3638.677280                   NaN   \n",
       "68601         0.462591  2071.219000                   NaN   \n",
       "\n",
       "       void_volume [cm^3/g]  CO2/N2_selectivity  n_atoms  charges  \n",
       "0                  0.060700           22.864166       75       92  \n",
       "1                  0.104000           33.616780      194      250  \n",
       "2                  0.126200           19.263726       82      120  \n",
       "3                  0.222000           25.701377      112      204  \n",
       "4                  0.059100           30.001838       94       90  \n",
       "...                     ...                 ...      ...      ...  \n",
       "68597              0.473400           24.131770      119      216  \n",
       "68598              0.006732            6.071818      126      126  \n",
       "68599              0.473400            9.876134      204      366  \n",
       "68600              0.473400            5.285051      364      652  \n",
       "68601              0.007443            2.621272      116      222  \n",
       "\n",
       "[68602 rows x 7 columns]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_cat = df_imp[['MOFname', 'topology']]\n",
    "df_imp = df_imp[['volume [cm^3/g]', 'weight [u]', 'surface_area [m^2/g]', 'void_volume [cm^3/g]',\n",
    "                 'CO2/N2_selectivity', 'n_atoms', 'charges']]\n",
    "df_imp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "b1b4e9bf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.730720Z",
     "start_time": "2021-10-17T09:25:42.717722Z"
    }
   },
   "outputs": [],
   "source": [
    "surface_mean = df_imp['surface_area [m^2/g]'].mean()\n",
    "surface_std = df_imp['surface_area [m^2/g]'].std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "967c9b73",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.806721Z",
     "start_time": "2021-10-17T09:25:42.735724Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "volume [cm^3/g]\n",
      "weight [u]\n",
      "surface_area [m^2/g]\n",
      "CO2/N2_selectivity\n",
      "n_atoms\n",
      "charges\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\users\\admin\\anaconda3\\envs\\chemistry-ml\\lib\\site-packages\\ipykernel_launcher.py:10: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  # Remove the CWD from sys.path while we load stuff.\n",
      "c:\\users\\admin\\anaconda3\\envs\\chemistry-ml\\lib\\site-packages\\ipykernel_launcher.py:10: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  # Remove the CWD from sys.path while we load stuff.\n",
      "c:\\users\\admin\\anaconda3\\envs\\chemistry-ml\\lib\\site-packages\\ipykernel_launcher.py:10: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  # Remove the CWD from sys.path while we load stuff.\n",
      "c:\\users\\admin\\anaconda3\\envs\\chemistry-ml\\lib\\site-packages\\ipykernel_launcher.py:10: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  # Remove the CWD from sys.path while we load stuff.\n",
      "c:\\users\\admin\\anaconda3\\envs\\chemistry-ml\\lib\\site-packages\\ipykernel_launcher.py:10: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  # Remove the CWD from sys.path while we load stuff.\n",
      "c:\\users\\admin\\anaconda3\\envs\\chemistry-ml\\lib\\site-packages\\ipykernel_launcher.py:10: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  # Remove the CWD from sys.path while we load stuff.\n"
     ]
    }
   ],
   "source": [
    "# features to be scaled before using in imputation\n",
    "feats_to_scale = ['volume [cm^3/g]', 'weight [u]', 'surface_area [m^2/g]', 'CO2/N2_selectivity',\n",
    "                 'n_atoms', 'charges']\n",
    "\n",
    "imp_scalers = {}\n",
    "for feat in feats_to_scale:\n",
    "    print(feat)\n",
    "    scaler = RobustScaler()\n",
    "    scaler.fit(df_imp[[feat]])\n",
    "    scaled_feat = scaler.transform(df_imp[[feat]])\n",
    "    df_imp[feat] = scaled_feat\n",
    "    imp_scalers[feat] = scaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "b59da34e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:25:42.836722Z",
     "start_time": "2021-10-17T09:25:42.808725Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing values in Heat adsorption: 2083, Inf value in Heat Adsorption: 2\n",
      "0 values in void_volume: 0\n",
      "0 values in void_fraction: 0, -1 value: 0\n",
      "0 values in surface_area: 14479, -1 value: 56\n"
     ]
    }
   ],
   "source": [
    "check_all_problems(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "98755ed3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.258435Z",
     "start_time": "2021-10-17T09:25:42.838722Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(68602, 7)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from impyute.imputation.cs import mice\n",
    "\n",
    "# start the MICE training\n",
    "imputed_surface=mice(df_imp.values)\n",
    "imputed_surface.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "308c27a5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.274431Z",
     "start_time": "2021-10-17T09:26:31.260404Z"
    }
   },
   "outputs": [],
   "source": [
    "col_list = df_imp.columns.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "5bd5ca6a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.305403Z",
     "start_time": "2021-10-17T09:26:31.276402Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>volume [cm^3/g]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>CO2/N2_selectivity</th>\n",
       "      <th>n_atoms</th>\n",
       "      <th>charges</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.584699</td>\n",
       "      <td>-0.383085</td>\n",
       "      <td>-0.481798</td>\n",
       "      <td>0.060700</td>\n",
       "      <td>0.157631</td>\n",
       "      <td>-0.406977</td>\n",
       "      <td>-0.539474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.615547</td>\n",
       "      <td>0.949117</td>\n",
       "      <td>-0.846039</td>\n",
       "      <td>0.104000</td>\n",
       "      <td>0.691708</td>\n",
       "      <td>0.976744</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.411402</td>\n",
       "      <td>-0.484315</td>\n",
       "      <td>-0.727178</td>\n",
       "      <td>0.126200</td>\n",
       "      <td>-0.021201</td>\n",
       "      <td>-0.325581</td>\n",
       "      <td>-0.355263</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.043730</td>\n",
       "      <td>0.044946</td>\n",
       "      <td>-0.307363</td>\n",
       "      <td>0.222000</td>\n",
       "      <td>0.298554</td>\n",
       "      <td>0.023256</td>\n",
       "      <td>0.197368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.603040</td>\n",
       "      <td>-0.356674</td>\n",
       "      <td>-0.632685</td>\n",
       "      <td>0.059100</td>\n",
       "      <td>0.512156</td>\n",
       "      <td>-0.186047</td>\n",
       "      <td>-0.552632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>-0.701596</td>\n",
       "      <td>-0.257028</td>\n",
       "      <td>-1.432939</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>0.220592</td>\n",
       "      <td>0.104651</td>\n",
       "      <td>0.276316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>-0.933076</td>\n",
       "      <td>0.233003</td>\n",
       "      <td>-0.643068</td>\n",
       "      <td>0.006732</td>\n",
       "      <td>-0.676437</td>\n",
       "      <td>0.186047</td>\n",
       "      <td>-0.315789</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>-0.894122</td>\n",
       "      <td>0.697741</td>\n",
       "      <td>-1.536209</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>-0.487479</td>\n",
       "      <td>1.093023</td>\n",
       "      <td>1.263158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>-0.820367</td>\n",
       "      <td>2.371556</td>\n",
       "      <td>-1.472261</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>-0.715515</td>\n",
       "      <td>2.953488</td>\n",
       "      <td>3.144737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>-1.247394</td>\n",
       "      <td>0.809086</td>\n",
       "      <td>-0.809562</td>\n",
       "      <td>0.007443</td>\n",
       "      <td>-0.847824</td>\n",
       "      <td>0.069767</td>\n",
       "      <td>0.315789</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       volume [cm^3/g]  weight [u]  surface_area [m^2/g]  \\\n",
       "0            -0.584699   -0.383085             -0.481798   \n",
       "1            -0.615547    0.949117             -0.846039   \n",
       "2            -0.411402   -0.484315             -0.727178   \n",
       "3            -0.043730    0.044946             -0.307363   \n",
       "4            -0.603040   -0.356674             -0.632685   \n",
       "...                ...         ...                   ...   \n",
       "68597        -0.701596   -0.257028             -1.432939   \n",
       "68598        -0.933076    0.233003             -0.643068   \n",
       "68599        -0.894122    0.697741             -1.536209   \n",
       "68600        -0.820367    2.371556             -1.472261   \n",
       "68601        -1.247394    0.809086             -0.809562   \n",
       "\n",
       "       void_volume [cm^3/g]  CO2/N2_selectivity   n_atoms   charges  \n",
       "0                  0.060700            0.157631 -0.406977 -0.539474  \n",
       "1                  0.104000            0.691708  0.976744  0.500000  \n",
       "2                  0.126200           -0.021201 -0.325581 -0.355263  \n",
       "3                  0.222000            0.298554  0.023256  0.197368  \n",
       "4                  0.059100            0.512156 -0.186047 -0.552632  \n",
       "...                     ...                 ...       ...       ...  \n",
       "68597              0.473400            0.220592  0.104651  0.276316  \n",
       "68598              0.006732           -0.676437  0.186047 -0.315789  \n",
       "68599              0.473400           -0.487479  1.093023  1.263158  \n",
       "68600              0.473400           -0.715515  2.953488  3.144737  \n",
       "68601              0.007443           -0.847824  0.069767  0.315789  \n",
       "\n",
       "[68602 rows x 7 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_impute = pd.DataFrame(imputed_surface, columns = col_list)\n",
    "df_impute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "1443e724",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.337400Z",
     "start_time": "2021-10-17T09:26:31.307403Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>volume [cm^3/g]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>CO2/N2_selectivity</th>\n",
       "      <th>n_atoms</th>\n",
       "      <th>charges</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.584699</td>\n",
       "      <td>-0.383085</td>\n",
       "      <td>1170.192841</td>\n",
       "      <td>0.060700</td>\n",
       "      <td>0.157631</td>\n",
       "      <td>-0.406977</td>\n",
       "      <td>-0.539474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.615547</td>\n",
       "      <td>0.949117</td>\n",
       "      <td>603.610000</td>\n",
       "      <td>0.104000</td>\n",
       "      <td>0.691708</td>\n",
       "      <td>0.976744</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.411402</td>\n",
       "      <td>-0.484315</td>\n",
       "      <td>788.500000</td>\n",
       "      <td>0.126200</td>\n",
       "      <td>-0.021201</td>\n",
       "      <td>-0.325581</td>\n",
       "      <td>-0.355263</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.043730</td>\n",
       "      <td>0.044946</td>\n",
       "      <td>1441.530000</td>\n",
       "      <td>0.222000</td>\n",
       "      <td>0.298554</td>\n",
       "      <td>0.023256</td>\n",
       "      <td>0.197368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.603040</td>\n",
       "      <td>-0.356674</td>\n",
       "      <td>935.486251</td>\n",
       "      <td>0.059100</td>\n",
       "      <td>0.512156</td>\n",
       "      <td>-0.186047</td>\n",
       "      <td>-0.552632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>-0.701596</td>\n",
       "      <td>-0.257028</td>\n",
       "      <td>-309.324898</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>0.220592</td>\n",
       "      <td>0.104651</td>\n",
       "      <td>0.276316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>-0.933076</td>\n",
       "      <td>0.233003</td>\n",
       "      <td>919.334794</td>\n",
       "      <td>0.006732</td>\n",
       "      <td>-0.676437</td>\n",
       "      <td>0.186047</td>\n",
       "      <td>-0.315789</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>-0.894122</td>\n",
       "      <td>0.697741</td>\n",
       "      <td>-469.964067</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>-0.487479</td>\n",
       "      <td>1.093023</td>\n",
       "      <td>1.263158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>-0.820367</td>\n",
       "      <td>2.371556</td>\n",
       "      <td>-370.490957</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>-0.715515</td>\n",
       "      <td>2.953488</td>\n",
       "      <td>3.144737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>-1.247394</td>\n",
       "      <td>0.809086</td>\n",
       "      <td>660.350715</td>\n",
       "      <td>0.007443</td>\n",
       "      <td>-0.847824</td>\n",
       "      <td>0.069767</td>\n",
       "      <td>0.315789</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       volume [cm^3/g]  weight [u]  surface_area [m^2/g]  \\\n",
       "0            -0.584699   -0.383085           1170.192841   \n",
       "1            -0.615547    0.949117            603.610000   \n",
       "2            -0.411402   -0.484315            788.500000   \n",
       "3            -0.043730    0.044946           1441.530000   \n",
       "4            -0.603040   -0.356674            935.486251   \n",
       "...                ...         ...                   ...   \n",
       "68597        -0.701596   -0.257028           -309.324898   \n",
       "68598        -0.933076    0.233003            919.334794   \n",
       "68599        -0.894122    0.697741           -469.964067   \n",
       "68600        -0.820367    2.371556           -370.490957   \n",
       "68601        -1.247394    0.809086            660.350715   \n",
       "\n",
       "       void_volume [cm^3/g]  CO2/N2_selectivity   n_atoms   charges  \n",
       "0                  0.060700            0.157631 -0.406977 -0.539474  \n",
       "1                  0.104000            0.691708  0.976744  0.500000  \n",
       "2                  0.126200           -0.021201 -0.325581 -0.355263  \n",
       "3                  0.222000            0.298554  0.023256  0.197368  \n",
       "4                  0.059100            0.512156 -0.186047 -0.552632  \n",
       "...                     ...                 ...       ...       ...  \n",
       "68597              0.473400            0.220592  0.104651  0.276316  \n",
       "68598              0.006732           -0.676437  0.186047 -0.315789  \n",
       "68599              0.473400           -0.487479  1.093023  1.263158  \n",
       "68600              0.473400           -0.715515  2.953488  3.144737  \n",
       "68601              0.007443           -0.847824  0.069767  0.315789  \n",
       "\n",
       "[68602 rows x 7 columns]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# transform the value back before feeding back into the main data\n",
    "df_impute['surface_area [m^2/g]'] = imp_scalers['surface_area [m^2/g]'].inverse_transform(df_impute[['surface_area [m^2/g]']])\n",
    "df_impute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "340208f6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.367404Z",
     "start_time": "2021-10-17T09:26:31.339404Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>volume [cm^3/g]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>CO2/N2_selectivity</th>\n",
       "      <th>n_atoms</th>\n",
       "      <th>charges</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.584699</td>\n",
       "      <td>-0.383085</td>\n",
       "      <td>1170.192841</td>\n",
       "      <td>0.060700</td>\n",
       "      <td>0.157631</td>\n",
       "      <td>-0.406977</td>\n",
       "      <td>-0.539474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.615547</td>\n",
       "      <td>0.949117</td>\n",
       "      <td>603.610000</td>\n",
       "      <td>0.104000</td>\n",
       "      <td>0.691708</td>\n",
       "      <td>0.976744</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.411402</td>\n",
       "      <td>-0.484315</td>\n",
       "      <td>788.500000</td>\n",
       "      <td>0.126200</td>\n",
       "      <td>-0.021201</td>\n",
       "      <td>-0.325581</td>\n",
       "      <td>-0.355263</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.043730</td>\n",
       "      <td>0.044946</td>\n",
       "      <td>1441.530000</td>\n",
       "      <td>0.222000</td>\n",
       "      <td>0.298554</td>\n",
       "      <td>0.023256</td>\n",
       "      <td>0.197368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.603040</td>\n",
       "      <td>-0.356674</td>\n",
       "      <td>935.486251</td>\n",
       "      <td>0.059100</td>\n",
       "      <td>0.512156</td>\n",
       "      <td>-0.186047</td>\n",
       "      <td>-0.552632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>-0.701596</td>\n",
       "      <td>-0.257028</td>\n",
       "      <td>-309.324898</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>0.220592</td>\n",
       "      <td>0.104651</td>\n",
       "      <td>0.276316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>-0.933076</td>\n",
       "      <td>0.233003</td>\n",
       "      <td>919.334794</td>\n",
       "      <td>0.006732</td>\n",
       "      <td>-0.676437</td>\n",
       "      <td>0.186047</td>\n",
       "      <td>-0.315789</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>-0.894122</td>\n",
       "      <td>0.697741</td>\n",
       "      <td>-469.964067</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>-0.487479</td>\n",
       "      <td>1.093023</td>\n",
       "      <td>1.263158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>-0.820367</td>\n",
       "      <td>2.371556</td>\n",
       "      <td>-370.490957</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>-0.715515</td>\n",
       "      <td>2.953488</td>\n",
       "      <td>3.144737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>-1.247394</td>\n",
       "      <td>0.809086</td>\n",
       "      <td>660.350715</td>\n",
       "      <td>0.007443</td>\n",
       "      <td>-0.847824</td>\n",
       "      <td>0.069767</td>\n",
       "      <td>0.315789</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       volume [cm^3/g]  weight [u]  surface_area [m^2/g]  \\\n",
       "0            -0.584699   -0.383085           1170.192841   \n",
       "1            -0.615547    0.949117            603.610000   \n",
       "2            -0.411402   -0.484315            788.500000   \n",
       "3            -0.043730    0.044946           1441.530000   \n",
       "4            -0.603040   -0.356674            935.486251   \n",
       "...                ...         ...                   ...   \n",
       "68597        -0.701596   -0.257028           -309.324898   \n",
       "68598        -0.933076    0.233003            919.334794   \n",
       "68599        -0.894122    0.697741           -469.964067   \n",
       "68600        -0.820367    2.371556           -370.490957   \n",
       "68601        -1.247394    0.809086            660.350715   \n",
       "\n",
       "       void_volume [cm^3/g]  CO2/N2_selectivity   n_atoms   charges  \n",
       "0                  0.060700            0.157631 -0.406977 -0.539474  \n",
       "1                  0.104000            0.691708  0.976744  0.500000  \n",
       "2                  0.126200           -0.021201 -0.325581 -0.355263  \n",
       "3                  0.222000            0.298554  0.023256  0.197368  \n",
       "4                  0.059100            0.512156 -0.186047 -0.552632  \n",
       "...                     ...                 ...       ...       ...  \n",
       "68597              0.473400            0.220592  0.104651  0.276316  \n",
       "68598              0.006732           -0.676437  0.186047 -0.315789  \n",
       "68599              0.473400           -0.487479  1.093023  1.263158  \n",
       "68600              0.473400           -0.715515  2.953488  3.144737  \n",
       "68601              0.007443           -0.847824  0.069767  0.315789  \n",
       "\n",
       "[68602 rows x 7 columns]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_impute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "95c8f6e0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.382403Z",
     "start_time": "2021-10-17T09:26:31.369403Z"
    }
   },
   "outputs": [],
   "source": [
    "df = df.reset_index(drop=True)\n",
    "df_impute = df_impute.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "7e3098ef",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.444402Z",
     "start_time": "2021-10-17T09:26:31.384404Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MOFname</th>\n",
       "      <th>volume [A^3]</th>\n",
       "      <th>volume [cm^3/g]</th>\n",
       "      <th>density [g/cm^3]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>metal_linker</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>...</th>\n",
       "      <th>di_cnn_8</th>\n",
       "      <th>di_cnn_10</th>\n",
       "      <th>di_cnn_12</th>\n",
       "      <th>di_cnn_13</th>\n",
       "      <th>di_cnn_14</th>\n",
       "      <th>di_cnn_15</th>\n",
       "      <th>comb_cnn_0</th>\n",
       "      <th>comb_cnn_1</th>\n",
       "      <th>comb_cnn_2</th>\n",
       "      <th>comb_cnn_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mof_unit_1</td>\n",
       "      <td>1116.667429</td>\n",
       "      <td>0.768329</td>\n",
       "      <td>1.301526</td>\n",
       "      <td>875.240600</td>\n",
       "      <td>1170.192841</td>\n",
       "      <td>0.078990</td>\n",
       "      <td>0.060700</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.025457</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>58.904250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mof_unit_2</td>\n",
       "      <td>2769.503842</td>\n",
       "      <td>0.754097</td>\n",
       "      <td>1.326090</td>\n",
       "      <td>2211.697211</td>\n",
       "      <td>603.610000</td>\n",
       "      <td>0.137940</td>\n",
       "      <td>0.104000</td>\n",
       "      <td>10</td>\n",
       "      <td>44</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.279602</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>69.381874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>mof_unit_3</td>\n",
       "      <td>1089.818728</td>\n",
       "      <td>0.848280</td>\n",
       "      <td>1.178856</td>\n",
       "      <td>773.687960</td>\n",
       "      <td>788.500000</td>\n",
       "      <td>0.148740</td>\n",
       "      <td>0.126200</td>\n",
       "      <td>2</td>\n",
       "      <td>22</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.338663</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.913395</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.915159</td>\n",
       "      <td>82.243230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>mof_unit_4</td>\n",
       "      <td>2205.198301</td>\n",
       "      <td>1.017907</td>\n",
       "      <td>0.982408</td>\n",
       "      <td>1304.638720</td>\n",
       "      <td>1441.530000</td>\n",
       "      <td>0.218140</td>\n",
       "      <td>0.222000</td>\n",
       "      <td>9</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>20.764702</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.767157</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>117.579170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>mof_unit_5</td>\n",
       "      <td>1137.800963</td>\n",
       "      <td>0.759867</td>\n",
       "      <td>1.316020</td>\n",
       "      <td>901.736120</td>\n",
       "      <td>935.486251</td>\n",
       "      <td>0.077780</td>\n",
       "      <td>0.059100</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.455626</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>61.207540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>mof_unit_68609</td>\n",
       "      <td>1188.302573</td>\n",
       "      <td>0.714398</td>\n",
       "      <td>1.399781</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>-309.324898</td>\n",
       "      <td>0.662656</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>2.536157</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.716291</td>\n",
       "      <td>7.768239</td>\n",
       "      <td>5.032241</td>\n",
       "      <td>8.482945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>mof_unit_68610</td>\n",
       "      <td>1506.660363</td>\n",
       "      <td>0.607603</td>\n",
       "      <td>1.645811</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>919.334794</td>\n",
       "      <td>0.011080</td>\n",
       "      <td>0.006732</td>\n",
       "      <td>10</td>\n",
       "      <td>42</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.974198</td>\n",
       "      <td>8.142911</td>\n",
       "      <td>5.304706</td>\n",
       "      <td>7.986442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>mof_unit_68611</td>\n",
       "      <td>2035.532738</td>\n",
       "      <td>0.625575</td>\n",
       "      <td>1.598529</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>-469.964067</td>\n",
       "      <td>0.756744</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.778554</td>\n",
       "      <td>7.931119</td>\n",
       "      <td>5.137025</td>\n",
       "      <td>8.463214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>mof_unit_68612</td>\n",
       "      <td>3985.426053</td>\n",
       "      <td>0.659602</td>\n",
       "      <td>1.516066</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>-370.490957</td>\n",
       "      <td>0.717706</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.799744</td>\n",
       "      <td>7.906666</td>\n",
       "      <td>5.131299</td>\n",
       "      <td>8.344973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>mof_unit_68613</td>\n",
       "      <td>1591.009408</td>\n",
       "      <td>0.462591</td>\n",
       "      <td>2.161736</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>660.350715</td>\n",
       "      <td>0.016090</td>\n",
       "      <td>0.007443</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.005248</td>\n",
       "      <td>6.179784</td>\n",
       "      <td>3.833471</td>\n",
       "      <td>5.429558</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              MOFname  volume [A^3]  volume [cm^3/g]  density [g/cm^3]  \\\n",
       "0          mof_unit_1   1116.667429         0.768329          1.301526   \n",
       "1          mof_unit_2   2769.503842         0.754097          1.326090   \n",
       "2          mof_unit_3   1089.818728         0.848280          1.178856   \n",
       "3          mof_unit_4   2205.198301         1.017907          0.982408   \n",
       "4          mof_unit_5   1137.800963         0.759867          1.316020   \n",
       "...               ...           ...              ...               ...   \n",
       "68597  mof_unit_68609   1188.302573         0.714398          1.399781   \n",
       "68598  mof_unit_68610   1506.660363         0.607603          1.645811   \n",
       "68599  mof_unit_68611   2035.532738         0.625575          1.598529   \n",
       "68600  mof_unit_68612   3985.426053         0.659602          1.516066   \n",
       "68601  mof_unit_68613   1591.009408         0.462591          2.161736   \n",
       "\n",
       "        weight [u]  surface_area [m^2/g]  void_fraction  void_volume [cm^3/g]  \\\n",
       "0       875.240600           1170.192841       0.078990              0.060700   \n",
       "1      2211.697211            603.610000       0.137940              0.104000   \n",
       "2       773.687960            788.500000       0.148740              0.126200   \n",
       "3      1304.638720           1441.530000       0.218140              0.222000   \n",
       "4       901.736120            935.486251       0.077780              0.059100   \n",
       "...            ...                   ...            ...                   ...   \n",
       "68597  1001.700216           -309.324898       0.662656              0.473400   \n",
       "68598  1493.296496            919.334794       0.011080              0.006732   \n",
       "68599  1959.518320           -469.964067       0.756744              0.473400   \n",
       "68600  3638.677280           -370.490957       0.717706              0.473400   \n",
       "68601  2071.219000            660.350715       0.016090              0.007443   \n",
       "\n",
       "       metal_linker  organic_linker1  ...  di_cnn_8  di_cnn_10  di_cnn_12  \\\n",
       "0                 3                4  ...  0.000000   0.000000        0.0   \n",
       "1                10               44  ...  0.000000   0.000000        0.0   \n",
       "2                 2               22  ...  0.000000   5.338663        0.0   \n",
       "3                 9               17  ...  0.000000  20.764702        0.0   \n",
       "4                 2                1  ...  0.000000   0.000000        0.0   \n",
       "...             ...              ...  ...       ...        ...        ...   \n",
       "68597             3                4  ...  2.536157   0.000000        0.0   \n",
       "68598            10               42  ...  0.000000   0.000000        0.0   \n",
       "68599             4               14  ...  0.000000   0.000000        0.0   \n",
       "68600             4                4  ...  0.000000   0.000000        0.0   \n",
       "68601             2                9  ...  0.000000   0.000000        0.0   \n",
       "\n",
       "       di_cnn_13  di_cnn_14  di_cnn_15  comb_cnn_0  comb_cnn_1  comb_cnn_2  \\\n",
       "0            0.0  24.025457        0.0    0.000000    0.000000    0.000000   \n",
       "1            0.0  26.279602        0.0    0.000000    0.000000    0.000000   \n",
       "2            0.0  26.913395        0.0    0.000000    0.000000    7.915159   \n",
       "3            0.0  29.767157        0.0    0.000000    0.000000    0.000000   \n",
       "4            0.0  24.455626        0.0    0.000000    0.000000    0.000000   \n",
       "...          ...        ...        ...         ...         ...         ...   \n",
       "68597        0.0   0.000000        0.0    0.716291    7.768239    5.032241   \n",
       "68598        0.0   0.000000        0.0    0.974198    8.142911    5.304706   \n",
       "68599        0.0   0.000000        0.0    0.778554    7.931119    5.137025   \n",
       "68600        0.0   0.000000        0.0    0.799744    7.906666    5.131299   \n",
       "68601        0.0   0.000000        0.0    1.005248    6.179784    3.833471   \n",
       "\n",
       "       comb_cnn_3  \n",
       "0       58.904250  \n",
       "1       69.381874  \n",
       "2       82.243230  \n",
       "3      117.579170  \n",
       "4       61.207540  \n",
       "...           ...  \n",
       "68597    8.482945  \n",
       "68598    7.986442  \n",
       "68599    8.463214  \n",
       "68600    8.344973  \n",
       "68601    5.429558  \n",
       "\n",
       "[68602 rows x 36 columns]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['surface_area [m^2/g]'] = df_impute['surface_area [m^2/g]']\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "7e22b450",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.459406Z",
     "start_time": "2021-10-17T09:26:31.446403Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing values in Heat adsorption: 2083, Inf value in Heat Adsorption: 2\n",
      "0 values in void_volume: 0\n",
      "0 values in void_fraction: 0, -1 value: 0\n",
      "0 values in surface_area: 0, -1 value: 0\n"
     ]
    }
   ],
   "source": [
    "check_all_problems(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4f40495",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-12T01:36:29.964035Z",
     "start_time": "2021-10-12T01:36:29.947067Z"
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c452c914",
   "metadata": {},
   "source": [
    "### Find MIN of heat absorption in inf value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "2cf9bced",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.490403Z",
     "start_time": "2021-10-17T09:26:31.462406Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.612299"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "heat_min = df[(df['heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]'] != np.inf) & (~df['heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]'].isnull())]['heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]'].min()\n",
    "heat_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "1769c6dc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.505404Z",
     "start_time": "2021-10-17T09:26:31.493404Z"
    }
   },
   "outputs": [],
   "source": [
    "df.loc[df['heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]'] == np.inf, 'heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]'] = heat_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "8fb0600d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.521402Z",
     "start_time": "2021-10-17T09:26:31.507402Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing values in Heat adsorption: 2083, Inf value in Heat Adsorption: 0\n",
      "0 values in void_volume: 0\n",
      "0 values in void_fraction: 0, -1 value: 0\n",
      "0 values in surface_area: 0, -1 value: 0\n"
     ]
    }
   ],
   "source": [
    "check_all_problems(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "602e3ca1",
   "metadata": {},
   "source": [
    "# Let's try fill heat with min\n",
    "from testing, fill with min is **better than mean, 0, q0.005 and drop**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "1e73b631",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.536401Z",
     "start_time": "2021-10-17T09:26:31.524406Z"
    }
   },
   "outputs": [],
   "source": [
    "heat_min = df['heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]'].min()\n",
    "df['heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]'] = df['heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]'].fillna(heat_min)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "7b5b2578",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.551405Z",
     "start_time": "2021-10-17T09:26:31.537402Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing values in Heat adsorption: 0, Inf value in Heat Adsorption: 0\n",
      "0 values in void_volume: 0\n",
      "0 values in void_fraction: 0, -1 value: 0\n",
      "0 values in surface_area: 0, -1 value: 0\n"
     ]
    }
   ],
   "source": [
    "check_all_problems(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb082ef1",
   "metadata": {},
   "source": [
    "### specific heat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "5702066b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.567406Z",
     "start_time": "2021-10-17T09:26:31.553403Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        0.499411\n",
       "1        0.208154\n",
       "2        0.528491\n",
       "3        0.305615\n",
       "4        0.462737\n",
       "           ...   \n",
       "68597    0.103676\n",
       "68598    0.069545\n",
       "68599    0.052999\n",
       "68600    0.028541\n",
       "68601    0.050140\n",
       "Name: specific_heat, Length: 68602, dtype: float64"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['specific_heat'] = (df['heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]']*4.1868*1000)/(df['weight [u]']*65)\n",
    "df['specific_heat']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "844127be",
   "metadata": {},
   "source": [
    "## Prepare X, y for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "40e15f97",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.597400Z",
     "start_time": "2021-10-17T09:26:31.569406Z"
    }
   },
   "outputs": [],
   "source": [
    "# drop name and those columns we used\n",
    "X = df.drop(['MOFname','CO2_working_capacity [mL/g]'], axis = 1)\n",
    "y = df['CO2_working_capacity [mL/g]']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "fcbe1e66",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.658404Z",
     "start_time": "2021-10-17T09:26:31.599405Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>volume [A^3]</th>\n",
       "      <th>volume [cm^3/g]</th>\n",
       "      <th>density [g/cm^3]</th>\n",
       "      <th>weight [u]</th>\n",
       "      <th>surface_area [m^2/g]</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume [cm^3/g]</th>\n",
       "      <th>metal_linker</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>organic_linker2</th>\n",
       "      <th>...</th>\n",
       "      <th>di_cnn_10</th>\n",
       "      <th>di_cnn_12</th>\n",
       "      <th>di_cnn_13</th>\n",
       "      <th>di_cnn_14</th>\n",
       "      <th>di_cnn_15</th>\n",
       "      <th>comb_cnn_0</th>\n",
       "      <th>comb_cnn_1</th>\n",
       "      <th>comb_cnn_2</th>\n",
       "      <th>comb_cnn_3</th>\n",
       "      <th>specific_heat</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1116.667429</td>\n",
       "      <td>0.768329</td>\n",
       "      <td>1.301526</td>\n",
       "      <td>875.240600</td>\n",
       "      <td>1170.192841</td>\n",
       "      <td>0.078990</td>\n",
       "      <td>0.060700</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.025457</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>58.904250</td>\n",
       "      <td>0.499411</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2769.503842</td>\n",
       "      <td>0.754097</td>\n",
       "      <td>1.326090</td>\n",
       "      <td>2211.697211</td>\n",
       "      <td>603.610000</td>\n",
       "      <td>0.137940</td>\n",
       "      <td>0.104000</td>\n",
       "      <td>10</td>\n",
       "      <td>44</td>\n",
       "      <td>57</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.279602</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>69.381874</td>\n",
       "      <td>0.208154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1089.818728</td>\n",
       "      <td>0.848280</td>\n",
       "      <td>1.178856</td>\n",
       "      <td>773.687960</td>\n",
       "      <td>788.500000</td>\n",
       "      <td>0.148740</td>\n",
       "      <td>0.126200</td>\n",
       "      <td>2</td>\n",
       "      <td>22</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>5.338663</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.913395</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.915159</td>\n",
       "      <td>82.243230</td>\n",
       "      <td>0.528491</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2205.198301</td>\n",
       "      <td>1.017907</td>\n",
       "      <td>0.982408</td>\n",
       "      <td>1304.638720</td>\n",
       "      <td>1441.530000</td>\n",
       "      <td>0.218140</td>\n",
       "      <td>0.222000</td>\n",
       "      <td>9</td>\n",
       "      <td>17</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>20.764702</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.767157</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>117.579170</td>\n",
       "      <td>0.305615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1137.800963</td>\n",
       "      <td>0.759867</td>\n",
       "      <td>1.316020</td>\n",
       "      <td>901.736120</td>\n",
       "      <td>935.486251</td>\n",
       "      <td>0.077780</td>\n",
       "      <td>0.059100</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.455626</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>61.207540</td>\n",
       "      <td>0.462737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>1188.302573</td>\n",
       "      <td>0.714398</td>\n",
       "      <td>1.399781</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>-309.324898</td>\n",
       "      <td>0.662656</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.716291</td>\n",
       "      <td>7.768239</td>\n",
       "      <td>5.032241</td>\n",
       "      <td>8.482945</td>\n",
       "      <td>0.103676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>1506.660363</td>\n",
       "      <td>0.607603</td>\n",
       "      <td>1.645811</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>919.334794</td>\n",
       "      <td>0.011080</td>\n",
       "      <td>0.006732</td>\n",
       "      <td>10</td>\n",
       "      <td>42</td>\n",
       "      <td>46</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.974198</td>\n",
       "      <td>8.142911</td>\n",
       "      <td>5.304706</td>\n",
       "      <td>7.986442</td>\n",
       "      <td>0.069545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>2035.532738</td>\n",
       "      <td>0.625575</td>\n",
       "      <td>1.598529</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>-469.964067</td>\n",
       "      <td>0.756744</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>22</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.778554</td>\n",
       "      <td>7.931119</td>\n",
       "      <td>5.137025</td>\n",
       "      <td>8.463214</td>\n",
       "      <td>0.052999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>3985.426053</td>\n",
       "      <td>0.659602</td>\n",
       "      <td>1.516066</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>-370.490957</td>\n",
       "      <td>0.717706</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.799744</td>\n",
       "      <td>7.906666</td>\n",
       "      <td>5.131299</td>\n",
       "      <td>8.344973</td>\n",
       "      <td>0.028541</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>1591.009408</td>\n",
       "      <td>0.462591</td>\n",
       "      <td>2.161736</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>660.350715</td>\n",
       "      <td>0.016090</td>\n",
       "      <td>0.007443</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.005248</td>\n",
       "      <td>6.179784</td>\n",
       "      <td>3.833471</td>\n",
       "      <td>5.429558</td>\n",
       "      <td>0.050140</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       volume [A^3]  volume [cm^3/g]  density [g/cm^3]   weight [u]  \\\n",
       "0       1116.667429         0.768329          1.301526   875.240600   \n",
       "1       2769.503842         0.754097          1.326090  2211.697211   \n",
       "2       1089.818728         0.848280          1.178856   773.687960   \n",
       "3       2205.198301         1.017907          0.982408  1304.638720   \n",
       "4       1137.800963         0.759867          1.316020   901.736120   \n",
       "...             ...              ...               ...          ...   \n",
       "68597   1188.302573         0.714398          1.399781  1001.700216   \n",
       "68598   1506.660363         0.607603          1.645811  1493.296496   \n",
       "68599   2035.532738         0.625575          1.598529  1959.518320   \n",
       "68600   3985.426053         0.659602          1.516066  3638.677280   \n",
       "68601   1591.009408         0.462591          2.161736  2071.219000   \n",
       "\n",
       "       surface_area [m^2/g]  void_fraction  void_volume [cm^3/g]  \\\n",
       "0               1170.192841       0.078990              0.060700   \n",
       "1                603.610000       0.137940              0.104000   \n",
       "2                788.500000       0.148740              0.126200   \n",
       "3               1441.530000       0.218140              0.222000   \n",
       "4                935.486251       0.077780              0.059100   \n",
       "...                     ...            ...                   ...   \n",
       "68597           -309.324898       0.662656              0.473400   \n",
       "68598            919.334794       0.011080              0.006732   \n",
       "68599           -469.964067       0.756744              0.473400   \n",
       "68600           -370.490957       0.717706              0.473400   \n",
       "68601            660.350715       0.016090              0.007443   \n",
       "\n",
       "       metal_linker  organic_linker1  organic_linker2  ...  di_cnn_10  \\\n",
       "0                 3                4               11  ...   0.000000   \n",
       "1                10               44               57  ...   0.000000   \n",
       "2                 2               22               24  ...   5.338663   \n",
       "3                 9               17               24  ...  20.764702   \n",
       "4                 2                1               22  ...   0.000000   \n",
       "...             ...              ...              ...  ...        ...   \n",
       "68597             3                4               24  ...   0.000000   \n",
       "68598            10               42               46  ...   0.000000   \n",
       "68599             4               14               22  ...   0.000000   \n",
       "68600             4                4               15  ...   0.000000   \n",
       "68601             2                9               16  ...   0.000000   \n",
       "\n",
       "       di_cnn_12  di_cnn_13  di_cnn_14  di_cnn_15  comb_cnn_0  comb_cnn_1  \\\n",
       "0            0.0        0.0  24.025457        0.0    0.000000    0.000000   \n",
       "1            0.0        0.0  26.279602        0.0    0.000000    0.000000   \n",
       "2            0.0        0.0  26.913395        0.0    0.000000    0.000000   \n",
       "3            0.0        0.0  29.767157        0.0    0.000000    0.000000   \n",
       "4            0.0        0.0  24.455626        0.0    0.000000    0.000000   \n",
       "...          ...        ...        ...        ...         ...         ...   \n",
       "68597        0.0        0.0   0.000000        0.0    0.716291    7.768239   \n",
       "68598        0.0        0.0   0.000000        0.0    0.974198    8.142911   \n",
       "68599        0.0        0.0   0.000000        0.0    0.778554    7.931119   \n",
       "68600        0.0        0.0   0.000000        0.0    0.799744    7.906666   \n",
       "68601        0.0        0.0   0.000000        0.0    1.005248    6.179784   \n",
       "\n",
       "       comb_cnn_2  comb_cnn_3  specific_heat  \n",
       "0        0.000000   58.904250       0.499411  \n",
       "1        0.000000   69.381874       0.208154  \n",
       "2        7.915159   82.243230       0.528491  \n",
       "3        0.000000  117.579170       0.305615  \n",
       "4        0.000000   61.207540       0.462737  \n",
       "...           ...         ...            ...  \n",
       "68597    5.032241    8.482945       0.103676  \n",
       "68598    5.304706    7.986442       0.069545  \n",
       "68599    5.137025    8.463214       0.052999  \n",
       "68600    5.131299    8.344973       0.028541  \n",
       "68601    3.833471    5.429558       0.050140  \n",
       "\n",
       "[68602 rows x 35 columns]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "82dffe79",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.688402Z",
     "start_time": "2021-10-17T09:26:31.660405Z"
    }
   },
   "outputs": [],
   "source": [
    "X = X.rename(columns={'volume [A^3]':'volume', 'volume [cm^3/g]':'volume/g', 'density [g/cm^3]':'density',\n",
    "         'weight [u]':'weight', 'surface_area [m^2/g]':'surface_area', 'void_volume [cm^3/g]':'void_volume',\n",
    "         'heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]':'heat_adsorption'\n",
    "         })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "2a8096f9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.749405Z",
     "start_time": "2021-10-17T09:26:31.690404Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>volume</th>\n",
       "      <th>volume/g</th>\n",
       "      <th>density</th>\n",
       "      <th>weight</th>\n",
       "      <th>surface_area</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume</th>\n",
       "      <th>metal_linker</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>organic_linker2</th>\n",
       "      <th>...</th>\n",
       "      <th>di_cnn_10</th>\n",
       "      <th>di_cnn_12</th>\n",
       "      <th>di_cnn_13</th>\n",
       "      <th>di_cnn_14</th>\n",
       "      <th>di_cnn_15</th>\n",
       "      <th>comb_cnn_0</th>\n",
       "      <th>comb_cnn_1</th>\n",
       "      <th>comb_cnn_2</th>\n",
       "      <th>comb_cnn_3</th>\n",
       "      <th>specific_heat</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1116.667429</td>\n",
       "      <td>0.768329</td>\n",
       "      <td>1.301526</td>\n",
       "      <td>875.240600</td>\n",
       "      <td>1170.192841</td>\n",
       "      <td>0.078990</td>\n",
       "      <td>0.060700</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.025457</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>58.904250</td>\n",
       "      <td>0.499411</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2769.503842</td>\n",
       "      <td>0.754097</td>\n",
       "      <td>1.326090</td>\n",
       "      <td>2211.697211</td>\n",
       "      <td>603.610000</td>\n",
       "      <td>0.137940</td>\n",
       "      <td>0.104000</td>\n",
       "      <td>10</td>\n",
       "      <td>44</td>\n",
       "      <td>57</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.279602</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>69.381874</td>\n",
       "      <td>0.208154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1089.818728</td>\n",
       "      <td>0.848280</td>\n",
       "      <td>1.178856</td>\n",
       "      <td>773.687960</td>\n",
       "      <td>788.500000</td>\n",
       "      <td>0.148740</td>\n",
       "      <td>0.126200</td>\n",
       "      <td>2</td>\n",
       "      <td>22</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>5.338663</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>26.913395</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.915159</td>\n",
       "      <td>82.243230</td>\n",
       "      <td>0.528491</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2205.198301</td>\n",
       "      <td>1.017907</td>\n",
       "      <td>0.982408</td>\n",
       "      <td>1304.638720</td>\n",
       "      <td>1441.530000</td>\n",
       "      <td>0.218140</td>\n",
       "      <td>0.222000</td>\n",
       "      <td>9</td>\n",
       "      <td>17</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>20.764702</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.767157</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>117.579170</td>\n",
       "      <td>0.305615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1137.800963</td>\n",
       "      <td>0.759867</td>\n",
       "      <td>1.316020</td>\n",
       "      <td>901.736120</td>\n",
       "      <td>935.486251</td>\n",
       "      <td>0.077780</td>\n",
       "      <td>0.059100</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.455626</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>61.207540</td>\n",
       "      <td>0.462737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>1188.302573</td>\n",
       "      <td>0.714398</td>\n",
       "      <td>1.399781</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>-309.324898</td>\n",
       "      <td>0.662656</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.716291</td>\n",
       "      <td>7.768239</td>\n",
       "      <td>5.032241</td>\n",
       "      <td>8.482945</td>\n",
       "      <td>0.103676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>1506.660363</td>\n",
       "      <td>0.607603</td>\n",
       "      <td>1.645811</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>919.334794</td>\n",
       "      <td>0.011080</td>\n",
       "      <td>0.006732</td>\n",
       "      <td>10</td>\n",
       "      <td>42</td>\n",
       "      <td>46</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.974198</td>\n",
       "      <td>8.142911</td>\n",
       "      <td>5.304706</td>\n",
       "      <td>7.986442</td>\n",
       "      <td>0.069545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>2035.532738</td>\n",
       "      <td>0.625575</td>\n",
       "      <td>1.598529</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>-469.964067</td>\n",
       "      <td>0.756744</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>22</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.778554</td>\n",
       "      <td>7.931119</td>\n",
       "      <td>5.137025</td>\n",
       "      <td>8.463214</td>\n",
       "      <td>0.052999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>3985.426053</td>\n",
       "      <td>0.659602</td>\n",
       "      <td>1.516066</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>-370.490957</td>\n",
       "      <td>0.717706</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.799744</td>\n",
       "      <td>7.906666</td>\n",
       "      <td>5.131299</td>\n",
       "      <td>8.344973</td>\n",
       "      <td>0.028541</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>1591.009408</td>\n",
       "      <td>0.462591</td>\n",
       "      <td>2.161736</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>660.350715</td>\n",
       "      <td>0.016090</td>\n",
       "      <td>0.007443</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.005248</td>\n",
       "      <td>6.179784</td>\n",
       "      <td>3.833471</td>\n",
       "      <td>5.429558</td>\n",
       "      <td>0.050140</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            volume  volume/g   density       weight  surface_area  \\\n",
       "0      1116.667429  0.768329  1.301526   875.240600   1170.192841   \n",
       "1      2769.503842  0.754097  1.326090  2211.697211    603.610000   \n",
       "2      1089.818728  0.848280  1.178856   773.687960    788.500000   \n",
       "3      2205.198301  1.017907  0.982408  1304.638720   1441.530000   \n",
       "4      1137.800963  0.759867  1.316020   901.736120    935.486251   \n",
       "...            ...       ...       ...          ...           ...   \n",
       "68597  1188.302573  0.714398  1.399781  1001.700216   -309.324898   \n",
       "68598  1506.660363  0.607603  1.645811  1493.296496    919.334794   \n",
       "68599  2035.532738  0.625575  1.598529  1959.518320   -469.964067   \n",
       "68600  3985.426053  0.659602  1.516066  3638.677280   -370.490957   \n",
       "68601  1591.009408  0.462591  2.161736  2071.219000    660.350715   \n",
       "\n",
       "       void_fraction  void_volume  metal_linker  organic_linker1  \\\n",
       "0           0.078990     0.060700             3                4   \n",
       "1           0.137940     0.104000            10               44   \n",
       "2           0.148740     0.126200             2               22   \n",
       "3           0.218140     0.222000             9               17   \n",
       "4           0.077780     0.059100             2                1   \n",
       "...              ...          ...           ...              ...   \n",
       "68597       0.662656     0.473400             3                4   \n",
       "68598       0.011080     0.006732            10               42   \n",
       "68599       0.756744     0.473400             4               14   \n",
       "68600       0.717706     0.473400             4                4   \n",
       "68601       0.016090     0.007443             2                9   \n",
       "\n",
       "       organic_linker2  ...  di_cnn_10  di_cnn_12  di_cnn_13  di_cnn_14  \\\n",
       "0                   11  ...   0.000000        0.0        0.0  24.025457   \n",
       "1                   57  ...   0.000000        0.0        0.0  26.279602   \n",
       "2                   24  ...   5.338663        0.0        0.0  26.913395   \n",
       "3                   24  ...  20.764702        0.0        0.0  29.767157   \n",
       "4                   22  ...   0.000000        0.0        0.0  24.455626   \n",
       "...                ...  ...        ...        ...        ...        ...   \n",
       "68597               24  ...   0.000000        0.0        0.0   0.000000   \n",
       "68598               46  ...   0.000000        0.0        0.0   0.000000   \n",
       "68599               22  ...   0.000000        0.0        0.0   0.000000   \n",
       "68600               15  ...   0.000000        0.0        0.0   0.000000   \n",
       "68601               16  ...   0.000000        0.0        0.0   0.000000   \n",
       "\n",
       "       di_cnn_15  comb_cnn_0  comb_cnn_1  comb_cnn_2  comb_cnn_3  \\\n",
       "0            0.0    0.000000    0.000000    0.000000   58.904250   \n",
       "1            0.0    0.000000    0.000000    0.000000   69.381874   \n",
       "2            0.0    0.000000    0.000000    7.915159   82.243230   \n",
       "3            0.0    0.000000    0.000000    0.000000  117.579170   \n",
       "4            0.0    0.000000    0.000000    0.000000   61.207540   \n",
       "...          ...         ...         ...         ...         ...   \n",
       "68597        0.0    0.716291    7.768239    5.032241    8.482945   \n",
       "68598        0.0    0.974198    8.142911    5.304706    7.986442   \n",
       "68599        0.0    0.778554    7.931119    5.137025    8.463214   \n",
       "68600        0.0    0.799744    7.906666    5.131299    8.344973   \n",
       "68601        0.0    1.005248    6.179784    3.833471    5.429558   \n",
       "\n",
       "       specific_heat  \n",
       "0           0.499411  \n",
       "1           0.208154  \n",
       "2           0.528491  \n",
       "3           0.305615  \n",
       "4           0.462737  \n",
       "...              ...  \n",
       "68597       0.103676  \n",
       "68598       0.069545  \n",
       "68599       0.052999  \n",
       "68600       0.028541  \n",
       "68601       0.050140  \n",
       "\n",
       "[68602 rows x 35 columns]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beba569a",
   "metadata": {},
   "source": [
    "#### columns to one-hot encode\n",
    "['functional_groups', 'metal_linker', 'organic_linker1', 'organic_linker2', 'topology']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07ef5ad1",
   "metadata": {},
   "source": [
    "1) One-hot **functional_groups**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "01c23814",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.780401Z",
     "start_time": "2021-10-17T09:26:31.767403Z"
    }
   },
   "outputs": [],
   "source": [
    "# total_fg = len(fg_lb.classes_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "ac44e265",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.795402Z",
     "start_time": "2021-10-17T09:26:31.782404Z"
    }
   },
   "outputs": [],
   "source": [
    "## We comment this cuz we have already fir the binarizer on the top\n",
    "# fg_lb = LabelBinarizer()\n",
    "# fg_lb.fit(X['functional_groups'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "f99d5b6b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.810405Z",
     "start_time": "2021-10-17T09:26:31.797403Z"
    }
   },
   "outputs": [],
   "source": [
    "# encoded_fg = fg_lb.transform(X['functional_groups'])\n",
    "# encoded_fg.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "f29dfee8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.826402Z",
     "start_time": "2021-10-17T09:26:31.811402Z"
    }
   },
   "outputs": [],
   "source": [
    "# drop the original one\n",
    "# X = X.drop(['functional_groups'], axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "362ba8d9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.841412Z",
     "start_time": "2021-10-17T09:26:31.828406Z"
    }
   },
   "outputs": [],
   "source": [
    "# fill one-hot encoded functional_groups to X\n",
    "# for i in range(total_fg):\n",
    "#     X[f'functional_groups_{i}'] = encoded_fg[:, i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "6a3e49f5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.857402Z",
     "start_time": "2021-10-17T09:26:31.843401Z"
    }
   },
   "outputs": [],
   "source": [
    "# X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7bb6163",
   "metadata": {},
   "source": [
    "2) One-hot **metal_linker**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "5cd2875e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.872402Z",
     "start_time": "2021-10-17T09:26:31.859403Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "total_metal = len(metal_lb.classes_)\n",
    "total_metal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "ccdef0fb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.888404Z",
     "start_time": "2021-10-17T09:26:31.873402Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(68602, 7)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_metal = metal_lb.transform(X['metal_linker'])\n",
    "encoded_metal.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "fad0fffc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.920405Z",
     "start_time": "2021-10-17T09:26:31.892407Z"
    }
   },
   "outputs": [],
   "source": [
    "# drop the original one\n",
    "X = X.drop(['metal_linker'], axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "87ce969c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.935402Z",
     "start_time": "2021-10-17T09:26:31.923404Z"
    }
   },
   "outputs": [],
   "source": [
    "# fill one-hot encoded metal_linker to X\n",
    "for i in range(total_metal):\n",
    "    X[f'metal_linker_{i}'] = encoded_metal[:, i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "4fef2b62",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:31.996402Z",
     "start_time": "2021-10-17T09:26:31.937402Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>volume</th>\n",
       "      <th>volume/g</th>\n",
       "      <th>density</th>\n",
       "      <th>weight</th>\n",
       "      <th>surface_area</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume</th>\n",
       "      <th>organic_linker1</th>\n",
       "      <th>organic_linker2</th>\n",
       "      <th>topology</th>\n",
       "      <th>...</th>\n",
       "      <th>comb_cnn_2</th>\n",
       "      <th>comb_cnn_3</th>\n",
       "      <th>specific_heat</th>\n",
       "      <th>metal_linker_0</th>\n",
       "      <th>metal_linker_1</th>\n",
       "      <th>metal_linker_2</th>\n",
       "      <th>metal_linker_3</th>\n",
       "      <th>metal_linker_4</th>\n",
       "      <th>metal_linker_5</th>\n",
       "      <th>metal_linker_6</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1116.667429</td>\n",
       "      <td>0.768329</td>\n",
       "      <td>1.301526</td>\n",
       "      <td>875.240600</td>\n",
       "      <td>1170.192841</td>\n",
       "      <td>0.078990</td>\n",
       "      <td>0.060700</td>\n",
       "      <td>4</td>\n",
       "      <td>11</td>\n",
       "      <td>pcu</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>58.904250</td>\n",
       "      <td>0.499411</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2769.503842</td>\n",
       "      <td>0.754097</td>\n",
       "      <td>1.326090</td>\n",
       "      <td>2211.697211</td>\n",
       "      <td>603.610000</td>\n",
       "      <td>0.137940</td>\n",
       "      <td>0.104000</td>\n",
       "      <td>44</td>\n",
       "      <td>57</td>\n",
       "      <td>etb</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>69.381874</td>\n",
       "      <td>0.208154</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1089.818728</td>\n",
       "      <td>0.848280</td>\n",
       "      <td>1.178856</td>\n",
       "      <td>773.687960</td>\n",
       "      <td>788.500000</td>\n",
       "      <td>0.148740</td>\n",
       "      <td>0.126200</td>\n",
       "      <td>22</td>\n",
       "      <td>24</td>\n",
       "      <td>pcu</td>\n",
       "      <td>...</td>\n",
       "      <td>7.915159</td>\n",
       "      <td>82.243230</td>\n",
       "      <td>0.528491</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2205.198301</td>\n",
       "      <td>1.017907</td>\n",
       "      <td>0.982408</td>\n",
       "      <td>1304.638720</td>\n",
       "      <td>1441.530000</td>\n",
       "      <td>0.218140</td>\n",
       "      <td>0.222000</td>\n",
       "      <td>17</td>\n",
       "      <td>24</td>\n",
       "      <td>sra</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>117.579170</td>\n",
       "      <td>0.305615</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1137.800963</td>\n",
       "      <td>0.759867</td>\n",
       "      <td>1.316020</td>\n",
       "      <td>901.736120</td>\n",
       "      <td>935.486251</td>\n",
       "      <td>0.077780</td>\n",
       "      <td>0.059100</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "      <td>pcu</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>61.207540</td>\n",
       "      <td>0.462737</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>1188.302573</td>\n",
       "      <td>0.714398</td>\n",
       "      <td>1.399781</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>-309.324898</td>\n",
       "      <td>0.662656</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>4</td>\n",
       "      <td>24</td>\n",
       "      <td>pcu</td>\n",
       "      <td>...</td>\n",
       "      <td>5.032241</td>\n",
       "      <td>8.482945</td>\n",
       "      <td>0.103676</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>1506.660363</td>\n",
       "      <td>0.607603</td>\n",
       "      <td>1.645811</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>919.334794</td>\n",
       "      <td>0.011080</td>\n",
       "      <td>0.006732</td>\n",
       "      <td>42</td>\n",
       "      <td>46</td>\n",
       "      <td>etb</td>\n",
       "      <td>...</td>\n",
       "      <td>5.304706</td>\n",
       "      <td>7.986442</td>\n",
       "      <td>0.069545</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>2035.532738</td>\n",
       "      <td>0.625575</td>\n",
       "      <td>1.598529</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>-469.964067</td>\n",
       "      <td>0.756744</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>14</td>\n",
       "      <td>22</td>\n",
       "      <td>acs</td>\n",
       "      <td>...</td>\n",
       "      <td>5.137025</td>\n",
       "      <td>8.463214</td>\n",
       "      <td>0.052999</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>3985.426053</td>\n",
       "      <td>0.659602</td>\n",
       "      <td>1.516066</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>-370.490957</td>\n",
       "      <td>0.717706</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>acs</td>\n",
       "      <td>...</td>\n",
       "      <td>5.131299</td>\n",
       "      <td>8.344973</td>\n",
       "      <td>0.028541</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>1591.009408</td>\n",
       "      <td>0.462591</td>\n",
       "      <td>2.161736</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>660.350715</td>\n",
       "      <td>0.016090</td>\n",
       "      <td>0.007443</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>pcu</td>\n",
       "      <td>...</td>\n",
       "      <td>3.833471</td>\n",
       "      <td>5.429558</td>\n",
       "      <td>0.050140</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            volume  volume/g   density       weight  surface_area  \\\n",
       "0      1116.667429  0.768329  1.301526   875.240600   1170.192841   \n",
       "1      2769.503842  0.754097  1.326090  2211.697211    603.610000   \n",
       "2      1089.818728  0.848280  1.178856   773.687960    788.500000   \n",
       "3      2205.198301  1.017907  0.982408  1304.638720   1441.530000   \n",
       "4      1137.800963  0.759867  1.316020   901.736120    935.486251   \n",
       "...            ...       ...       ...          ...           ...   \n",
       "68597  1188.302573  0.714398  1.399781  1001.700216   -309.324898   \n",
       "68598  1506.660363  0.607603  1.645811  1493.296496    919.334794   \n",
       "68599  2035.532738  0.625575  1.598529  1959.518320   -469.964067   \n",
       "68600  3985.426053  0.659602  1.516066  3638.677280   -370.490957   \n",
       "68601  1591.009408  0.462591  2.161736  2071.219000    660.350715   \n",
       "\n",
       "       void_fraction  void_volume  organic_linker1  organic_linker2 topology  \\\n",
       "0           0.078990     0.060700                4               11      pcu   \n",
       "1           0.137940     0.104000               44               57      etb   \n",
       "2           0.148740     0.126200               22               24      pcu   \n",
       "3           0.218140     0.222000               17               24      sra   \n",
       "4           0.077780     0.059100                1               22      pcu   \n",
       "...              ...          ...              ...              ...      ...   \n",
       "68597       0.662656     0.473400                4               24      pcu   \n",
       "68598       0.011080     0.006732               42               46      etb   \n",
       "68599       0.756744     0.473400               14               22      acs   \n",
       "68600       0.717706     0.473400                4               15      acs   \n",
       "68601       0.016090     0.007443                9               16      pcu   \n",
       "\n",
       "       ...  comb_cnn_2  comb_cnn_3  specific_heat  metal_linker_0  \\\n",
       "0      ...    0.000000   58.904250       0.499411               0   \n",
       "1      ...    0.000000   69.381874       0.208154               0   \n",
       "2      ...    7.915159   82.243230       0.528491               0   \n",
       "3      ...    0.000000  117.579170       0.305615               0   \n",
       "4      ...    0.000000   61.207540       0.462737               0   \n",
       "...    ...         ...         ...            ...             ...   \n",
       "68597  ...    5.032241    8.482945       0.103676               0   \n",
       "68598  ...    5.304706    7.986442       0.069545               0   \n",
       "68599  ...    5.137025    8.463214       0.052999               0   \n",
       "68600  ...    5.131299    8.344973       0.028541               0   \n",
       "68601  ...    3.833471    5.429558       0.050140               0   \n",
       "\n",
       "       metal_linker_1  metal_linker_2  metal_linker_3  metal_linker_4  \\\n",
       "0                   0               1               0               0   \n",
       "1                   0               0               0               0   \n",
       "2                   1               0               0               0   \n",
       "3                   0               0               0               1   \n",
       "4                   1               0               0               0   \n",
       "...               ...             ...             ...             ...   \n",
       "68597               0               1               0               0   \n",
       "68598               0               0               0               0   \n",
       "68599               0               0               1               0   \n",
       "68600               0               0               1               0   \n",
       "68601               1               0               0               0   \n",
       "\n",
       "       metal_linker_5  metal_linker_6  \n",
       "0                   0               0  \n",
       "1                   1               0  \n",
       "2                   0               0  \n",
       "3                   0               0  \n",
       "4                   0               0  \n",
       "...               ...             ...  \n",
       "68597               0               0  \n",
       "68598               1               0  \n",
       "68599               0               0  \n",
       "68600               0               0  \n",
       "68601               0               0  \n",
       "\n",
       "[68602 rows x 41 columns]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85d5218a",
   "metadata": {},
   "source": [
    "3) One-hot **organic_linker1** and **organic_linker2**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "78fb97f3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.012401Z",
     "start_time": "2021-10-17T09:26:31.998406Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True])"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sort(df['organic_linker1'].unique()) == np.sort(df['organic_linker2'].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8059c310",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-09-25T09:48:53.605863Z",
     "start_time": "2021-09-25T09:48:53.593859Z"
    }
   },
   "source": [
    "Since both organic_linker1 and organic_linker2 contain **ALL the same** element, so we can fit only 1 lb and use for both columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "aa74ffa2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.027403Z",
     "start_time": "2021-10-17T09:26:32.014404Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "57"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "total_organic = len(organic_lb.classes_)\n",
    "total_organic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "a62412bf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.119405Z",
     "start_time": "2021-10-17T09:26:32.029403Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((68602, 57), (68602, 57))"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_organic1 = organic_lb.transform(X['organic_linker1'])\n",
    "encoded_organic2 = organic_lb.transform(X['organic_linker2'])\n",
    "encoded_organic1.shape, encoded_organic2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "f5dcfede",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.149405Z",
     "start_time": "2021-10-17T09:26:32.124406Z"
    }
   },
   "outputs": [],
   "source": [
    "# drop the original one\n",
    "X = X.drop(['organic_linker1', 'organic_linker2'], axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "10118062",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.382402Z",
     "start_time": "2021-10-17T09:26:32.151402Z"
    }
   },
   "outputs": [],
   "source": [
    "# fill one-hot encoded version to X\n",
    "\n",
    "# linker 1\n",
    "for i in range(total_organic):\n",
    "    X[f'organic_linker1_{i}'] = encoded_organic1[:, i]\n",
    "\n",
    "# linker 2\n",
    "for i in range(total_organic):\n",
    "    X[f'organic_linker2_{i}'] = encoded_organic2[:, i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "b04f7a34",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.460402Z",
     "start_time": "2021-10-17T09:26:32.384404Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>volume</th>\n",
       "      <th>volume/g</th>\n",
       "      <th>density</th>\n",
       "      <th>weight</th>\n",
       "      <th>surface_area</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume</th>\n",
       "      <th>topology</th>\n",
       "      <th>CO2/N2_selectivity</th>\n",
       "      <th>heat_adsorption</th>\n",
       "      <th>...</th>\n",
       "      <th>organic_linker2_47</th>\n",
       "      <th>organic_linker2_48</th>\n",
       "      <th>organic_linker2_49</th>\n",
       "      <th>organic_linker2_50</th>\n",
       "      <th>organic_linker2_51</th>\n",
       "      <th>organic_linker2_52</th>\n",
       "      <th>organic_linker2_53</th>\n",
       "      <th>organic_linker2_54</th>\n",
       "      <th>organic_linker2_55</th>\n",
       "      <th>organic_linker2_56</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1116.667429</td>\n",
       "      <td>0.768329</td>\n",
       "      <td>1.301526</td>\n",
       "      <td>875.240600</td>\n",
       "      <td>1170.192841</td>\n",
       "      <td>0.078990</td>\n",
       "      <td>0.060700</td>\n",
       "      <td>pcu</td>\n",
       "      <td>22.864166</td>\n",
       "      <td>6.786041</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2769.503842</td>\n",
       "      <td>0.754097</td>\n",
       "      <td>1.326090</td>\n",
       "      <td>2211.697211</td>\n",
       "      <td>603.610000</td>\n",
       "      <td>0.137940</td>\n",
       "      <td>0.104000</td>\n",
       "      <td>etb</td>\n",
       "      <td>33.616780</td>\n",
       "      <td>7.147286</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1089.818728</td>\n",
       "      <td>0.848280</td>\n",
       "      <td>1.178856</td>\n",
       "      <td>773.687960</td>\n",
       "      <td>788.500000</td>\n",
       "      <td>0.148740</td>\n",
       "      <td>0.126200</td>\n",
       "      <td>pcu</td>\n",
       "      <td>19.263726</td>\n",
       "      <td>6.347967</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2205.198301</td>\n",
       "      <td>1.017907</td>\n",
       "      <td>0.982408</td>\n",
       "      <td>1304.638720</td>\n",
       "      <td>1441.530000</td>\n",
       "      <td>0.218140</td>\n",
       "      <td>0.222000</td>\n",
       "      <td>sra</td>\n",
       "      <td>25.701377</td>\n",
       "      <td>6.190085</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1137.800963</td>\n",
       "      <td>0.759867</td>\n",
       "      <td>1.316020</td>\n",
       "      <td>901.736120</td>\n",
       "      <td>935.486251</td>\n",
       "      <td>0.077780</td>\n",
       "      <td>0.059100</td>\n",
       "      <td>pcu</td>\n",
       "      <td>30.001838</td>\n",
       "      <td>6.478063</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>1188.302573</td>\n",
       "      <td>0.714398</td>\n",
       "      <td>1.399781</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>-309.324898</td>\n",
       "      <td>0.662656</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>pcu</td>\n",
       "      <td>24.131770</td>\n",
       "      <td>1.612299</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>1506.660363</td>\n",
       "      <td>0.607603</td>\n",
       "      <td>1.645811</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>919.334794</td>\n",
       "      <td>0.011080</td>\n",
       "      <td>0.006732</td>\n",
       "      <td>etb</td>\n",
       "      <td>6.071818</td>\n",
       "      <td>1.612299</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>2035.532738</td>\n",
       "      <td>0.625575</td>\n",
       "      <td>1.598529</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>-469.964067</td>\n",
       "      <td>0.756744</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>acs</td>\n",
       "      <td>9.876134</td>\n",
       "      <td>1.612299</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>3985.426053</td>\n",
       "      <td>0.659602</td>\n",
       "      <td>1.516066</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>-370.490957</td>\n",
       "      <td>0.717706</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>acs</td>\n",
       "      <td>5.285051</td>\n",
       "      <td>1.612299</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>1591.009408</td>\n",
       "      <td>0.462591</td>\n",
       "      <td>2.161736</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>660.350715</td>\n",
       "      <td>0.016090</td>\n",
       "      <td>0.007443</td>\n",
       "      <td>pcu</td>\n",
       "      <td>2.621272</td>\n",
       "      <td>1.612299</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 153 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            volume  volume/g   density       weight  surface_area  \\\n",
       "0      1116.667429  0.768329  1.301526   875.240600   1170.192841   \n",
       "1      2769.503842  0.754097  1.326090  2211.697211    603.610000   \n",
       "2      1089.818728  0.848280  1.178856   773.687960    788.500000   \n",
       "3      2205.198301  1.017907  0.982408  1304.638720   1441.530000   \n",
       "4      1137.800963  0.759867  1.316020   901.736120    935.486251   \n",
       "...            ...       ...       ...          ...           ...   \n",
       "68597  1188.302573  0.714398  1.399781  1001.700216   -309.324898   \n",
       "68598  1506.660363  0.607603  1.645811  1493.296496    919.334794   \n",
       "68599  2035.532738  0.625575  1.598529  1959.518320   -469.964067   \n",
       "68600  3985.426053  0.659602  1.516066  3638.677280   -370.490957   \n",
       "68601  1591.009408  0.462591  2.161736  2071.219000    660.350715   \n",
       "\n",
       "       void_fraction  void_volume topology  CO2/N2_selectivity  \\\n",
       "0           0.078990     0.060700      pcu           22.864166   \n",
       "1           0.137940     0.104000      etb           33.616780   \n",
       "2           0.148740     0.126200      pcu           19.263726   \n",
       "3           0.218140     0.222000      sra           25.701377   \n",
       "4           0.077780     0.059100      pcu           30.001838   \n",
       "...              ...          ...      ...                 ...   \n",
       "68597       0.662656     0.473400      pcu           24.131770   \n",
       "68598       0.011080     0.006732      etb            6.071818   \n",
       "68599       0.756744     0.473400      acs            9.876134   \n",
       "68600       0.717706     0.473400      acs            5.285051   \n",
       "68601       0.016090     0.007443      pcu            2.621272   \n",
       "\n",
       "       heat_adsorption  ...  organic_linker2_47  organic_linker2_48  \\\n",
       "0             6.786041  ...                   0                   0   \n",
       "1             7.147286  ...                   0                   0   \n",
       "2             6.347967  ...                   0                   0   \n",
       "3             6.190085  ...                   0                   0   \n",
       "4             6.478063  ...                   0                   0   \n",
       "...                ...  ...                 ...                 ...   \n",
       "68597         1.612299  ...                   0                   0   \n",
       "68598         1.612299  ...                   0                   0   \n",
       "68599         1.612299  ...                   0                   0   \n",
       "68600         1.612299  ...                   0                   0   \n",
       "68601         1.612299  ...                   0                   0   \n",
       "\n",
       "       organic_linker2_49  organic_linker2_50  organic_linker2_51  \\\n",
       "0                       0                   0                   0   \n",
       "1                       0                   0                   0   \n",
       "2                       0                   0                   0   \n",
       "3                       0                   0                   0   \n",
       "4                       0                   0                   0   \n",
       "...                   ...                 ...                 ...   \n",
       "68597                   0                   0                   0   \n",
       "68598                   0                   0                   0   \n",
       "68599                   0                   0                   0   \n",
       "68600                   0                   0                   0   \n",
       "68601                   0                   0                   0   \n",
       "\n",
       "       organic_linker2_52  organic_linker2_53  organic_linker2_54  \\\n",
       "0                       0                   0                   0   \n",
       "1                       0                   0                   1   \n",
       "2                       0                   0                   0   \n",
       "3                       0                   0                   0   \n",
       "4                       0                   0                   0   \n",
       "...                   ...                 ...                 ...   \n",
       "68597                   0                   0                   0   \n",
       "68598                   0                   0                   0   \n",
       "68599                   0                   0                   0   \n",
       "68600                   0                   0                   0   \n",
       "68601                   0                   0                   0   \n",
       "\n",
       "       organic_linker2_55  organic_linker2_56  \n",
       "0                       0                   0  \n",
       "1                       0                   0  \n",
       "2                       0                   0  \n",
       "3                       0                   0  \n",
       "4                       0                   0  \n",
       "...                   ...                 ...  \n",
       "68597                   0                   0  \n",
       "68598                   0                   0  \n",
       "68599                   0                   0  \n",
       "68600                   0                   0  \n",
       "68601                   0                   0  \n",
       "\n",
       "[68602 rows x 153 columns]"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe536990",
   "metadata": {},
   "source": [
    "4) One-hot **topology**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "93589e0f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.475400Z",
     "start_time": "2021-10-17T09:26:32.462405Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "total_topology = len(topology_lb.classes_)\n",
    "total_topology"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "57d099ce",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.614403Z",
     "start_time": "2021-10-17T09:26:32.477403Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(68602, 11)"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_topology = topology_lb.transform(X['topology'])\n",
    "encoded_topology.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "c320dba2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.689404Z",
     "start_time": "2021-10-17T09:26:32.615402Z"
    }
   },
   "outputs": [],
   "source": [
    "# drop the original one\n",
    "X = X.drop(['topology'], axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "e85b0619",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.721403Z",
     "start_time": "2021-10-17T09:26:32.692405Z"
    }
   },
   "outputs": [],
   "source": [
    "# fill one-hot encoded functional_groups to X\n",
    "for i in range(total_topology):\n",
    "    X[f'topology_{i}'] = encoded_topology[:, i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "4d301447",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.798406Z",
     "start_time": "2021-10-17T09:26:32.729403Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>volume</th>\n",
       "      <th>volume/g</th>\n",
       "      <th>density</th>\n",
       "      <th>weight</th>\n",
       "      <th>surface_area</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume</th>\n",
       "      <th>CO2/N2_selectivity</th>\n",
       "      <th>heat_adsorption</th>\n",
       "      <th>n_atoms</th>\n",
       "      <th>...</th>\n",
       "      <th>topology_1</th>\n",
       "      <th>topology_2</th>\n",
       "      <th>topology_3</th>\n",
       "      <th>topology_4</th>\n",
       "      <th>topology_5</th>\n",
       "      <th>topology_6</th>\n",
       "      <th>topology_7</th>\n",
       "      <th>topology_8</th>\n",
       "      <th>topology_9</th>\n",
       "      <th>topology_10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1116.667429</td>\n",
       "      <td>0.768329</td>\n",
       "      <td>1.301526</td>\n",
       "      <td>875.240600</td>\n",
       "      <td>1170.192841</td>\n",
       "      <td>0.078990</td>\n",
       "      <td>0.060700</td>\n",
       "      <td>22.864166</td>\n",
       "      <td>6.786041</td>\n",
       "      <td>75</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2769.503842</td>\n",
       "      <td>0.754097</td>\n",
       "      <td>1.326090</td>\n",
       "      <td>2211.697211</td>\n",
       "      <td>603.610000</td>\n",
       "      <td>0.137940</td>\n",
       "      <td>0.104000</td>\n",
       "      <td>33.616780</td>\n",
       "      <td>7.147286</td>\n",
       "      <td>194</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1089.818728</td>\n",
       "      <td>0.848280</td>\n",
       "      <td>1.178856</td>\n",
       "      <td>773.687960</td>\n",
       "      <td>788.500000</td>\n",
       "      <td>0.148740</td>\n",
       "      <td>0.126200</td>\n",
       "      <td>19.263726</td>\n",
       "      <td>6.347967</td>\n",
       "      <td>82</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2205.198301</td>\n",
       "      <td>1.017907</td>\n",
       "      <td>0.982408</td>\n",
       "      <td>1304.638720</td>\n",
       "      <td>1441.530000</td>\n",
       "      <td>0.218140</td>\n",
       "      <td>0.222000</td>\n",
       "      <td>25.701377</td>\n",
       "      <td>6.190085</td>\n",
       "      <td>112</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1137.800963</td>\n",
       "      <td>0.759867</td>\n",
       "      <td>1.316020</td>\n",
       "      <td>901.736120</td>\n",
       "      <td>935.486251</td>\n",
       "      <td>0.077780</td>\n",
       "      <td>0.059100</td>\n",
       "      <td>30.001838</td>\n",
       "      <td>6.478063</td>\n",
       "      <td>94</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>1188.302573</td>\n",
       "      <td>0.714398</td>\n",
       "      <td>1.399781</td>\n",
       "      <td>1001.700216</td>\n",
       "      <td>-309.324898</td>\n",
       "      <td>0.662656</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>24.131770</td>\n",
       "      <td>1.612299</td>\n",
       "      <td>119</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>1506.660363</td>\n",
       "      <td>0.607603</td>\n",
       "      <td>1.645811</td>\n",
       "      <td>1493.296496</td>\n",
       "      <td>919.334794</td>\n",
       "      <td>0.011080</td>\n",
       "      <td>0.006732</td>\n",
       "      <td>6.071818</td>\n",
       "      <td>1.612299</td>\n",
       "      <td>126</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>2035.532738</td>\n",
       "      <td>0.625575</td>\n",
       "      <td>1.598529</td>\n",
       "      <td>1959.518320</td>\n",
       "      <td>-469.964067</td>\n",
       "      <td>0.756744</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>9.876134</td>\n",
       "      <td>1.612299</td>\n",
       "      <td>204</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>3985.426053</td>\n",
       "      <td>0.659602</td>\n",
       "      <td>1.516066</td>\n",
       "      <td>3638.677280</td>\n",
       "      <td>-370.490957</td>\n",
       "      <td>0.717706</td>\n",
       "      <td>0.473400</td>\n",
       "      <td>5.285051</td>\n",
       "      <td>1.612299</td>\n",
       "      <td>364</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>1591.009408</td>\n",
       "      <td>0.462591</td>\n",
       "      <td>2.161736</td>\n",
       "      <td>2071.219000</td>\n",
       "      <td>660.350715</td>\n",
       "      <td>0.016090</td>\n",
       "      <td>0.007443</td>\n",
       "      <td>2.621272</td>\n",
       "      <td>1.612299</td>\n",
       "      <td>116</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 163 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            volume  volume/g   density       weight  surface_area  \\\n",
       "0      1116.667429  0.768329  1.301526   875.240600   1170.192841   \n",
       "1      2769.503842  0.754097  1.326090  2211.697211    603.610000   \n",
       "2      1089.818728  0.848280  1.178856   773.687960    788.500000   \n",
       "3      2205.198301  1.017907  0.982408  1304.638720   1441.530000   \n",
       "4      1137.800963  0.759867  1.316020   901.736120    935.486251   \n",
       "...            ...       ...       ...          ...           ...   \n",
       "68597  1188.302573  0.714398  1.399781  1001.700216   -309.324898   \n",
       "68598  1506.660363  0.607603  1.645811  1493.296496    919.334794   \n",
       "68599  2035.532738  0.625575  1.598529  1959.518320   -469.964067   \n",
       "68600  3985.426053  0.659602  1.516066  3638.677280   -370.490957   \n",
       "68601  1591.009408  0.462591  2.161736  2071.219000    660.350715   \n",
       "\n",
       "       void_fraction  void_volume  CO2/N2_selectivity  heat_adsorption  \\\n",
       "0           0.078990     0.060700           22.864166         6.786041   \n",
       "1           0.137940     0.104000           33.616780         7.147286   \n",
       "2           0.148740     0.126200           19.263726         6.347967   \n",
       "3           0.218140     0.222000           25.701377         6.190085   \n",
       "4           0.077780     0.059100           30.001838         6.478063   \n",
       "...              ...          ...                 ...              ...   \n",
       "68597       0.662656     0.473400           24.131770         1.612299   \n",
       "68598       0.011080     0.006732            6.071818         1.612299   \n",
       "68599       0.756744     0.473400            9.876134         1.612299   \n",
       "68600       0.717706     0.473400            5.285051         1.612299   \n",
       "68601       0.016090     0.007443            2.621272         1.612299   \n",
       "\n",
       "       n_atoms  ...  topology_1  topology_2  topology_3  topology_4  \\\n",
       "0           75  ...           0           0           0           0   \n",
       "1          194  ...           0           1           0           0   \n",
       "2           82  ...           0           0           0           0   \n",
       "3          112  ...           0           0           0           0   \n",
       "4           94  ...           0           0           0           0   \n",
       "...        ...  ...         ...         ...         ...         ...   \n",
       "68597      119  ...           0           0           0           0   \n",
       "68598      126  ...           0           1           0           0   \n",
       "68599      204  ...           0           0           0           0   \n",
       "68600      364  ...           0           0           0           0   \n",
       "68601      116  ...           0           0           0           0   \n",
       "\n",
       "       topology_5  topology_6  topology_7  topology_8  topology_9  topology_10  \n",
       "0               1           0           0           0           0            0  \n",
       "1               0           0           0           0           0            0  \n",
       "2               1           0           0           0           0            0  \n",
       "3               0           0           0           1           0            0  \n",
       "4               1           0           0           0           0            0  \n",
       "...           ...         ...         ...         ...         ...          ...  \n",
       "68597           1           0           0           0           0            0  \n",
       "68598           0           0           0           0           0            0  \n",
       "68599           0           0           0           0           0            0  \n",
       "68600           0           0           0           0           0            0  \n",
       "68601           1           0           0           0           0            0  \n",
       "\n",
       "[68602 rows x 163 columns]"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "5999968c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.814402Z",
     "start_time": "2021-10-17T09:26:32.800404Z"
    }
   },
   "outputs": [],
   "source": [
    "# X.to_csv('train_extra_2_no_scaling.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "383982ad",
   "metadata": {},
   "source": [
    "### Feature scaling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "4c86103e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.829404Z",
     "start_time": "2021-10-17T09:26:32.816401Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['volume', 'volume/g', 'density', 'weight', 'surface_area',\n",
       "       'void_fraction', 'void_volume', 'CO2/N2_selectivity', 'heat_adsorption',\n",
       "       'n_atoms', 'mol_avg_mass', 'charges', 'mol_avg_radius', 'atoms_volume',\n",
       "       'atoms_area', 'di_cnn_1', 'di_cnn_2', 'di_cnn_4', 'di_cnn_5',\n",
       "       'di_cnn_6'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.columns[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "25560f2d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.845402Z",
     "start_time": "2021-10-17T09:26:32.839404Z"
    }
   },
   "outputs": [],
   "source": [
    "feats_to_scale = ['volume', 'volume/g', 'density', 'weight', 'surface_area',\n",
    "                 'void_fraction', 'void_volume', 'CO2/N2_selectivity', 'heat_adsorption',\n",
    "                 'n_atoms', 'mol_avg_mass', 'charges', 'mol_avg_radius', 'atoms_volume', 'atoms_area','di_cnn_1',\n",
    "         'di_cnn_2', 'di_cnn_4', 'di_cnn_5', 'di_cnn_6', 'di_cnn_8', 'di_cnn_10',\n",
    "         'di_cnn_12', 'di_cnn_13', 'di_cnn_14', 'di_cnn_15', 'comb_cnn_0', \n",
    "         'comb_cnn_1', 'comb_cnn_2', 'comb_cnn_3', 'specific_heat']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "0d3164db",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:32.999406Z",
     "start_time": "2021-10-17T09:26:32.863404Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RobustScaler()"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rb_scaler = RobustScaler()\n",
    "rb_scaler.fit(X[feats_to_scale])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "d211c703",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:33.045401Z",
     "start_time": "2021-10-17T09:26:33.002404Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(68602, 31)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaled_feats = rb_scaler.transform(X[feats_to_scale])\n",
    "scaled_feats.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "105b6c0c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:33.075405Z",
     "start_time": "2021-10-17T09:26:33.046402Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "volume\n",
      "volume/g\n",
      "density\n",
      "weight\n",
      "surface_area\n",
      "void_fraction\n",
      "void_volume\n",
      "CO2/N2_selectivity\n",
      "heat_adsorption\n",
      "n_atoms\n",
      "mol_avg_mass\n",
      "charges\n",
      "mol_avg_radius\n",
      "atoms_volume\n",
      "atoms_area\n",
      "di_cnn_1\n",
      "di_cnn_2\n",
      "di_cnn_4\n",
      "di_cnn_5\n",
      "di_cnn_6\n",
      "di_cnn_8\n",
      "di_cnn_10\n",
      "di_cnn_12\n",
      "di_cnn_13\n",
      "di_cnn_14\n",
      "di_cnn_15\n",
      "comb_cnn_0\n",
      "comb_cnn_1\n",
      "comb_cnn_2\n",
      "comb_cnn_3\n",
      "specific_heat\n"
     ]
    }
   ],
   "source": [
    "for i, feat in enumerate(feats_to_scale):\n",
    "    print(feat)\n",
    "    X[feat] = scaled_feats[:, i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "ecf5290a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:33.113403Z",
     "start_time": "2021-10-17T09:26:33.078406Z"
    }
   },
   "outputs": [],
   "source": [
    "## Same results with 1 scaler for all feats\n",
    "# let's try 1 scaler for 1 feature\n",
    "# scalers = {}\n",
    "# for feat in feats_to_scale:\n",
    "#     print(feat)\n",
    "#     scaler = RobustScaler()\n",
    "#     scaler.fit(X[[feat]])\n",
    "#     scaled_feat = scaler.transform(X[[feat]])\n",
    "#     X[feat] = scaled_feat\n",
    "#     scalers[feat] = scaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "6961854c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:33.232404Z",
     "start_time": "2021-10-17T09:26:33.134404Z"
    }
   },
   "outputs": [],
   "source": [
    "clean_X = X.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "beaa32cc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:33.352401Z",
     "start_time": "2021-10-17T09:26:33.240407Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>volume</th>\n",
       "      <th>volume/g</th>\n",
       "      <th>density</th>\n",
       "      <th>weight</th>\n",
       "      <th>surface_area</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume</th>\n",
       "      <th>CO2/N2_selectivity</th>\n",
       "      <th>heat_adsorption</th>\n",
       "      <th>n_atoms</th>\n",
       "      <th>...</th>\n",
       "      <th>topology_1</th>\n",
       "      <th>topology_2</th>\n",
       "      <th>topology_3</th>\n",
       "      <th>topology_4</th>\n",
       "      <th>topology_5</th>\n",
       "      <th>topology_6</th>\n",
       "      <th>topology_7</th>\n",
       "      <th>topology_8</th>\n",
       "      <th>topology_9</th>\n",
       "      <th>topology_10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.524386</td>\n",
       "      <td>-0.584699</td>\n",
       "      <td>0.798882</td>\n",
       "      <td>-0.383085</td>\n",
       "      <td>-0.257582</td>\n",
       "      <td>-0.872930</td>\n",
       "      <td>-0.642857</td>\n",
       "      <td>0.157631</td>\n",
       "      <td>0.609280</td>\n",
       "      <td>-0.406977</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.282926</td>\n",
       "      <td>-0.615547</td>\n",
       "      <td>0.856904</td>\n",
       "      <td>0.949117</td>\n",
       "      <td>-0.604123</td>\n",
       "      <td>-0.588382</td>\n",
       "      <td>-0.513449</td>\n",
       "      <td>0.691708</td>\n",
       "      <td>0.845745</td>\n",
       "      <td>0.976744</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.537500</td>\n",
       "      <td>-0.411402</td>\n",
       "      <td>0.509126</td>\n",
       "      <td>-0.484315</td>\n",
       "      <td>-0.491038</td>\n",
       "      <td>-0.536250</td>\n",
       "      <td>-0.447101</td>\n",
       "      <td>-0.021201</td>\n",
       "      <td>0.322523</td>\n",
       "      <td>-0.325581</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.007296</td>\n",
       "      <td>-0.043730</td>\n",
       "      <td>0.045099</td>\n",
       "      <td>0.044946</td>\n",
       "      <td>-0.091623</td>\n",
       "      <td>-0.201260</td>\n",
       "      <td>-0.160789</td>\n",
       "      <td>0.298554</td>\n",
       "      <td>0.219176</td>\n",
       "      <td>0.023256</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.514063</td>\n",
       "      <td>-0.603040</td>\n",
       "      <td>0.833117</td>\n",
       "      <td>-0.356674</td>\n",
       "      <td>-0.401136</td>\n",
       "      <td>-0.878771</td>\n",
       "      <td>-0.647639</td>\n",
       "      <td>0.512156</td>\n",
       "      <td>0.407682</td>\n",
       "      <td>-0.186047</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>-0.489396</td>\n",
       "      <td>-0.701596</td>\n",
       "      <td>1.030967</td>\n",
       "      <td>-0.257028</td>\n",
       "      <td>-1.162504</td>\n",
       "      <td>1.944400</td>\n",
       "      <td>0.590556</td>\n",
       "      <td>0.220592</td>\n",
       "      <td>-2.777374</td>\n",
       "      <td>0.104651</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>-0.333898</td>\n",
       "      <td>-0.933076</td>\n",
       "      <td>1.612110</td>\n",
       "      <td>0.233003</td>\n",
       "      <td>-0.411015</td>\n",
       "      <td>-1.200729</td>\n",
       "      <td>-0.804148</td>\n",
       "      <td>-0.676437</td>\n",
       "      <td>-2.777374</td>\n",
       "      <td>0.186047</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>-0.075575</td>\n",
       "      <td>-0.894122</td>\n",
       "      <td>1.500426</td>\n",
       "      <td>0.697741</td>\n",
       "      <td>-1.260756</td>\n",
       "      <td>2.398555</td>\n",
       "      <td>0.590556</td>\n",
       "      <td>-0.487479</td>\n",
       "      <td>-2.777374</td>\n",
       "      <td>1.093023</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>0.876831</td>\n",
       "      <td>-0.820367</td>\n",
       "      <td>1.305642</td>\n",
       "      <td>2.371556</td>\n",
       "      <td>-1.199915</td>\n",
       "      <td>2.210121</td>\n",
       "      <td>0.590556</td>\n",
       "      <td>-0.715515</td>\n",
       "      <td>-2.777374</td>\n",
       "      <td>2.953488</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>-0.292698</td>\n",
       "      <td>-1.247394</td>\n",
       "      <td>2.830765</td>\n",
       "      <td>0.809086</td>\n",
       "      <td>-0.569418</td>\n",
       "      <td>-1.176546</td>\n",
       "      <td>-0.802023</td>\n",
       "      <td>-0.847824</td>\n",
       "      <td>-2.777374</td>\n",
       "      <td>0.069767</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 163 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         volume  volume/g   density    weight  surface_area  void_fraction  \\\n",
       "0     -0.524386 -0.584699  0.798882 -0.383085     -0.257582      -0.872930   \n",
       "1      0.282926 -0.615547  0.856904  0.949117     -0.604123      -0.588382   \n",
       "2     -0.537500 -0.411402  0.509126 -0.484315     -0.491038      -0.536250   \n",
       "3      0.007296 -0.043730  0.045099  0.044946     -0.091623      -0.201260   \n",
       "4     -0.514063 -0.603040  0.833117 -0.356674     -0.401136      -0.878771   \n",
       "...         ...       ...       ...       ...           ...            ...   \n",
       "68597 -0.489396 -0.701596  1.030967 -0.257028     -1.162504       1.944400   \n",
       "68598 -0.333898 -0.933076  1.612110  0.233003     -0.411015      -1.200729   \n",
       "68599 -0.075575 -0.894122  1.500426  0.697741     -1.260756       2.398555   \n",
       "68600  0.876831 -0.820367  1.305642  2.371556     -1.199915       2.210121   \n",
       "68601 -0.292698 -1.247394  2.830765  0.809086     -0.569418      -1.176546   \n",
       "\n",
       "       void_volume  CO2/N2_selectivity  heat_adsorption   n_atoms  ...  \\\n",
       "0        -0.642857            0.157631         0.609280 -0.406977  ...   \n",
       "1        -0.513449            0.691708         0.845745  0.976744  ...   \n",
       "2        -0.447101           -0.021201         0.322523 -0.325581  ...   \n",
       "3        -0.160789            0.298554         0.219176  0.023256  ...   \n",
       "4        -0.647639            0.512156         0.407682 -0.186047  ...   \n",
       "...            ...                 ...              ...       ...  ...   \n",
       "68597     0.590556            0.220592        -2.777374  0.104651  ...   \n",
       "68598    -0.804148           -0.676437        -2.777374  0.186047  ...   \n",
       "68599     0.590556           -0.487479        -2.777374  1.093023  ...   \n",
       "68600     0.590556           -0.715515        -2.777374  2.953488  ...   \n",
       "68601    -0.802023           -0.847824        -2.777374  0.069767  ...   \n",
       "\n",
       "       topology_1  topology_2  topology_3  topology_4  topology_5  topology_6  \\\n",
       "0               0           0           0           0           1           0   \n",
       "1               0           1           0           0           0           0   \n",
       "2               0           0           0           0           1           0   \n",
       "3               0           0           0           0           0           0   \n",
       "4               0           0           0           0           1           0   \n",
       "...           ...         ...         ...         ...         ...         ...   \n",
       "68597           0           0           0           0           1           0   \n",
       "68598           0           1           0           0           0           0   \n",
       "68599           0           0           0           0           0           0   \n",
       "68600           0           0           0           0           0           0   \n",
       "68601           0           0           0           0           1           0   \n",
       "\n",
       "       topology_7  topology_8  topology_9  topology_10  \n",
       "0               0           0           0            0  \n",
       "1               0           0           0            0  \n",
       "2               0           0           0            0  \n",
       "3               0           1           0            0  \n",
       "4               0           0           0            0  \n",
       "...           ...         ...         ...          ...  \n",
       "68597           0           0           0            0  \n",
       "68598           0           0           0            0  \n",
       "68599           0           0           0            0  \n",
       "68600           0           0           0            0  \n",
       "68601           0           0           0            0  \n",
       "\n",
       "[68602 rows x 163 columns]"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "613df983",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:33.367405Z",
     "start_time": "2021-10-17T09:26:33.359406Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['volume', 'volume/g', 'density', 'weight', 'surface_area',\n",
       "       'void_fraction', 'void_volume', 'CO2/N2_selectivity', 'heat_adsorption',\n",
       "       'n_atoms', 'mol_avg_mass', 'charges', 'mol_avg_radius', 'atoms_volume',\n",
       "       'atoms_area'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_X.columns[:15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "978a239a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:33.444404Z",
     "start_time": "2021-10-17T09:26:33.369404Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>volume</th>\n",
       "      <th>volume/g</th>\n",
       "      <th>density</th>\n",
       "      <th>weight</th>\n",
       "      <th>surface_area</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume</th>\n",
       "      <th>CO2/N2_selectivity</th>\n",
       "      <th>heat_adsorption</th>\n",
       "      <th>n_atoms</th>\n",
       "      <th>...</th>\n",
       "      <th>topology_1</th>\n",
       "      <th>topology_2</th>\n",
       "      <th>topology_3</th>\n",
       "      <th>topology_4</th>\n",
       "      <th>topology_5</th>\n",
       "      <th>topology_6</th>\n",
       "      <th>topology_7</th>\n",
       "      <th>topology_8</th>\n",
       "      <th>topology_9</th>\n",
       "      <th>topology_10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.524386</td>\n",
       "      <td>-0.584699</td>\n",
       "      <td>0.798882</td>\n",
       "      <td>-0.383085</td>\n",
       "      <td>-0.257582</td>\n",
       "      <td>-0.872930</td>\n",
       "      <td>-0.642857</td>\n",
       "      <td>0.157631</td>\n",
       "      <td>0.609280</td>\n",
       "      <td>-0.406977</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.282926</td>\n",
       "      <td>-0.615547</td>\n",
       "      <td>0.856904</td>\n",
       "      <td>0.949117</td>\n",
       "      <td>-0.604123</td>\n",
       "      <td>-0.588382</td>\n",
       "      <td>-0.513449</td>\n",
       "      <td>0.691708</td>\n",
       "      <td>0.845745</td>\n",
       "      <td>0.976744</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.537500</td>\n",
       "      <td>-0.411402</td>\n",
       "      <td>0.509126</td>\n",
       "      <td>-0.484315</td>\n",
       "      <td>-0.491038</td>\n",
       "      <td>-0.536250</td>\n",
       "      <td>-0.447101</td>\n",
       "      <td>-0.021201</td>\n",
       "      <td>0.322523</td>\n",
       "      <td>-0.325581</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.007296</td>\n",
       "      <td>-0.043730</td>\n",
       "      <td>0.045099</td>\n",
       "      <td>0.044946</td>\n",
       "      <td>-0.091623</td>\n",
       "      <td>-0.201260</td>\n",
       "      <td>-0.160789</td>\n",
       "      <td>0.298554</td>\n",
       "      <td>0.219176</td>\n",
       "      <td>0.023256</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.514063</td>\n",
       "      <td>-0.603040</td>\n",
       "      <td>0.833117</td>\n",
       "      <td>-0.356674</td>\n",
       "      <td>-0.401136</td>\n",
       "      <td>-0.878771</td>\n",
       "      <td>-0.647639</td>\n",
       "      <td>0.512156</td>\n",
       "      <td>0.407682</td>\n",
       "      <td>-0.186047</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68597</th>\n",
       "      <td>-0.489396</td>\n",
       "      <td>-0.701596</td>\n",
       "      <td>1.030967</td>\n",
       "      <td>-0.257028</td>\n",
       "      <td>-1.162504</td>\n",
       "      <td>1.944400</td>\n",
       "      <td>0.590556</td>\n",
       "      <td>0.220592</td>\n",
       "      <td>-2.777374</td>\n",
       "      <td>0.104651</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68598</th>\n",
       "      <td>-0.333898</td>\n",
       "      <td>-0.933076</td>\n",
       "      <td>1.612110</td>\n",
       "      <td>0.233003</td>\n",
       "      <td>-0.411015</td>\n",
       "      <td>-1.200729</td>\n",
       "      <td>-0.804148</td>\n",
       "      <td>-0.676437</td>\n",
       "      <td>-2.777374</td>\n",
       "      <td>0.186047</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68599</th>\n",
       "      <td>-0.075575</td>\n",
       "      <td>-0.894122</td>\n",
       "      <td>1.500426</td>\n",
       "      <td>0.697741</td>\n",
       "      <td>-1.260756</td>\n",
       "      <td>2.398555</td>\n",
       "      <td>0.590556</td>\n",
       "      <td>-0.487479</td>\n",
       "      <td>-2.777374</td>\n",
       "      <td>1.093023</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68600</th>\n",
       "      <td>0.876831</td>\n",
       "      <td>-0.820367</td>\n",
       "      <td>1.305642</td>\n",
       "      <td>2.371556</td>\n",
       "      <td>-1.199915</td>\n",
       "      <td>2.210121</td>\n",
       "      <td>0.590556</td>\n",
       "      <td>-0.715515</td>\n",
       "      <td>-2.777374</td>\n",
       "      <td>2.953488</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68601</th>\n",
       "      <td>-0.292698</td>\n",
       "      <td>-1.247394</td>\n",
       "      <td>2.830765</td>\n",
       "      <td>0.809086</td>\n",
       "      <td>-0.569418</td>\n",
       "      <td>-1.176546</td>\n",
       "      <td>-0.802023</td>\n",
       "      <td>-0.847824</td>\n",
       "      <td>-2.777374</td>\n",
       "      <td>0.069767</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>68602 rows × 163 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         volume  volume/g   density    weight  surface_area  void_fraction  \\\n",
       "0     -0.524386 -0.584699  0.798882 -0.383085     -0.257582      -0.872930   \n",
       "1      0.282926 -0.615547  0.856904  0.949117     -0.604123      -0.588382   \n",
       "2     -0.537500 -0.411402  0.509126 -0.484315     -0.491038      -0.536250   \n",
       "3      0.007296 -0.043730  0.045099  0.044946     -0.091623      -0.201260   \n",
       "4     -0.514063 -0.603040  0.833117 -0.356674     -0.401136      -0.878771   \n",
       "...         ...       ...       ...       ...           ...            ...   \n",
       "68597 -0.489396 -0.701596  1.030967 -0.257028     -1.162504       1.944400   \n",
       "68598 -0.333898 -0.933076  1.612110  0.233003     -0.411015      -1.200729   \n",
       "68599 -0.075575 -0.894122  1.500426  0.697741     -1.260756       2.398555   \n",
       "68600  0.876831 -0.820367  1.305642  2.371556     -1.199915       2.210121   \n",
       "68601 -0.292698 -1.247394  2.830765  0.809086     -0.569418      -1.176546   \n",
       "\n",
       "       void_volume  CO2/N2_selectivity  heat_adsorption   n_atoms  ...  \\\n",
       "0        -0.642857            0.157631         0.609280 -0.406977  ...   \n",
       "1        -0.513449            0.691708         0.845745  0.976744  ...   \n",
       "2        -0.447101           -0.021201         0.322523 -0.325581  ...   \n",
       "3        -0.160789            0.298554         0.219176  0.023256  ...   \n",
       "4        -0.647639            0.512156         0.407682 -0.186047  ...   \n",
       "...            ...                 ...              ...       ...  ...   \n",
       "68597     0.590556            0.220592        -2.777374  0.104651  ...   \n",
       "68598    -0.804148           -0.676437        -2.777374  0.186047  ...   \n",
       "68599     0.590556           -0.487479        -2.777374  1.093023  ...   \n",
       "68600     0.590556           -0.715515        -2.777374  2.953488  ...   \n",
       "68601    -0.802023           -0.847824        -2.777374  0.069767  ...   \n",
       "\n",
       "       topology_1  topology_2  topology_3  topology_4  topology_5  topology_6  \\\n",
       "0               0           0           0           0           1           0   \n",
       "1               0           1           0           0           0           0   \n",
       "2               0           0           0           0           1           0   \n",
       "3               0           0           0           0           0           0   \n",
       "4               0           0           0           0           1           0   \n",
       "...           ...         ...         ...         ...         ...         ...   \n",
       "68597           0           0           0           0           1           0   \n",
       "68598           0           1           0           0           0           0   \n",
       "68599           0           0           0           0           0           0   \n",
       "68600           0           0           0           0           0           0   \n",
       "68601           0           0           0           0           1           0   \n",
       "\n",
       "       topology_7  topology_8  topology_9  topology_10  \n",
       "0               0           0           0            0  \n",
       "1               0           0           0            0  \n",
       "2               0           0           0            0  \n",
       "3               0           1           0            0  \n",
       "4               0           0           0            0  \n",
       "...           ...         ...         ...          ...  \n",
       "68597           0           0           0            0  \n",
       "68598           0           0           0            0  \n",
       "68599           0           0           0            0  \n",
       "68600           0           0           0            0  \n",
       "68601           0           0           0            0  \n",
       "\n",
       "[68602 rows x 163 columns]"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34ca6e8f",
   "metadata": {},
   "source": [
    "## Notes\n",
    "current ranking:\n",
    "- MLP\n",
    "- LGBM\n",
    "- GB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "e439f425",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:26:39.559882Z",
     "start_time": "2021-10-17T09:26:36.890378Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor\n",
    "from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error\n",
    "from sklearn.neural_network import MLPRegressor\n",
    "from sklearn.svm import SVR\n",
    "from xgboost import XGBRegressor\n",
    "from lightgbm import LGBMRegressor\n",
    "from pytorch_tabnet.tab_model import TabNetRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "273dcfd5",
   "metadata": {},
   "source": [
    "## Load params from optuna"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ea9130c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T12:04:35.183092Z",
     "start_time": "2021-10-16T12:04:35.183092Z"
    }
   },
   "outputs": [],
   "source": [
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a478e04",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T12:04:35.184092Z",
     "start_time": "2021-10-16T12:04:35.184092Z"
    }
   },
   "outputs": [],
   "source": [
    "# gb_best_params_am = pickle.load(open('pickle/gb_best_params_am.pickle', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4daf273",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T12:04:35.186092Z",
     "start_time": "2021-10-16T12:04:35.186092Z"
    }
   },
   "outputs": [],
   "source": [
    "# gb_best_params_wine = pickle.load(open('pickle/gb_best_params_wine.pickle', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42e27207",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T12:04:35.187092Z",
     "start_time": "2021-10-16T12:04:35.187092Z"
    }
   },
   "outputs": [],
   "source": [
    "# mlp_best_params = pickle.load(open('pickle/mlp_best_params.pickle', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3484f88",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T12:04:35.189091Z",
     "start_time": "2021-10-16T12:04:35.189091Z"
    }
   },
   "outputs": [],
   "source": [
    "# lgb_best_params = pickle.load(open('pickle/lgb_best_params.pickle', 'rb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5357fe6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T12:04:35.191093Z",
     "start_time": "2021-10-16T12:04:35.191093Z"
    }
   },
   "outputs": [],
   "source": [
    "# tab_best_params = pickle.load(open('pickle/tabnet_best_params.pickle', 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90b0b1de",
   "metadata": {},
   "source": [
    "### Gadient Boosting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2614,
   "id": "4a7fd835",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:08:02.700581Z",
     "start_time": "2021-10-16T15:49:07.506755Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GradientBoostingRegressor(learning_rate=0.07336551102484597, max_depth=20,\n",
       "                          max_leaf_nodes=30, n_estimators=975,\n",
       "                          subsample=0.9780814357664563,\n",
       "                          validation_fraction=0.23861317396041049)"
      ]
     },
     "execution_count": 2614,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gb_best_params = {'n_estimators': 975, \n",
    "                  'max_depth': 20, \n",
    "                  'learning_rate': 0.07336551102484597, \n",
    "                  'subsample': 0.9780814357664563, \n",
    "                  'validation_fraction': 0.23861317396041049, \n",
    "                  'max_leaf_nodes': 30}\n",
    "gb = GradientBoostingRegressor(**gb_best_params)\n",
    "gb.fit(clean_X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2615,
   "id": "6878bd8e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:08:05.566583Z",
     "start_time": "2021-10-16T16:08:02.703582Z"
    }
   },
   "outputs": [],
   "source": [
    "y_hat = gb.predict(clean_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2616,
   "id": "b6ea1bb8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:08:08.988596Z",
     "start_time": "2021-10-16T16:08:08.973592Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13.382120543161381"
      ]
     },
     "execution_count": 2616,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_absolute_error(y, y_hat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2617,
   "id": "dc6d079e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:08:10.959608Z",
     "start_time": "2021-10-16T16:08:10.941607Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9575112806742445"
      ]
     },
     "execution_count": 2617,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r2_score(y, y_hat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2618,
   "id": "820dd33d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:08:16.100626Z",
     "start_time": "2021-10-16T16:08:16.013625Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['models/gb_best_model.joblib']"
      ]
     },
     "execution_count": 2618,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# joblib.dump(gb, 'models/gb_best_model.joblib')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c72cbec",
   "metadata": {},
   "source": [
    "## MLP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 890,
   "id": "818d433f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T10:43:10.645758Z",
     "start_time": "2021-10-16T10:41:58.714786Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 1, loss = 1442.73342612\n",
      "Validation score: 0.913845\n",
      "Iteration 2, loss = 357.79165794\n",
      "Validation score: 0.915173\n",
      "Iteration 3, loss = 354.24238166\n",
      "Validation score: 0.915162\n",
      "Iteration 4, loss = 356.11294425\n",
      "Validation score: 0.913517\n",
      "Iteration 5, loss = 350.75659481\n",
      "Validation score: 0.916669\n",
      "Iteration 6, loss = 352.74777847\n",
      "Validation score: 0.912908\n",
      "Iteration 7, loss = 348.73032167\n",
      "Validation score: 0.905030\n",
      "Iteration 8, loss = 355.75492122\n",
      "Validation score: 0.916073\n",
      "Iteration 9, loss = 346.63812487\n",
      "Validation score: 0.910881\n",
      "Iteration 10, loss = 346.65057995\n",
      "Validation score: 0.917029\n",
      "Iteration 11, loss = 344.96513343\n",
      "Validation score: 0.911771\n",
      "Iteration 12, loss = 343.28964006\n",
      "Validation score: 0.915064\n",
      "Iteration 13, loss = 342.38614522\n",
      "Validation score: 0.917365\n",
      "Iteration 14, loss = 345.56449837\n",
      "Validation score: 0.903331\n",
      "Iteration 15, loss = 351.44812042\n",
      "Validation score: 0.910155\n",
      "Iteration 16, loss = 341.80958294\n",
      "Validation score: 0.915936\n",
      "Iteration 17, loss = 341.80853920\n",
      "Validation score: 0.913732\n",
      "Iteration 18, loss = 346.63825835\n",
      "Validation score: 0.916550\n",
      "Iteration 19, loss = 340.23027188\n",
      "Validation score: 0.902290\n",
      "Iteration 20, loss = 341.74275672\n",
      "Validation score: 0.912319\n",
      "Iteration 21, loss = 342.14433145\n",
      "Validation score: 0.914689\n",
      "Iteration 22, loss = 336.29356435\n",
      "Validation score: 0.916865\n",
      "Iteration 23, loss = 338.90157567\n",
      "Validation score: 0.916807\n",
      "Iteration 24, loss = 339.15099034\n",
      "Validation score: 0.917057\n",
      "Validation score did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "MLPRegressor(alpha=0.01, batch_size=200, early_stopping=True,\n",
       "             hidden_layer_sizes=(128, 256, 256, 128, 64, 16), random_state=42,\n",
       "             validation_fraction=0.2, verbose=True)"
      ]
     },
     "execution_count": 890,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# current best params\n",
    "mlp = MLPRegressor(activation = \"relu\", alpha = 0.01, hidden_layer_sizes = (128,256,256,128,64,16),\n",
    "                   solver='adam', max_iter = 200, random_state = 42, verbose=True, early_stopping=True,\n",
    "                   validation_fraction=0.2, learning_rate_init=0.001, batch_size=200)\n",
    "mlp.fit(clean_X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 891,
   "id": "ee23c846",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T10:43:22.248052Z",
     "start_time": "2021-10-16T10:43:21.318058Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE (more accurate):  17.472911079782566\n",
      "r2 square: 0.9168742154213051\n"
     ]
    }
   ],
   "source": [
    "#Calculate Surface\n",
    "y_hat = mlp.predict(clean_X)\n",
    "print('MAE (more accurate): ',mean_absolute_error(y, y_hat))\n",
    "## look at MAE is more accurate\n",
    "print('r2 square:', r2_score(y, y_hat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4b76653",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.852746Z",
     "start_time": "2021-10-13T15:48:52.852746Z"
    }
   },
   "outputs": [],
   "source": [
    "# joblib.dump(mlp, 'models/mlp_current_best_impute.joblib')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92526e3b",
   "metadata": {},
   "source": [
    "### LightGBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2601,
   "id": "14d947cb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T15:45:25.647166Z",
     "start_time": "2021-10-16T15:45:25.638132Z"
    }
   },
   "outputs": [],
   "source": [
    "lgb_params = {'n_estimators': 975, \n",
    "              'reg_lambda': 0.01,\n",
    "              'max_depth': 25, \n",
    "              'learning_rate': 0.07336551102484597, \n",
    "              'subsample': 0.9780814357664563, \n",
    "              'num_leaves': 30,\n",
    "              'random_state':42}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2602,
   "id": "0843db3a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T15:45:32.754134Z",
     "start_time": "2021-10-16T15:45:26.244131Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMRegressor(learning_rate=0.07336551102484597, max_depth=25, n_estimators=975,\n",
       "              num_leaves=30, random_state=42, reg_lambda=0.01,\n",
       "              subsample=0.9780814357664563)"
      ]
     },
     "execution_count": 2602,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# current best params\n",
    "lgb = LGBMRegressor(**lgb_params)\n",
    "lgb.fit(clean_X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2629,
   "id": "fd693968",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:10:06.131820Z",
     "start_time": "2021-10-16T16:10:04.186823Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE (more accurate):  13.742429322623202\n",
      "r2 square: 0.9526661705590186\n"
     ]
    }
   ],
   "source": [
    "y_hat = lgb.predict(clean_X)\n",
    "print('MAE (more accurate): ',mean_absolute_error(y, y_hat))\n",
    "## look at MAE is more accurate\n",
    "print('r2 square:', r2_score(y, y_hat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2630,
   "id": "ed5e3133",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:10:10.838734Z",
     "start_time": "2021-10-16T16:10:10.691731Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['models/lgb_current_best.joblib']"
      ]
     },
     "execution_count": 2630,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# joblib.dump(lgb, 'models/lgb_current_best.joblib')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62fbcd83",
   "metadata": {},
   "source": [
    "### TabNet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "id": "21f80fa1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T06:53:14.085228Z",
     "start_time": "2021-10-16T06:53:14.079226Z"
    }
   },
   "outputs": [],
   "source": [
    "fixed_tab_best_params = {'mask_type': 'entmax',\n",
    "                         'n_d': 25,\n",
    "                         'n_a': 25,\n",
    "                         'n_steps': 3,\n",
    "                         'momentum': 0.010908117575012286,\n",
    "                         'gamma': 0.7,\n",
    "                         'n_shared': 2,\n",
    "                         'lambda_sparse': 0.00027473030620813675}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "id": "8757b4f6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T07:37:24.890036Z",
     "start_time": "2021-10-16T06:53:16.276228Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device used : cuda\n",
      "No early stopping will be performed, last training weights will be used.\n",
      "epoch 0  | loss: 8953.12903|  0:00:33s\n",
      "epoch 1  | loss: 852.49627|  0:00:58s\n",
      "epoch 2  | loss: 814.31683|  0:01:24s\n",
      "epoch 3  | loss: 792.81048|  0:01:50s\n",
      "epoch 4  | loss: 770.55513|  0:02:16s\n",
      "epoch 5  | loss: 765.43522|  0:02:41s\n",
      "epoch 6  | loss: 751.48773|  0:03:07s\n",
      "epoch 7  | loss: 771.54519|  0:03:31s\n",
      "epoch 8  | loss: 740.09581|  0:03:56s\n",
      "epoch 9  | loss: 733.72821|  0:04:21s\n",
      "epoch 10 | loss: 748.211 |  0:04:47s\n",
      "epoch 11 | loss: 736.82183|  0:05:13s\n",
      "epoch 12 | loss: 742.05901|  0:05:42s\n",
      "epoch 13 | loss: 733.09496|  0:06:11s\n",
      "epoch 14 | loss: 732.48891|  0:06:42s\n",
      "epoch 15 | loss: 740.575 |  0:07:07s\n",
      "epoch 16 | loss: 735.54345|  0:07:37s\n",
      "epoch 17 | loss: 721.28023|  0:07:59s\n",
      "epoch 18 | loss: 729.66604|  0:08:05s\n",
      "epoch 19 | loss: 714.50833|  0:08:11s\n",
      "epoch 20 | loss: 721.77896|  0:08:18s\n",
      "epoch 21 | loss: 714.39629|  0:08:46s\n",
      "epoch 22 | loss: 722.22848|  0:09:11s\n",
      "epoch 23 | loss: 712.15957|  0:09:36s\n",
      "epoch 24 | loss: 705.44512|  0:10:01s\n",
      "epoch 25 | loss: 708.09923|  0:10:30s\n",
      "epoch 26 | loss: 703.12696|  0:10:58s\n",
      "epoch 27 | loss: 707.21793|  0:11:29s\n",
      "epoch 28 | loss: 705.9106|  0:11:55s\n",
      "epoch 29 | loss: 694.68936|  0:12:20s\n",
      "epoch 30 | loss: 700.87022|  0:12:44s\n",
      "epoch 31 | loss: 716.3981|  0:13:11s\n",
      "epoch 32 | loss: 691.63925|  0:13:35s\n",
      "epoch 33 | loss: 692.69606|  0:14:02s\n",
      "epoch 34 | loss: 701.88751|  0:14:27s\n",
      "epoch 35 | loss: 690.54492|  0:14:53s\n",
      "epoch 36 | loss: 691.28716|  0:15:23s\n",
      "epoch 37 | loss: 691.62379|  0:15:51s\n",
      "epoch 38 | loss: 692.16076|  0:16:17s\n",
      "epoch 39 | loss: 684.27196|  0:16:45s\n",
      "epoch 40 | loss: 687.81668|  0:17:12s\n",
      "epoch 41 | loss: 677.19113|  0:17:39s\n",
      "epoch 42 | loss: 681.67597|  0:18:02s\n",
      "epoch 43 | loss: 676.0142|  0:18:09s\n",
      "epoch 44 | loss: 679.84201|  0:18:15s\n",
      "epoch 45 | loss: 674.48857|  0:18:21s\n",
      "epoch 46 | loss: 676.14036|  0:18:42s\n",
      "epoch 47 | loss: 669.31486|  0:19:10s\n",
      "epoch 48 | loss: 670.04879|  0:19:36s\n",
      "epoch 49 | loss: 668.82035|  0:20:04s\n",
      "epoch 50 | loss: 660.84246|  0:20:31s\n",
      "epoch 51 | loss: 670.62756|  0:20:59s\n",
      "epoch 52 | loss: 661.31606|  0:21:26s\n",
      "epoch 53 | loss: 660.19258|  0:21:54s\n",
      "epoch 54 | loss: 654.09388|  0:22:24s\n",
      "epoch 55 | loss: 649.882 |  0:22:52s\n",
      "epoch 56 | loss: 648.81973|  0:23:20s\n",
      "epoch 57 | loss: 652.7906|  0:23:49s\n",
      "epoch 58 | loss: 652.61544|  0:24:17s\n",
      "epoch 59 | loss: 650.78513|  0:24:45s\n",
      "epoch 60 | loss: 646.05664|  0:25:13s\n",
      "epoch 61 | loss: 653.3171|  0:25:41s\n",
      "epoch 62 | loss: 647.61954|  0:26:08s\n",
      "epoch 63 | loss: 651.46016|  0:26:36s\n",
      "epoch 64 | loss: 643.71479|  0:27:04s\n",
      "epoch 65 | loss: 637.70394|  0:27:32s\n",
      "epoch 66 | loss: 639.8815|  0:28:00s\n",
      "epoch 67 | loss: 637.08897|  0:28:28s\n",
      "epoch 68 | loss: 631.35882|  0:28:56s\n",
      "epoch 69 | loss: 637.44037|  0:29:25s\n",
      "epoch 70 | loss: 639.53357|  0:29:54s\n",
      "epoch 71 | loss: 630.20927|  0:30:21s\n",
      "epoch 72 | loss: 619.02502|  0:30:49s\n",
      "epoch 73 | loss: 630.20647|  0:31:18s\n",
      "epoch 74 | loss: 629.56747|  0:31:47s\n",
      "epoch 75 | loss: 625.35182|  0:32:15s\n",
      "epoch 76 | loss: 616.51182|  0:32:42s\n",
      "epoch 77 | loss: 621.1279|  0:33:10s\n",
      "epoch 78 | loss: 626.442 |  0:33:38s\n",
      "epoch 79 | loss: 617.54879|  0:34:07s\n",
      "epoch 80 | loss: 615.93781|  0:34:36s\n",
      "epoch 81 | loss: 610.93298|  0:35:03s\n",
      "epoch 82 | loss: 609.96735|  0:35:31s\n",
      "epoch 83 | loss: 610.75022|  0:35:59s\n",
      "epoch 84 | loss: 608.97901|  0:36:27s\n",
      "epoch 85 | loss: 604.28721|  0:36:55s\n",
      "epoch 86 | loss: 607.05691|  0:37:22s\n",
      "epoch 87 | loss: 617.39979|  0:37:50s\n",
      "epoch 88 | loss: 602.43495|  0:38:18s\n",
      "epoch 89 | loss: 599.22233|  0:38:45s\n",
      "epoch 90 | loss: 602.12066|  0:39:26s\n",
      "epoch 91 | loss: 598.58312|  0:39:54s\n",
      "epoch 92 | loss: 599.88362|  0:40:22s\n",
      "epoch 93 | loss: 597.34844|  0:40:51s\n",
      "epoch 94 | loss: 592.66699|  0:41:20s\n",
      "epoch 95 | loss: 595.19304|  0:41:48s\n",
      "epoch 96 | loss: 592.30365|  0:42:17s\n",
      "epoch 97 | loss: 596.72432|  0:42:44s\n",
      "epoch 98 | loss: 587.99535|  0:43:13s\n",
      "epoch 99 | loss: 590.58971|  0:43:42s\n"
     ]
    }
   ],
   "source": [
    "# current best params\n",
    "tab = TabNetRegressor(**fixed_tab_best_params)\n",
    "tab.fit(np.array(clean_X),np.expand_dims(y, axis=-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "id": "ca71eddd",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T07:42:11.883037Z",
     "start_time": "2021-10-16T07:41:56.422037Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE (more accurate):  16.404377488337662\n",
      "r2 square: 0.9307090121817564\n"
     ]
    }
   ],
   "source": [
    "y_hat = tab.predict(np.array(clean_X))\n",
    "print('MAE (more accurate): ',mean_absolute_error(y, y_hat))\n",
    "## look at MAE is more accurate\n",
    "print('r2 square:', r2_score(y, y_hat))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06daefee",
   "metadata": {},
   "source": [
    "## SVM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "806bdd3a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.866745Z",
     "start_time": "2021-10-13T15:48:52.865747Z"
    }
   },
   "outputs": [],
   "source": [
    "svr_best_params = {'kernel': 'rbf',\n",
    "                  'gamma': 'scale',\n",
    "                  'C': 0.1}\n",
    "svr = SVR(**svr_best_params)\n",
    "svr.fit(clean_X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0abc92d5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.867746Z",
     "start_time": "2021-10-13T15:48:52.867746Z"
    }
   },
   "outputs": [],
   "source": [
    "y_hat = svr.predict(clean_X)\n",
    "print('MAE (more accurate): ',mean_absolute_error(y, y_hat))\n",
    "print('r2 square:', r2_score(y, y_hat))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08c0af6d",
   "metadata": {},
   "source": [
    "### XGBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0426586f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.868745Z",
     "start_time": "2021-10-13T15:48:52.868745Z"
    }
   },
   "outputs": [],
   "source": [
    "xg_best_params = {'booster': 'dart', \n",
    "                  'lambda': 0.1978947368034619, \n",
    "                  'alpha': 0.0009102278884164788, \n",
    "                  'max_depth': 4, \n",
    "                  'eta': 0.6864948971995669, \n",
    "                  'gamma': 5.2062504664042595e-05, \n",
    "                  'grow_policy': 'lossguide', \n",
    "                  'sample_type': 'weighted', \n",
    "                  'normalize_type': 'forest', \n",
    "                  'rate_drop': 4.887412967843037e-08, \n",
    "                  'skip_drop': 2.0347272399655997e-05}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e3ef955",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.869747Z",
     "start_time": "2021-10-13T15:48:52.869747Z"
    }
   },
   "outputs": [],
   "source": [
    "xg = XGBRegressor(**xg_best_params)\n",
    "xg.fit(clean_X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66eea6f1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.871747Z",
     "start_time": "2021-10-13T15:48:52.871747Z"
    }
   },
   "outputs": [],
   "source": [
    "y_hat = xg.predict(clean_X)\n",
    "print('MAE: ',mean_absolute_error(y, y_hat))\n",
    "print('r2 square:', r2_score(y, y_hat))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed9c4076",
   "metadata": {},
   "source": [
    "### RandomForest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "accc17fd",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.873746Z",
     "start_time": "2021-10-13T15:48:52.873746Z"
    }
   },
   "outputs": [],
   "source": [
    "rf_best_params = {'n_estimators': 934, 'max_depth': 12}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a59d5d2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.874747Z",
     "start_time": "2021-10-13T15:48:52.874747Z"
    }
   },
   "outputs": [],
   "source": [
    "rf = RandomForestRegressor(**rf_best_params)\n",
    "rf.fit(clean_X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b22a7fd",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.876746Z",
     "start_time": "2021-10-13T15:48:52.876746Z"
    }
   },
   "outputs": [],
   "source": [
    "y_hat = rf.predict(clean_X)\n",
    "print('MSE: ',mean_squared_error(y, y_hat))\n",
    "print('r2 square:', r2_score(y, y_hat))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f9994ff",
   "metadata": {},
   "source": [
    "### Tensorflow NN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b26a7df8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-17T09:21:55.513755Z",
     "start_time": "2021-10-17T09:21:52.117223Z"
    }
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import InputLayer, Dense, Dropout, Conv1D, AveragePooling1D, AveragePooling2D\n",
    "from tensorflow.keras.layers import Input, Flatten, Reshape, Lambda, BatchNormalization, Conv2D\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from tensorflow.keras.regularizers import L2\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "import tensorflow as tf\n",
    "import tensorflow_addons as tfa\n",
    "import tensorflow.keras.backend as K\n",
    "\n",
    "from sklearn.model_selection import KFold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b305cade",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.879747Z",
     "start_time": "2021-10-13T15:48:52.879747Z"
    }
   },
   "outputs": [],
   "source": [
    "tf.keras.backend.clear_session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "419e3478",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.880749Z",
     "start_time": "2021-10-13T15:48:52.880749Z"
    }
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE = 200\n",
    "EPOCHS = 60\n",
    "INIT_LR = 0.001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6be89f18",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.882747Z",
     "start_time": "2021-10-13T15:48:52.882747Z"
    }
   },
   "outputs": [],
   "source": [
    "# sklearn use GlorotUniform (from source code)\n",
    "xavier = tf.keras.initializers.GlorotUniform(seed=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "367429bb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.884747Z",
     "start_time": "2021-10-13T15:48:52.884747Z"
    }
   },
   "outputs": [],
   "source": [
    "optimizer = Adam(learning_rate=INIT_LR, beta_1=0.9, beta_2=0.999, epsilon=1e-8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0162c0d0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.885747Z",
     "start_time": "2021-10-13T15:48:52.885747Z"
    }
   },
   "outputs": [],
   "source": [
    "# , kernel_regularizer=L2(0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97990774",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.886746Z",
     "start_time": "2021-10-13T15:48:52.886746Z"
    }
   },
   "outputs": [],
   "source": [
    "#(128,256,256,128,64,16)\n",
    "model = Sequential([\n",
    "    InputLayer(input_shape=(clean_X.shape[1],), dtype='float32'),\n",
    "    Dense(128, activation='relu', kernel_initializer=xavier),\n",
    "    Dense(256, activation='relu', kernel_initializer=xavier),\n",
    "    Dense(256, activation='relu', kernel_initializer=xavier, kernel_regularizer=L2(0.01)),\n",
    "    Dense(128, activation='relu', kernel_initializer=xavier),\n",
    "    Dense(64, activation='relu', kernel_initializer=xavier, kernel_regularizer=L2(0.01)),\n",
    "    Dense(16, activation='relu', kernel_initializer=xavier),\n",
    "    Dense(1, activation='linear', kernel_initializer=xavier, kernel_regularizer=L2(0.01)),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "092fb766",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.888746Z",
     "start_time": "2021-10-13T15:48:52.888746Z"
    }
   },
   "outputs": [],
   "source": [
    "# callbacks\n",
    "early_stopping = EarlyStopping(patience=10, restore_best_weights=True, monitor='val_r2_metric', mode='max')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "908087d2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.890747Z",
     "start_time": "2021-10-13T15:48:52.890747Z"
    }
   },
   "outputs": [],
   "source": [
    "def r2_metric(y_true, y_pred):\n",
    "    numerator = K.sum((y_true - y_pred) ** 2, axis=0)\n",
    "    denominator = K.sum((y_true - K.mean(y_pred, axis=0)) ** 2, axis=0)\n",
    "    return 1-(numerator/denominator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e67aae14",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.892746Z",
     "start_time": "2021-10-13T15:48:52.892746Z"
    }
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer=optimizer, loss='mean_squared_error', metrics=[r2_metric])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8184dd5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.893746Z",
     "start_time": "2021-10-13T15:48:52.893746Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model.fit(clean_X, y,\n",
    "          batch_size=BATCH_SIZE,\n",
    "          epochs=EPOCHS,\n",
    "          verbose=1,\n",
    "          validation_split=0.2,\n",
    "          callbacks=[early_stopping]\n",
    "         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e551cbde",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.895747Z",
     "start_time": "2021-10-13T15:48:52.895747Z"
    }
   },
   "outputs": [],
   "source": [
    "y_hat = model.predict(clean_X).flatten()\n",
    "print('MAE (more accurate): ',mean_absolute_error(y, y_hat))\n",
    "## look at MAE is more accurate\n",
    "print('r2 square:', r2_score(y, y_hat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a5ab8df",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.896747Z",
     "start_time": "2021-10-13T15:48:52.896747Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# TF model with KFold cross-validation\n",
    "kfold = KFold(n_splits=5, shuffle=True)\n",
    "cv_loss_per_fold = []\n",
    "train_loss_per_fold = []\n",
    "fold_no = 1\n",
    "inputs = np.array(clean_X)\n",
    "targets = np.array(y)\n",
    "for train, test in kfold.split(inputs, targets):\n",
    "    model = Sequential([\n",
    "    InputLayer(input_shape=(clean_X.shape[1],), dtype='float32'),\n",
    "    Dense(128, activation='relu'),\n",
    "    Dense(256, activation='relu'),\n",
    "    Dropout(0.5),\n",
    "    Dense(256, activation='relu'),\n",
    "    Dropout(0.5),\n",
    "    Dense(64, activation='relu'),\n",
    "    Dense(16, activation='relu'),\n",
    "    Dense(1, activation='linear')\n",
    "    ])\n",
    "    \n",
    "    model.compile(optimizer=optimizer, loss='mean_absolute_error', metrics=[])\n",
    "    \n",
    "    print('------------------------------------------------------------------------')\n",
    "    print(f'Training for fold {fold_no} ...')\n",
    "    \n",
    "    H = model.fit(inputs[train], targets[train],\n",
    "          batch_size=BATCH_SIZE,\n",
    "          epochs=EPOCHS,\n",
    "          verbose=1,\n",
    "#           validation_split=0.2,\n",
    "#           callbacks=[early_stopping]\n",
    "          )\n",
    "    \n",
    "    # Generate generalization metrics\n",
    "    scores = model.evaluate(inputs[test], targets[test], verbose=0)\n",
    "    print(f'Training loss for fold {fold_no}: {H.history[\"loss\"][-1]}')\n",
    "    print(f'CV Score for fold {fold_no}: {model.metrics_names[0]} of {scores}')\n",
    "    cv_loss_per_fold.append(scores)\n",
    "    train_loss_per_fold.append(H.history[\"loss\"][-1])\n",
    "    fold_no += 1\n",
    "    \n",
    "# == Provide average scores ==\n",
    "print('------------------------------------------------------------------------')\n",
    "print('CV Score per fold')\n",
    "for i in range(0, len(cv_loss_per_fold)):\n",
    "    print('------------------------------------------------------------------------')\n",
    "    print(f'> Fold {i+1} - Loss: {cv_loss_per_fold[i]}')\n",
    "print('------------------------------------------------------------------------')\n",
    "print('Average scores for all folds:')\n",
    "print(f'> Train Average Loss: {np.mean(train_loss_per_fold)}')\n",
    "print(f'> CV Average Loss: {np.mean(cv_loss_per_fold)}')\n",
    "print('------------------------------------------------------------------------')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53587bcf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.897747Z",
     "start_time": "2021-10-13T15:48:52.897747Z"
    }
   },
   "outputs": [],
   "source": [
    "test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3edb7b11",
   "metadata": {},
   "source": [
    "### 1D-CNN\n",
    "not working well, or incorrect implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dabdd7a0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.898746Z",
     "start_time": "2021-10-13T15:48:52.898746Z"
    }
   },
   "outputs": [],
   "source": [
    "def r2_metric(y_true, y_pred):\n",
    "    numerator = K.sum((y_true - y_pred) ** 2, axis=0)\n",
    "    denominator = K.sum((y_true - K.mean(y_pred, axis=0)) ** 2, axis=0)\n",
    "    return 1-(numerator/denominator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc2ffb0e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.900746Z",
     "start_time": "2021-10-13T15:48:52.900746Z"
    }
   },
   "outputs": [],
   "source": [
    "cnn_X = np.expand_dims(clean_X, axis=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28c65075",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.901746Z",
     "start_time": "2021-10-13T15:48:52.901746Z"
    }
   },
   "outputs": [],
   "source": [
    "tf.keras.backend.clear_session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "440d068d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.903746Z",
     "start_time": "2021-10-13T15:48:52.903746Z"
    }
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE = 200\n",
    "EPOCHS = 60\n",
    "INIT_LR = 0.001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74fca842",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.905747Z",
     "start_time": "2021-10-13T15:48:52.905747Z"
    }
   },
   "outputs": [],
   "source": [
    "# sklearn use GlorotUniform (from source code)\n",
    "xavier = tf.keras.initializers.GlorotUniform(seed=42)\n",
    "optimizer = Adam(learning_rate=INIT_LR, beta_1=0.9, beta_2=0.999, epsilon=1e-8)\n",
    "\n",
    "# callbacks\n",
    "early_stopping = EarlyStopping(patience=10, restore_best_weights=True, monitor='val_r2_metric', mode='max')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6f61c7e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.906746Z",
     "start_time": "2021-10-13T15:48:52.906746Z"
    }
   },
   "outputs": [],
   "source": [
    "#(128,256,256,128,64,16)\n",
    "model = Sequential([\n",
    "    InputLayer(input_shape=(clean_X.shape[1],1), dtype='float32'),\n",
    "    Dense(128, activation='relu', kernel_initializer=xavier),\n",
    "    Conv1D(256, 3, activation='relu', padding='same'),\n",
    "    AveragePooling1D(),\n",
    "    Conv1D(256, 3, activation='relu', padding='same'),\n",
    "    Dropout(0.2),\n",
    "    Conv1D(128, 3, activation='relu', padding='same'),\n",
    "    Dropout(0.2),\n",
    "    Conv1D(64, 3, activation='relu', padding='same'),\n",
    "    AveragePooling1D(),\n",
    "    Dropout(0.2),\n",
    "    Flatten(),\n",
    "    Dense(16, activation='relu'),\n",
    "    Dense(1, activation='linear')\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f351162d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.907747Z",
     "start_time": "2021-10-13T15:48:52.907747Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "inputs = Input(shape=(clean_X.shape[1],), dtype='float32')\n",
    "x = Dense(4096, activation='relu')(inputs)\n",
    "x = Reshape((256, 16))(x)\n",
    "x = BatchNormalization()(x)\n",
    "x = Conv1D(512, 3, activation='relu', padding='same')(x)\n",
    "x = AveragePooling1D()(x)\n",
    "x = BatchNormalization()(x)\n",
    "x_s = Conv1D(512, 3, activation='relu', padding='same')(x)\n",
    "x = BatchNormalization()(x)\n",
    "x = Conv1D(512, 3, activation='relu', padding='same')(x_s)\n",
    "x = BatchNormalization()(x)\n",
    "x = Conv1D(512, 3, activation='relu', padding='same')(x)\n",
    "\n",
    "sum_layer = Lambda(lambda x: K.sum(x, axis=0, keepdims=False))([x, x_s])\n",
    "x = AveragePooling1D()(sum_layer)\n",
    "x = Flatten()(x)\n",
    "x = BatchNormalization()(x)\n",
    "x = Dense(64, activation='relu')(x)\n",
    "x = Dense(1, activation='linear')(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7368cbb7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.909745Z",
     "start_time": "2021-10-13T15:48:52.909745Z"
    }
   },
   "outputs": [],
   "source": [
    "model = tf.keras.models.Model(inputs=inputs, outputs=x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce4cb37f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.911746Z",
     "start_time": "2021-10-13T15:48:52.911746Z"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d0cc264",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.912747Z",
     "start_time": "2021-10-13T15:48:52.912747Z"
    }
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer=optimizer, loss='mean_squared_error', metrics=[r2_metric])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bfe6ae8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.913746Z",
     "start_time": "2021-10-13T15:48:52.913746Z"
    }
   },
   "outputs": [],
   "source": [
    "cnn_X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e661a257",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.915747Z",
     "start_time": "2021-10-13T15:48:52.915747Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model.fit(cnn_X.reshape((clean_X.shape[0], 1,clean_X.shape[1])), y,\n",
    "          batch_size=BATCH_SIZE,\n",
    "          epochs=EPOCHS,\n",
    "          verbose=1,\n",
    "          validation_split=0.2\n",
    "#           callbacks=[early_stopping]\n",
    "         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81a47690",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.916746Z",
     "start_time": "2021-10-13T15:48:52.916746Z"
    }
   },
   "outputs": [],
   "source": [
    "cnn_X.reshape((clean_X.shape[0], 1,clean_X.shape[1])).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ec29a57",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.918747Z",
     "start_time": "2021-10-13T15:48:52.918747Z"
    }
   },
   "outputs": [],
   "source": [
    "cnn_X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "747bfef5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.919746Z",
     "start_time": "2021-10-13T15:48:52.919746Z"
    }
   },
   "outputs": [],
   "source": [
    "y_hat = model.predict(cnn_X.reshape((clean_X.shape[0], 1,clean_X.shape[1]))).flatten()\n",
    "print('MAE (more accurate): ',mean_absolute_error(y, y_hat))\n",
    "## look at MAE is more accurate\n",
    "print('r2 square:', r2_score(y, y_hat))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58c3015e",
   "metadata": {},
   "source": [
    "## Submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2619,
   "id": "3bd29492",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:08:36.236275Z",
     "start_time": "2021-10-16T16:08:36.087299Z"
    }
   },
   "outputs": [],
   "source": [
    "test_df = pd.read_csv('test_extra_2.csv')\n",
    "test_new_feats = pd.read_csv('test_feats_from_di_cnn.csv')\n",
    "test_comb_feats = pd.read_csv('test_feats_from_comb_cnn.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2620,
   "id": "dbe91501",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:08:36.500302Z",
     "start_time": "2021-10-16T16:08:36.480272Z"
    }
   },
   "outputs": [],
   "source": [
    "test_df = pd.concat([test_df, test_new_feats, test_comb_feats], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2621,
   "id": "753c0614",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:08:36.872275Z",
     "start_time": "2021-10-16T16:08:36.847270Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 17000 entries, 0 to 16999\n",
      "Data columns (total 40 columns):\n",
      " #   Column                                         Non-Null Count  Dtype  \n",
      "---  ------                                         --------------  -----  \n",
      " 0   MOFname                                        17000 non-null  object \n",
      " 1   volume [A^3]                                   17000 non-null  float64\n",
      " 2   weight [u]                                     17000 non-null  float64\n",
      " 3   surface_area [m^2/g]                           17000 non-null  float64\n",
      " 4   void_fraction                                  17000 non-null  float64\n",
      " 5   void_volume [cm^3/g]                           17000 non-null  float64\n",
      " 6   functional_groups                              17000 non-null  object \n",
      " 7   metal_linker                                   17000 non-null  int64  \n",
      " 8   organic_linker1                                17000 non-null  int64  \n",
      " 9   organic_linker2                                17000 non-null  int64  \n",
      " 10  topology                                       17000 non-null  object \n",
      " 11  CO2/N2_selectivity                             17000 non-null  float64\n",
      " 12  heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]  17000 non-null  float64\n",
      " 13  _cell_length_a                                 17000 non-null  float64\n",
      " 14  _cell_length_b                                 17000 non-null  float64\n",
      " 15  _cell_length_c                                 17000 non-null  float64\n",
      " 16  _cell_angle_alpha                              17000 non-null  float64\n",
      " 17  _cell_angle_beta                               17000 non-null  float64\n",
      " 18  _cell_angle_gamma                              17000 non-null  float64\n",
      " 19  n_atoms                                        17000 non-null  int64  \n",
      " 20  mol_avg_mass                                   17000 non-null  float64\n",
      " 21  charges                                        17000 non-null  int64  \n",
      " 22  mol_avg_radius                                 17000 non-null  float64\n",
      " 23  atoms_volume                                   17000 non-null  float64\n",
      " 24  atoms_area                                     17000 non-null  float64\n",
      " 25  di_cnn_1                                       17000 non-null  float64\n",
      " 26  di_cnn_2                                       17000 non-null  float64\n",
      " 27  di_cnn_4                                       17000 non-null  float64\n",
      " 28  di_cnn_5                                       17000 non-null  float64\n",
      " 29  di_cnn_6                                       17000 non-null  float64\n",
      " 30  di_cnn_8                                       17000 non-null  float64\n",
      " 31  di_cnn_10                                      17000 non-null  float64\n",
      " 32  di_cnn_12                                      17000 non-null  float64\n",
      " 33  di_cnn_13                                      17000 non-null  float64\n",
      " 34  di_cnn_14                                      17000 non-null  float64\n",
      " 35  di_cnn_15                                      17000 non-null  float64\n",
      " 36  comb_cnn_0                                     17000 non-null  float64\n",
      " 37  comb_cnn_1                                     17000 non-null  float64\n",
      " 38  comb_cnn_2                                     17000 non-null  float64\n",
      " 39  comb_cnn_3                                     17000 non-null  float64\n",
      "dtypes: float64(32), int64(5), object(3)\n",
      "memory usage: 5.2+ MB\n"
     ]
    }
   ],
   "source": [
    "test_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2622,
   "id": "e3eaea97",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:08:39.542897Z",
     "start_time": "2021-10-16T16:08:39.516897Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing values in Functional_groups: 0\n",
      "Missing values in Heat adsorption: 0, Inf value in Heat Adsorption: 0\n",
      "0 values in void_volume: 0\n",
      "0 values in void_fraction: 0, -1 value: 0\n",
      "0 values in surface_area: 0, -1 value: 0\n"
     ]
    }
   ],
   "source": [
    "check_all_problems(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2623,
   "id": "53d2b8ff",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:08:44.628467Z",
     "start_time": "2021-10-16T16:08:44.378463Z"
    }
   },
   "outputs": [],
   "source": [
    "# preprocess\n",
    "\n",
    "test_df = test_df[['MOFname', 'volume [A^3]', 'weight [u]', 'surface_area [m^2/g]',\n",
    "       'void_fraction', 'void_volume [cm^3/g]', 'functional_groups',\n",
    "       'metal_linker', 'organic_linker1', 'organic_linker2', 'topology',\n",
    "       'CO2/N2_selectivity', 'heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]',\n",
    "       'n_atoms', 'mol_avg_mass', 'charges', 'mol_avg_radius', 'atoms_volume', 'atoms_area','di_cnn_1',\n",
    "         'di_cnn_2', 'di_cnn_4', 'di_cnn_5', 'di_cnn_6', 'di_cnn_8', 'di_cnn_10',\n",
    "         'di_cnn_12', 'di_cnn_13', 'di_cnn_14', 'di_cnn_15', 'comb_cnn_0', 'comb_cnn_1', 'comb_cnn_2', 'comb_cnn_3']]\n",
    "\n",
    "test_df = test_df.drop(['functional_groups'], axis=1)\n",
    "\n",
    "test_df.insert(\n",
    "    loc=2,\n",
    "    column=\"density [g/cm^3]\",\n",
    "    value=(test_df[\"weight [u]\"] / test_df[\"volume [A^3]\"]) * 1.66054,\n",
    ")\n",
    "\n",
    "test_df.insert(\n",
    "    loc=2,\n",
    "    column=\"volume [cm^3/g]\",\n",
    "    value=test_df['volume [A^3]'] / (test_df['weight [u]'] * 1.66054),\n",
    ")\n",
    "\n",
    "test_df['specific_heat'] = (test_df['heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]']*4.1868*1000)/(test_df['weight [u]']*65)\n",
    "\n",
    "# drop name and those ratio we used\n",
    "test_X = test_df.drop(['MOFname'], axis = 1)\n",
    "test_X = test_X.rename(columns={'volume [A^3]':'volume', 'volume [cm^3/g]':'volume/g', 'density [g/cm^3]':'density',\n",
    "         'weight [u]':'weight', 'surface_area [m^2/g]':'surface_area', 'void_volume [cm^3/g]':'void_volume',\n",
    "         'heat_adsorption_CO2_P0.15bar_T298K [kcal/mol]':'heat_adsorption'\n",
    "         })\n",
    "\n",
    "# one hot\n",
    "# ## functioncal groups\n",
    "# test_encoded_fg = fg_lb.transform(test_X['functional_groups'])\n",
    "# test_X = test_X.drop(['functional_groups'], axis = 1)\n",
    "# # fill one-hot encoded functional_groups to X\n",
    "# for i in range(total_fg):\n",
    "#     test_X[f'functional_groups_{i}'] = test_encoded_fg[:, i]\n",
    "\n",
    "## metal_linker\n",
    "test_encoded_metal = metal_lb.transform(test_X['metal_linker'])\n",
    "test_X = test_X.drop(['metal_linker'], axis = 1)\n",
    "# fill one-hot encoded metal_linker to X\n",
    "for i in range(total_metal):\n",
    "    test_X[f'metal_linker_{i}'] = test_encoded_metal[:, i]\n",
    "\n",
    "## organic_linker\n",
    "test_encoded_organic1 = organic_lb.transform(test_X['organic_linker1'])\n",
    "test_encoded_organic2 = organic_lb.transform(test_X['organic_linker2'])\n",
    "test_X = test_X.drop(['organic_linker1', 'organic_linker2'], axis = 1)\n",
    "\n",
    "# fill one-hot encoded version to X\n",
    "# linker 1\n",
    "for i in range(total_organic):\n",
    "    test_X[f'organic_linker1_{i}'] = test_encoded_organic1[:, i]\n",
    "\n",
    "# linker 2\n",
    "for i in range(total_organic):\n",
    "    test_X[f'organic_linker2_{i}'] = test_encoded_organic2[:, i]\n",
    "\n",
    "## topology\n",
    "test_encoded_topology = topology_lb.transform(test_X['topology'])\n",
    "test_X = test_X.drop(['topology'], axis = 1)\n",
    "# fill one-hot encoded functional_groups to X\n",
    "for i in range(total_topology):\n",
    "    test_X[f'topology_{i}'] = test_encoded_topology[:, i]\n",
    "    \n",
    "# feature scaling\n",
    "# test_scaled_feats = mm_scaler.transform(test_X[feats_to_scale])\n",
    "test_scaled_feats = rb_scaler.transform(test_X[feats_to_scale])\n",
    "for i, feat in enumerate(feats_to_scale):\n",
    "    test_X[feat] = test_scaled_feats[:, i]\n",
    "\n",
    "## 1 scaler for 1 feat (same result)\n",
    "# for feat in feats_to_scale:\n",
    "#     scaled_feat = scalers[feat].transform(test_X[[feat]])\n",
    "#     test_X[feat] = scaled_feat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2624,
   "id": "c725c5be",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:08:45.851464Z",
     "start_time": "2021-10-16T16:08:45.819467Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>volume</th>\n",
       "      <th>volume/g</th>\n",
       "      <th>density</th>\n",
       "      <th>weight</th>\n",
       "      <th>surface_area</th>\n",
       "      <th>void_fraction</th>\n",
       "      <th>void_volume</th>\n",
       "      <th>CO2/N2_selectivity</th>\n",
       "      <th>heat_adsorption</th>\n",
       "      <th>n_atoms</th>\n",
       "      <th>...</th>\n",
       "      <th>topology_1</th>\n",
       "      <th>topology_2</th>\n",
       "      <th>topology_3</th>\n",
       "      <th>topology_4</th>\n",
       "      <th>topology_5</th>\n",
       "      <th>topology_6</th>\n",
       "      <th>topology_7</th>\n",
       "      <th>topology_8</th>\n",
       "      <th>topology_9</th>\n",
       "      <th>topology_10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.479628</td>\n",
       "      <td>-0.272893</td>\n",
       "      <td>0.314058</td>\n",
       "      <td>-0.460370</td>\n",
       "      <td>-0.614563</td>\n",
       "      <td>-0.704325</td>\n",
       "      <td>-0.513748</td>\n",
       "      <td>0.841859</td>\n",
       "      <td>0.753026</td>\n",
       "      <td>-0.302326</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.945696</td>\n",
       "      <td>-0.807447</td>\n",
       "      <td>1.273569</td>\n",
       "      <td>2.466234</td>\n",
       "      <td>-0.451899</td>\n",
       "      <td>-0.222836</td>\n",
       "      <td>-0.399283</td>\n",
       "      <td>-0.064565</td>\n",
       "      <td>-0.481679</td>\n",
       "      <td>0.558140</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.287257</td>\n",
       "      <td>0.548750</td>\n",
       "      <td>-0.446130</td>\n",
       "      <td>-0.510708</td>\n",
       "      <td>0.956839</td>\n",
       "      <td>0.381305</td>\n",
       "      <td>0.483264</td>\n",
       "      <td>-0.329196</td>\n",
       "      <td>-0.530118</td>\n",
       "      <td>-0.581395</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.120337</td>\n",
       "      <td>0.943906</td>\n",
       "      <td>-0.672449</td>\n",
       "      <td>-0.262907</td>\n",
       "      <td>1.180306</td>\n",
       "      <td>0.698967</td>\n",
       "      <td>0.957860</td>\n",
       "      <td>-0.501135</td>\n",
       "      <td>-0.490294</td>\n",
       "      <td>-0.186047</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.455792</td>\n",
       "      <td>0.798101</td>\n",
       "      <td>-0.595773</td>\n",
       "      <td>0.077741</td>\n",
       "      <td>0.664925</td>\n",
       "      <td>0.626321</td>\n",
       "      <td>0.813210</td>\n",
       "      <td>-0.333562</td>\n",
       "      <td>-0.371552</td>\n",
       "      <td>0.093023</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16995</th>\n",
       "      <td>14.883110</td>\n",
       "      <td>6.775274</td>\n",
       "      <td>-1.708149</td>\n",
       "      <td>3.453108</td>\n",
       "      <td>2.525318</td>\n",
       "      <td>2.492180</td>\n",
       "      <td>8.834429</td>\n",
       "      <td>-0.752689</td>\n",
       "      <td>-1.772980</td>\n",
       "      <td>3.558140</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16996</th>\n",
       "      <td>1.407069</td>\n",
       "      <td>2.165319</td>\n",
       "      <td>-1.115874</td>\n",
       "      <td>0.238821</td>\n",
       "      <td>1.483792</td>\n",
       "      <td>1.602959</td>\n",
       "      <td>2.779438</td>\n",
       "      <td>-0.642975</td>\n",
       "      <td>-1.437719</td>\n",
       "      <td>0.162791</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16997</th>\n",
       "      <td>1.211109</td>\n",
       "      <td>2.360628</td>\n",
       "      <td>-1.164993</td>\n",
       "      <td>0.062301</td>\n",
       "      <td>1.649838</td>\n",
       "      <td>1.398200</td>\n",
       "      <td>2.669157</td>\n",
       "      <td>-0.746253</td>\n",
       "      <td>-1.480803</td>\n",
       "      <td>0.023256</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16998</th>\n",
       "      <td>1.217125</td>\n",
       "      <td>2.786262</td>\n",
       "      <td>-1.258839</td>\n",
       "      <td>-0.045891</td>\n",
       "      <td>1.676200</td>\n",
       "      <td>1.672998</td>\n",
       "      <td>3.387029</td>\n",
       "      <td>-0.738450</td>\n",
       "      <td>-1.571499</td>\n",
       "      <td>-0.069767</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16999</th>\n",
       "      <td>11.484388</td>\n",
       "      <td>4.749416</td>\n",
       "      <td>-1.543962</td>\n",
       "      <td>3.522421</td>\n",
       "      <td>2.088765</td>\n",
       "      <td>2.208162</td>\n",
       "      <td>6.098326</td>\n",
       "      <td>-0.765526</td>\n",
       "      <td>-1.757809</td>\n",
       "      <td>3.604651</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>17000 rows × 163 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          volume  volume/g   density    weight  surface_area  void_fraction  \\\n",
       "0      -0.479628 -0.272893  0.314058 -0.460370     -0.614563      -0.704325   \n",
       "1       0.945696 -0.807447  1.273569  2.466234     -0.451899      -0.222836   \n",
       "2      -0.287257  0.548750 -0.446130 -0.510708      0.956839       0.381305   \n",
       "3       0.120337  0.943906 -0.672449 -0.262907      1.180306       0.698967   \n",
       "4       0.455792  0.798101 -0.595773  0.077741      0.664925       0.626321   \n",
       "...          ...       ...       ...       ...           ...            ...   \n",
       "16995  14.883110  6.775274 -1.708149  3.453108      2.525318       2.492180   \n",
       "16996   1.407069  2.165319 -1.115874  0.238821      1.483792       1.602959   \n",
       "16997   1.211109  2.360628 -1.164993  0.062301      1.649838       1.398200   \n",
       "16998   1.217125  2.786262 -1.258839 -0.045891      1.676200       1.672998   \n",
       "16999  11.484388  4.749416 -1.543962  3.522421      2.088765       2.208162   \n",
       "\n",
       "       void_volume  CO2/N2_selectivity  heat_adsorption   n_atoms  ...  \\\n",
       "0        -0.513748            0.841859         0.753026 -0.302326  ...   \n",
       "1        -0.399283           -0.064565        -0.481679  0.558140  ...   \n",
       "2         0.483264           -0.329196        -0.530118 -0.581395  ...   \n",
       "3         0.957860           -0.501135        -0.490294 -0.186047  ...   \n",
       "4         0.813210           -0.333562        -0.371552  0.093023  ...   \n",
       "...            ...                 ...              ...       ...  ...   \n",
       "16995     8.834429           -0.752689        -1.772980  3.558140  ...   \n",
       "16996     2.779438           -0.642975        -1.437719  0.162791  ...   \n",
       "16997     2.669157           -0.746253        -1.480803  0.023256  ...   \n",
       "16998     3.387029           -0.738450        -1.571499 -0.069767  ...   \n",
       "16999     6.098326           -0.765526        -1.757809  3.604651  ...   \n",
       "\n",
       "       topology_1  topology_2  topology_3  topology_4  topology_5  topology_6  \\\n",
       "0               0           0           0           0           1           0   \n",
       "1               0           0           0           0           0           0   \n",
       "2               0           0           0           0           1           0   \n",
       "3               0           0           0           0           1           0   \n",
       "4               0           0           0           0           1           0   \n",
       "...           ...         ...         ...         ...         ...         ...   \n",
       "16995           0           0           0           1           0           0   \n",
       "16996           0           0           0           1           0           0   \n",
       "16997           0           0           0           0           1           0   \n",
       "16998           0           0           0           0           1           0   \n",
       "16999           0           0           0           0           1           0   \n",
       "\n",
       "       topology_7  topology_8  topology_9  topology_10  \n",
       "0               0           0           0            0  \n",
       "1               0           0           0            0  \n",
       "2               0           0           0            0  \n",
       "3               0           0           0            0  \n",
       "4               0           0           0            0  \n",
       "...           ...         ...         ...          ...  \n",
       "16995           0           0           0            0  \n",
       "16996           0           0           0            0  \n",
       "16997           0           0           0            0  \n",
       "16998           0           0           0            0  \n",
       "16999           0           0           0            0  \n",
       "\n",
       "[17000 rows x 163 columns]"
      ]
     },
     "execution_count": 2624,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2625,
   "id": "0a2dee98",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:08:46.330467Z",
     "start_time": "2021-10-16T16:08:46.305466Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Missing values in Heat adsorption: 0, Inf value in Heat Adsorption: 0\n",
      "0 values in void_volume: 0\n",
      "0 values in void_fraction: 0, -1 value: 0\n",
      "0 values in surface_area: 0, -1 value: 0\n"
     ]
    }
   ],
   "source": [
    "check_all_problems(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2626,
   "id": "d85478bf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:08:47.287466Z",
     "start_time": "2021-10-16T16:08:47.275468Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True,  True,  True,  True,  True,  True,  True,  True,  True,\n",
       "        True])"
      ]
     },
     "execution_count": 2626,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_X.columns == test_X.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2627,
   "id": "3c437ff3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:08:53.922264Z",
     "start_time": "2021-10-16T16:08:53.396271Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>CO2_working_capacity [mL/g]</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>68614</td>\n",
       "      <td>180.180567</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>68615</td>\n",
       "      <td>63.891765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>68616</td>\n",
       "      <td>66.823295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>68617</td>\n",
       "      <td>62.815305</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>68618</td>\n",
       "      <td>62.736945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16995</th>\n",
       "      <td>85609</td>\n",
       "      <td>-8.060973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16996</th>\n",
       "      <td>85610</td>\n",
       "      <td>1.188486</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16997</th>\n",
       "      <td>85611</td>\n",
       "      <td>-0.403545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16998</th>\n",
       "      <td>85612</td>\n",
       "      <td>-1.161775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16999</th>\n",
       "      <td>85613</td>\n",
       "      <td>-4.178828</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>17000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          id  CO2_working_capacity [mL/g]\n",
       "0      68614                   180.180567\n",
       "1      68615                    63.891765\n",
       "2      68616                    66.823295\n",
       "3      68617                    62.815305\n",
       "4      68618                    62.736945\n",
       "...      ...                          ...\n",
       "16995  85609                    -8.060973\n",
       "16996  85610                     1.188486\n",
       "16997  85611                    -0.403545\n",
       "16998  85612                    -1.161775\n",
       "16999  85613                    -4.178828\n",
       "\n",
       "[17000 rows x 2 columns]"
      ]
     },
     "execution_count": 2627,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict = gb.predict(test_X)\n",
    "\n",
    "predicted_df = pd.DataFrame({\"id\":range(68614,68614+len(test_X)), \"CO2_working_capacity [mL/g]\":predict})\n",
    "predicted_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "id": "45e5ca7a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T07:44:22.802551Z",
     "start_time": "2021-10-16T07:44:18.867549Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>CO2_working_capacity [mL/g]</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>68614</td>\n",
       "      <td>180.315536</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>68615</td>\n",
       "      <td>66.231522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>68616</td>\n",
       "      <td>73.615433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>68617</td>\n",
       "      <td>62.306282</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>68618</td>\n",
       "      <td>62.686466</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16995</th>\n",
       "      <td>85609</td>\n",
       "      <td>-5.499111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16996</th>\n",
       "      <td>85610</td>\n",
       "      <td>-0.111550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16997</th>\n",
       "      <td>85611</td>\n",
       "      <td>-0.917235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16998</th>\n",
       "      <td>85612</td>\n",
       "      <td>-0.976805</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16999</th>\n",
       "      <td>85613</td>\n",
       "      <td>-3.520573</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>17000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          id  CO2_working_capacity [mL/g]\n",
       "0      68614                   180.315536\n",
       "1      68615                    66.231522\n",
       "2      68616                    73.615433\n",
       "3      68617                    62.306282\n",
       "4      68618                    62.686466\n",
       "...      ...                          ...\n",
       "16995  85609                    -5.499111\n",
       "16996  85610                    -0.111550\n",
       "16997  85611                    -0.917235\n",
       "16998  85612                    -0.976805\n",
       "16999  85613                    -3.520573\n",
       "\n",
       "[17000 rows x 2 columns]"
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# tabnet\n",
    "predict = tab.predict(np.array(test_X))\n",
    "\n",
    "predicted_df = pd.DataFrame({\"id\":range(68614,68614+len(test_X)), \"CO2_working_capacity [mL/g]\":predict[:,0]})\n",
    "predicted_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "647c5df1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.932747Z",
     "start_time": "2021-10-13T15:48:52.932747Z"
    }
   },
   "outputs": [],
   "source": [
    "# conv1d\n",
    "predict = model.predict(np.expand_dims(test_X, axis=-1).reshape((test_X.shape[0], 1,test_X.shape[1]))).flatten()\n",
    "\n",
    "predicted_df = pd.DataFrame({\"id\":range(68614,68614+len(test_X)), \"CO2_working_capacity [mL/g]\":predict})\n",
    "predicted_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f31a55a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.935748Z",
     "start_time": "2021-10-13T15:48:52.935748Z"
    }
   },
   "outputs": [],
   "source": [
    "# tf\n",
    "predict = model.predict(test_X)\n",
    "\n",
    "predicted_list = []\n",
    "for i in range(len(predict)):\n",
    "    predicted_list.append([f\"pretest_{i+1}\",predict[i][0]])\n",
    "predicted_df = pd.DataFrame(data=predicted_list,columns=[\"Id\",\"CO2\"])\n",
    "predicted_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2628,
   "id": "76054cdb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:09:02.820796Z",
     "start_time": "2021-10-16T16:09:02.746795Z"
    }
   },
   "outputs": [],
   "source": [
    "predicted_df.to_csv(\"output/phase2/submission.csv\",index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a0a2dfe",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.938746Z",
     "start_time": "2021-10-13T15:48:52.938746Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "gb3_importance = pd.Series(gb.feature_importances_, index=clean_X.columns)\n",
    "gb3_importance.sort_values(ascending=False).to_frame().rename(columns={0:'Importance'}).head(25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 889,
   "id": "e3c701b2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T10:41:29.721594Z",
     "start_time": "2021-10-16T10:41:29.700578Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CO2/N2_selectivity</th>\n",
       "      <td>2028</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mol_avg_mass</th>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>heat_adsorption</th>\n",
       "      <td>1981</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>surface_area</th>\n",
       "      <td>1853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>comb_cnn_3</th>\n",
       "      <td>1848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>charges</th>\n",
       "      <td>1726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>volume/g</th>\n",
       "      <td>1487</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>void_fraction</th>\n",
       "      <td>1417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>di_cnn_1</th>\n",
       "      <td>1345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight</th>\n",
       "      <td>1281</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>di_cnn_14</th>\n",
       "      <td>1211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>di_cnn_10</th>\n",
       "      <td>1126</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>n_atoms</th>\n",
       "      <td>1090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>void_volume</th>\n",
       "      <td>1087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>volume</th>\n",
       "      <td>1006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>di_cnn_12</th>\n",
       "      <td>787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>di_cnn_5</th>\n",
       "      <td>743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>di_cnn_2</th>\n",
       "      <td>684</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>di_cnn_6</th>\n",
       "      <td>534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>comb_cnn_2</th>\n",
       "      <td>398</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>comb_cnn_1</th>\n",
       "      <td>392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>density</th>\n",
       "      <td>212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>metal_linker_1</th>\n",
       "      <td>145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>di_cnn_15</th>\n",
       "      <td>103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>metal_linker_2</th>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Importance\n",
       "CO2/N2_selectivity        2028\n",
       "mol_avg_mass              2016\n",
       "heat_adsorption           1981\n",
       "surface_area              1853\n",
       "comb_cnn_3                1848\n",
       "charges                   1726\n",
       "volume/g                  1487\n",
       "void_fraction             1417\n",
       "di_cnn_1                  1345\n",
       "weight                    1281\n",
       "di_cnn_14                 1211\n",
       "di_cnn_10                 1126\n",
       "n_atoms                   1090\n",
       "void_volume               1087\n",
       "volume                    1006\n",
       "di_cnn_12                  787\n",
       "di_cnn_5                   743\n",
       "di_cnn_2                   684\n",
       "di_cnn_6                   534\n",
       "comb_cnn_2                 398\n",
       "comb_cnn_1                 392\n",
       "density                    212\n",
       "metal_linker_1             145\n",
       "di_cnn_15                  103\n",
       "metal_linker_2              83"
      ]
     },
     "execution_count": 889,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lgb_importance = pd.Series(lgb.feature_importances_, index=clean_X.columns)\n",
    "lgb_importance.sort_values(ascending=False).to_frame().rename(columns={0:'Importance'}).head(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd386196",
   "metadata": {},
   "source": [
    "## Ensemble\n",
    "Ensemble the same model (different training doesn't work)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2631,
   "id": "6c5546b9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:10:24.951479Z",
     "start_time": "2021-10-16T16:10:24.913479Z"
    }
   },
   "outputs": [],
   "source": [
    "mlp_best = pd.read_csv('output/phase2/best_predicts/mlp_best.csv')\n",
    "lgb_best = pd.read_csv('output/phase2/best_predicts/lgb_best.csv')\n",
    "gb_best = pd.read_csv('output/phase2/best_predicts/gb_best.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2667,
   "id": "4b8334ee",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:41:30.656582Z",
     "start_time": "2021-10-16T16:41:30.619580Z"
    }
   },
   "outputs": [],
   "source": [
    "tab_best = pd.read_csv('output/phase2/best_predicts/tab_best.csv')\n",
    "di_cnn_best = pd.read_csv('output/phase2/best_predicts/di_cnn_best.csv')\n",
    "combined_cnn_best = pd.read_csv('output/phase2/best_predicts/combined_cnn_best.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2632,
   "id": "09b6bb33",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:10:25.867181Z",
     "start_time": "2021-10-16T16:10:25.861182Z"
    }
   },
   "outputs": [],
   "source": [
    "ensemble = mlp_best[['id']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2633,
   "id": "9795b906",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:10:26.674180Z",
     "start_time": "2021-10-16T16:10:26.654180Z"
    }
   },
   "outputs": [],
   "source": [
    "ensemble['CO2_working_capacity [mL/g]'] = (0.34 * lgb_best['CO2_working_capacity [mL/g]']) + \\\n",
    "                                          (0.33 * mlp_best['CO2_working_capacity [mL/g]']) + \\\n",
    "                                          (0.33 * gb_best['CO2_working_capacity [mL/g]'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2634,
   "id": "b4357edf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:10:32.678888Z",
     "start_time": "2021-10-16T16:10:32.659891Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>CO2_working_capacity [mL/g]</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>68614</td>\n",
       "      <td>173.741324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>68615</td>\n",
       "      <td>64.919034</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>68616</td>\n",
       "      <td>70.031861</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>68617</td>\n",
       "      <td>63.992929</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>68618</td>\n",
       "      <td>63.005615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16995</th>\n",
       "      <td>85609</td>\n",
       "      <td>-7.920903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16996</th>\n",
       "      <td>85610</td>\n",
       "      <td>0.856687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16997</th>\n",
       "      <td>85611</td>\n",
       "      <td>-0.066948</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16998</th>\n",
       "      <td>85612</td>\n",
       "      <td>-1.142232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16999</th>\n",
       "      <td>85613</td>\n",
       "      <td>-4.423955</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>17000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          id  CO2_working_capacity [mL/g]\n",
       "0      68614                   173.741324\n",
       "1      68615                    64.919034\n",
       "2      68616                    70.031861\n",
       "3      68617                    63.992929\n",
       "4      68618                    63.005615\n",
       "...      ...                          ...\n",
       "16995  85609                    -7.920903\n",
       "16996  85610                     0.856687\n",
       "16997  85611                    -0.066948\n",
       "16998  85612                    -1.142232\n",
       "16999  85613                    -4.423955\n",
       "\n",
       "[17000 rows x 2 columns]"
      ]
     },
     "execution_count": 2634,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ensemble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2635,
   "id": "c49dc142",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:10:38.157555Z",
     "start_time": "2021-10-16T16:10:38.088554Z"
    }
   },
   "outputs": [],
   "source": [
    "ensemble.to_csv('output/phase2/submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b567834b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.941749Z",
     "start_time": "2021-10-13T15:48:52.941749Z"
    }
   },
   "outputs": [],
   "source": [
    "# best_mlp = joblib.load('models/mlp_current_best_impute.joblib')\n",
    "# best_gb = joblib.load('models/gb_best_model.joblib')\n",
    "# best_lgb = joblib.load('models/lgb_current_best.joblib')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a1dcd7e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.943749Z",
     "start_time": "2021-10-13T15:48:52.943749Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# mlp_predict = best_mlp.predict(test_X)\n",
    "# gb_predict = best_gb.predict(test_X)\n",
    "# lgb_predict = best_lgb.predict(test_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80331956",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.945747Z",
     "start_time": "2021-10-13T15:48:52.945747Z"
    }
   },
   "outputs": [],
   "source": [
    "# weighted average\n",
    "# mean_ensemble = (0.7 * mlp_predict) + (0.05 * gb_predict) + (0.25 * lgb_predict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9b6e8e0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.946747Z",
     "start_time": "2021-10-13T15:48:52.946747Z"
    }
   },
   "outputs": [],
   "source": [
    "# mean_ensemble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73199a6a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.948748Z",
     "start_time": "2021-10-13T15:48:52.948748Z"
    }
   },
   "outputs": [],
   "source": [
    "# ensemble_df = pd.DataFrame({\"id\":range(68614,68614+len(test_X)), \"CO2_working_capacity [mL/g]\":mean_ensemble})\n",
    "# ensemble_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2eb354b2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-13T15:48:52.949748Z",
     "start_time": "2021-10-13T15:48:52.949748Z"
    }
   },
   "outputs": [],
   "source": [
    "# ensemble_df.to_csv('output/phase2/submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4719d494",
   "metadata": {},
   "source": [
    "## 2nd iter ensemble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2636,
   "id": "46cef743",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:14:35.549135Z",
     "start_time": "2021-10-16T16:14:35.536109Z"
    }
   },
   "outputs": [],
   "source": [
    "ensemble_df_2nd = ensemble.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2637,
   "id": "4c444e34",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:14:37.457318Z",
     "start_time": "2021-10-16T16:14:37.450318Z"
    }
   },
   "outputs": [],
   "source": [
    "ensemble_df_2nd['CO2_working_capacity [mL/g]'] = (0.7*ensemble['CO2_working_capacity [mL/g]']) + \\\n",
    "                                                 (0.3*lgb_best['CO2_working_capacity [mL/g]'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2638,
   "id": "dcc3ac6e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:14:38.813708Z",
     "start_time": "2021-10-16T16:14:38.749678Z"
    }
   },
   "outputs": [],
   "source": [
    "ensemble_df_2nd.to_csv('output/phase2/submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e499d876",
   "metadata": {},
   "source": [
    "# Best ensemble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2639,
   "id": "0a5e116f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:20:09.433634Z",
     "start_time": "2021-10-16T16:20:09.398633Z"
    }
   },
   "outputs": [],
   "source": [
    "best = pd.read_csv('output/phase2/best_predicts/first_iter_best.csv')\n",
    "second_best = pd.read_csv('output/phase2/best_predicts/second_best.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2642,
   "id": "7829f96b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:21:07.858977Z",
     "start_time": "2021-10-16T16:21:07.842976Z"
    }
   },
   "outputs": [],
   "source": [
    "best_ensemble = best[['id']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2643,
   "id": "bc1c5b48",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:21:08.756679Z",
     "start_time": "2021-10-16T16:21:08.738680Z"
    }
   },
   "outputs": [],
   "source": [
    "best_ensemble['CO2_working_capacity [mL/g]'] = (0.5*best['CO2_working_capacity [mL/g]']) + \\\n",
    "                                               (0.5*second_best['CO2_working_capacity [mL/g]'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2644,
   "id": "f6f09a5a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-16T16:21:09.777417Z",
     "start_time": "2021-10-16T16:21:09.714418Z"
    }
   },
   "outputs": [],
   "source": [
    "best_ensemble.to_csv('output/phase2/submission.csv', index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "chemistry-ml",
   "language": "python",
   "name": "chemistry-ml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}