01_The_Basic_Tools_of_the_Deep_Life_Sciences_번역.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7f277cc4",
   "metadata": {},
   "source": [
    "# Totorial  1: THhe Basic Tools of the Deep Life Sciences\n",
    "Deepchem의 생명과학 입문 자습서에 오신 것을 환영합니다. 이 노트북 시리즈는 생명과학의 딥 러닝에 필요한 새로운 도구와 기술을 알아보는 단계별(step-by-step) 가이드입니다. 우리는 기초부터 시작하겠습니다. 여러분이 기계 학습과 생명 과학을 처음 접한다고 가정하고, 생명 과학에서 의미 있는 일을 하는데 사용할 수 있는 도구와 기술의 레퍼토리를 만들 것입니다.\n",
    "##### scope: 이 튜토리얼에는 Deep Life Sciences을 위한 시스템을 구축하는 데 필요한 머신 러닝과 데이터 처리가 모두 포함됩니다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eba2be84",
   "metadata": {},
   "source": [
    "## Why do the DeepChem Tutorial?\n",
    " 1. Career Advancement : 생명과학에 AI를 적용하는 것은 현재 호황산업이다. AI를 중심으로 한 대형 제약·바이오 기업에는 신규 자금 지원 스타트업과 시책이 줄을 잇고 있다. DeepChem을 배우고 익히면 이 분야의 선두에 서게 되고, 이 분야에서 경력을 쌓을 준비가 될 것입니다.\n",
    " \n",
    " 2. Humanitarian Considerations: 질병은 인간의 고통의 가장 오래된 원인이다. 인류 문명의 시작부터, 인간은 병원체, 암, 그리고 신경학적 조건에 시달려왔다. 지난 몇 세기 동안의 가장 큰 업적 중 하나는 많은 질병에 대한 효과적인 치료법의 개발이었다. 이 튜토리얼의 기술을 마스터함으로써, 여러분은 새로운 의학을 개발하는 것을 돕기 위해 과거의 거인들의 어깨에 설 수 있을 것입니다.\n",
    "\n",
    " 3. Lowering the Cost of Medicine: 신약을 개발하는 기술은 현재 소수의 전문 실무자만이 실천할 수 있는 엘리트 기술이다. 약물 발견을 위한 오픈 소스 도구의 성장을 가능하게 함으로써, 이러한 기술을 민주화하고 약물 발견을 더 많은 경쟁으로 이끄는 데 도움을 줄 수 있습니다. 경쟁이 치열해지면 약값을 낮추는 데 도움이 될 수 있다.\n",
    "\n",
    "### Prerequisites\n",
    "이 튜토리얼 시퀀스는 파이썬 데이터 과학 생태계에 대한 기본적인 친숙함을 가정한다. 우리는 여러분이 Numpy, Pandas, TensorFlow와 같은 libraries에 익숙하다고 가정할 것입니다. 튜토리얼을 통해 기본 사항에 대한 간략한 refreshers를 제공해 드릴 테니 전문가가 아니더라도 걱정하지 마십시오.\n",
    "\n",
    "### Setup\n",
    "첫 번째 단계는 DeepChem을 running시키는 것입니다. anaconda을 사용하여 이 튜토리얼 시리즈를 진행할 것이다. jupyter 노트북에 DeepChem을 설치하려면 다음 명령을 실행해야 합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ab16e2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# anaconda prompt에서 가상환경 설정 (python=3.7 환경에서 지원하기 때문)\n",
    "conda create -n deepchem python=3.7 \n",
    "conda activate deepchem\n",
    "# package 설치\n",
    "pip install joblib\n",
    "pip install numpt\n",
    "pip install pandas\n",
    "pip install scikit-learn\n",
    "pip install scipy\n",
    "pip install tensorflow=2.4\n",
    "pip install deepchem\n",
    "conda install -y -c conda-forge rdkit\n",
    "# jupyter notebook 실행\n",
    "jupyter notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "68865a12",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.5.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import deepchem as dc\n",
    "dc.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "990e546f",
   "metadata": {},
   "source": [
    "# Training a Model with DeepChem: A First Example\n",
    "\n",
    "딥 러닝은 많은 종류의 문제를 해결하는 데 사용될 수 있지만, 기본적인 workflow는 대개 동일합니다.  다음은 일반적인 단계입니다.\n",
    "\n",
    "1. 모델을 학습할 데이터 세트를 선택하거나, 적합한 데이터 세트가 없는 경우 새 데이터 세트를 만듭니다.\n",
    "2. 모델을 만듭니다.\n",
    "3. 모델에 데이터를 training합니다.\n",
    "4. 독립 테스트 세트에서 모델을 평가하여 얼마나 잘 작동하는지 확인합니다.\n",
    "5. 모형을 사용하여 새 데이터에 대해 예측합니다.\n",
    "\n",
    "DeepChem을 사용하면 이러한 각 단계는 한두 줄의 파이썬 코드만큼 작을 수 있습니다.  이 튜토리얼에서는 실제 과학 문제를 해결하기 위한 전체 workflow를 보여주는 기본적인 예를 살펴보겠습니다.\n",
    "\n",
    "우리가 해결할 문제는 화학 공식이 주어진 작은 분자의 용해도를 예측하는 것입니다.  이것은 약물 개발에서 매우 중요한 특성입니다. 만약 제안된 약이 충분히 용해되지 않는다면, 여러분은 아마도 치료 효과를 얻기 위해 환자의 혈류로 충분히 유입될 수 없을 것입니다.  우리가 가장 먼저 필요한 것은 실제 분자에 대해 측정된 용해성의 데이터 세트입니다.  DeepChem의 핵심 요소 중 하나는 다양한 화학 및 분자 데이터 세트 모음인 MolecularNet이다.  이 튜토리얼에서는 Delaney 용해성 데이터 세트를 사용할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "62790385",
   "metadata": {},
   "outputs": [],
   "source": [
    "tasks, datasets, transformers = dc.molnet.load_delaney(featurizer='GraphConv')\n",
    "train_dataset, valid_dataset, test_dataset = datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e38c8886",
   "metadata": {},
   "source": [
    "지금은 이 코드에 대해 너무 많이 말하지 않겠습니다. 나중에 튜토리얼에서 유사한 예시를 많이 볼 수 있을 것이다. 두 가지 세부 사항이 있습니다. \n",
    "1. featurizer 인수가 load_delaney() 함수에 전달됩니다. 분자는 여러 가지 방법으로 표현될 수 있다. 따라서 어떤 표현을 사용할지, 또는 더 기술적인 언어로 데이터를 \"featurizer\"하는 방법을 알려줍니다. \n",
    "2. 실제로 교육 세트, 검증 세트 및 테스트 세트의 세 가지 데이터 세트를 얻을 수 있습니다. 이러한 각 기능은 표준 딥 러닝 워크플로우에서 서로 다른 기능을 제공합니다.\n",
    "\n",
    "이제 데이터를 확보했으니, 다음 단계는 모델을 만드는 것입니다. 우리는 \"graph convolutional network\" 또는 줄여서 \"graphconv\"이라고 불리는 특정한 종류의 모델을 사용할 것이다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0cc39215",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = dc.models.GraphConvModel(n_tasks=1, mode='regression', dropout=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85d3fab8",
   "metadata": {},
   "source": [
    "여기서 다시 나는 그 코드에 대해 많은 것을 말하지 않을 것이다. 이후 튜토리얼에서는 Deepchem이 제공하는 다른 종류의 모델뿐만 아니라 'GraphConvModel'에 대한 더 많은 정보를 제공할 것이다.\n",
    "\n",
    "이제 데이터 세트에 대한 모델을 교육해야 합니다. 데이터 세트를 제공하고 training을 몇 epoch을 돌릴 것인지를 알려줍니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a237df98",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_14:0\", shape=(370,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_13:0\", shape=(370, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Cast_4:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_17:0\", shape=(1068,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_16:0\", shape=(1068, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Cast_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_20:0\", shape=(888,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_19:0\", shape=(888, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Cast_6:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_23:0\", shape=(140,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_22:0\", shape=(140, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Cast_7:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_11:0\", shape=(370,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_10:0\", shape=(370, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_13:0\", shape=(1068,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_12:0\", shape=(1068, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_1:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_15:0\", shape=(888,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_14:0\", shape=(888, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_2:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_17:0\", shape=(140,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_16:0\", shape=(140, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_3:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_19:0\", shape=(0,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_18:0\", shape=(0, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_4:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_21:0\", shape=(0,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_20:0\", shape=(0, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_23:0\", shape=(0,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_22:0\", shape=(0, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_6:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_25:0\", shape=(0,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_24:0\", shape=(0, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_7:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_27:0\", shape=(0,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_26:0\", shape=(0, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_8:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_29:0\", shape=(0,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_28:0\", shape=(0, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_9:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_14:0\", shape=(370,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_13:0\", shape=(370, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Cast_4:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_17:0\", shape=(1068,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_16:0\", shape=(1068, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Cast_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_20:0\", shape=(888,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_19:0\", shape=(888, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Cast_6:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_23:0\", shape=(140,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_22:0\", shape=(140, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Cast_7:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_14:0\", shape=(298,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_13:0\", shape=(298, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Cast_4:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_17:0\", shape=(1002,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_16:0\", shape=(1002, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Cast_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_20:0\", shape=(810,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_19:0\", shape=(810, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Cast_6:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_23:0\", shape=(68,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_22:0\", shape=(68, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Cast_7:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_11:0\", shape=(298,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_10:0\", shape=(298, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_13:0\", shape=(1002,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_12:0\", shape=(1002, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_1:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_15:0\", shape=(810,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_14:0\", shape=(810, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_2:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_17:0\", shape=(68,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_16:0\", shape=(68, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_3:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_14:0\", shape=(298,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_13:0\", shape=(298, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Cast_4:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_17:0\", shape=(1002,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_16:0\", shape=(1002, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Cast_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_20:0\", shape=(810,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_19:0\", shape=(810, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Cast_6:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_23:0\", shape=(68,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_22:0\", shape=(68, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Cast_7:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_14:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_13:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Cast_4:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_17:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_16:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Cast_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_20:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_19:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Cast_6:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_23:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Reshape_22:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool_1/Cast_7:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_11:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_10:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_13:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_12:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_1:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_15:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_14:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_2:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_17:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Reshape_16:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_conv_1/Cast_3:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_14:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_13:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Cast_4:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_17:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_16:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Cast_5:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_20:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_19:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Cast_6:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n",
      "C:\\Users\\HYEJI\\anaconda3\\envs\\deepchem\\lib\\site-packages\\tensorflow\\python\\framework\\indexed_slices.py:437: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_23:0\", shape=(None,), dtype=int32), values=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Reshape_22:0\", shape=(None, 64), dtype=float32), dense_shape=Tensor(\"gradient_tape/private__graph_conv_keras_model/graph_pool/Cast_7:0\", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"shape. This may consume a large amount of memory.\" % value)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.10109295845031738"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_dataset, nb_epoch=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bf7455e",
   "metadata": {},
   "source": [
    "모든 것이 잘 되었다면, 우리는 이제 fully trained model을 가져야 합니다! 이를 확인하기 위해 테스트 세트의 모형을 평가해야 합니다. 우리는 평가 지표를 선택하고 모델에서 evaluate() 를 호출하여 이를 수행한다. 이 예에서는 $r^2$라고도 하는 피어슨 상관 관계를 metric으로 사용하겠습니다. 우리는 훈련 세트와 테스트 세트 모두에서 그것을 평가할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "78ca713e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: {'pearson_r2_score': 0.9274230895189561}\n",
      "Test set score: {'pearson_r2_score': 0.6419653405600827}\n"
     ]
    }
   ],
   "source": [
    "metric = dc.metrics.Metric(dc.metrics.pearson_r2_score)\n",
    "print(\"Training set score:\", model.evaluate(train_dataset, [metric], transformers))\n",
    "print(\"Test set score:\", model.evaluate(test_dataset, [metric], transformers))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35254d25",
   "metadata": {},
   "source": [
    "test 세트보다 train 세트에서 점수가 더 높습니다. 모델은 일반적으로 유사하지만 독립적인 데이터보다 훈련 받은 특정 데이터에 대해 더 나은 성능을 발휘한다. 이를 \"overfitting\"이라고 하며, 독립 검정 집합에서 모형을 평가하는 것이 중요한 이유입니다.\n",
    "\n",
    "우리 모델은 test 세트에서 여전히 꽤 훌륭한 성능을 가지고 있다. 비교를 위해, 완전 랜덤 출력을 생성하는 모형은 0의 상관 관계를 갖는 반면, 완벽한 예측을 하는 모형은 1의 상관 관계를 갖는다. 우리의 모델은 꽤 잘 작동하기 때문에, 이제 우리는 그것을 이용해서 다른 분자들에 대한 예측을 할 수 있습니다.\n",
    "\n",
    "이것은 단지 자습서일 뿐이고 우리가 특별히 예측하고 싶은 다른 분자가 없기 때문에, 테스트 세트의 첫 열 개의 분자를 사용해보자. 각 항목에 대해 화학 구조(SMILS 문자열로 표시됨)와 예측 용해도를 출력합니다. 이러한 예측을 상황에 맞게 표현하기 위해 테스트 세트에서 용해도 값도 출력한다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "576aa8d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-1.9413632] [-1.60114461] c1cc2ccc3cccc4ccc(c1)c2c34\n",
      "[0.92616993] [0.20848251] Cc1cc(=O)[nH]c(=S)[nH]1\n",
      "[-1.3494276] [-0.01602738] Oc1ccc(cc1)C2(OC(=O)c3ccccc23)c4ccc(O)cc4 \n",
      "[-1.9886135] [-2.82191713] c1ccc2c(c1)cc3ccc4cccc5ccc2c3c45\n",
      "[-2.0315654] [-0.52891635] C1=Cc2cccc3cccc1c23\n",
      "[1.7977729] [1.10168349] CC1CO1\n",
      "[-0.54089457] [-0.88987406] CCN2c1ccccc1N(C)C(=S)c3cccnc23 \n",
      "[-1.2526282] [-0.52649706] CC12CCC3C(CCc4cc(O)ccc34)C2CCC1=O\n",
      "[-0.74748135] [-0.76358725] Cn2cc(c1ccccc1)c(=O)c(c2)c3cccc(c3)C(F)(F)F\n",
      "[-0.42629468] [-0.64020358] ClC(Cl)(Cl)C(NC=O)N1C=CN(C=C1)C(NC=O)C(Cl)(Cl)Cl \n"
     ]
    }
   ],
   "source": [
    "solubilities = model.predict_on_batch(test_dataset.X[:10])\n",
    "for molecule, solubility, test_solubility in zip(test_dataset.ids, solubilities, test_dataset.y):\n",
    "    print(solubility, test_solubility, molecule)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d7c7184",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "deepchem",
   "language": "python",
   "name": "deepchem"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}