if isinstance(data, bytes):\n",
+ " import pickle\n",
+ " return pickle.loads(data) \n",
+ " \n",
+ "and delete the lines:\n",
"\n",
- "compartment_id = os.environ['NB_SESSION_COMPARTMENT_OCID']\n",
- "project_id = os.environ[\"PROJECT_OCID\"]"
+ "if isinstance(data, bytes):\n",
+ " logging.warning(\n",
+ " \"bytes are passed directly to the model. If the model expects a specific data format, you need to write the conversion logic in `deserialize()` yourself.\"\n",
+ " )\n",
+ " return data "
]
},
{
@@ -777,29 +636,14 @@
"metadata": {},
"outputs": [],
"source": [
- "mc_model = artifact.save(project_id=project_id, compartment_id=compartment_id, display_name=\"xray_cnn_model\",\n",
- " description=f\"Simple CNN model to classify xray images as having pneumonia or not\",\n",
- " training_script_path=f\"./ChestXrays_Train.ipynb\", \n",
- " ignore_pending_changes=True)\n",
- "mc_model"
+ "tensorflow_model.reload()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "# Deploy the Model with Data Science Model Deployment "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Here you have a couple of options: \n",
- "* you can deploy directly from the console **-or-** \n",
- "* you can use the OCI Python SDK to deploy the model programatically without leaving the notebook environment. \n",
- "\n",
- "The code snippet below will show how you can deploy using the Python SDK: "
+ "Next, pass an image as a payload to the model endpoint. The image is an numpy narray that will be serialized as a pickle object "
]
},
{
@@ -808,19 +652,17 @@
"metadata": {},
"outputs": [],
"source": [
- "import oci \n",
- "from oci.data_science import DataScienceClient, DataScienceClientCompositeOperations\n",
+ "ndarray_image = test.next()[0]\n",
"\n",
- "from oci.data_science.models import ModelConfigurationDetails, InstanceConfiguration, \\\n",
- " FixedSizeScalingPolicy, CategoryLogDetails, LogDetails, \\\n",
- " SingleModelDeploymentConfigurationDetails, CreateModelDeploymentDetails"
+ "import pickle \n",
+ "image_as_bytes = pickle.dumps(ndarray_image)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "A couple of ways to authenticate against model deployment. First one is through resource principals. Second one is to use the config + pem files in your notebook session. Pick the one that you want to use. "
+ "Verify the prediction before you save the model to the catalog: "
]
},
{
@@ -829,20 +671,14 @@
"metadata": {},
"outputs": [],
"source": [
- "# using resource principals\n",
- "auth = oci.auth.signers.get_resource_principals_signer()\n",
- "data_science = DataScienceClient({}, signer=auth)\n",
- "\n",
- "# OR the config + pem authn flow: \n",
- "#oci_config = oci.config.from_file('~/.oci/config', \"DEFAULT\")\n",
- "#data_science = DataScienceClient(oci_config)"
+ "tensorflow_model.verify(image_as_bytes)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "The cell below provides the necessary parameters for a model deployment. The first three parameters are standard to all the Data Science resources: "
+ "Saving the model to the catalog: "
]
},
{
@@ -851,68 +687,17 @@
"metadata": {},
"outputs": [],
"source": [
- "# Configuration of the model: \n",
- "# We deploy the sklearn model we saved to the model catalog in this notebook (catalog_entry.id)\n",
- "# You can change the shape, instance_count, or the bandwidth of the load balancer (in Mbps)\n",
- "model_configuration_details_object = ModelConfigurationDetails(model_id=mc_model.id,\n",
- " instance_configuration=InstanceConfiguration(instance_shape_name='VM.Standard2.1'),\n",
- " scaling_policy=FixedSizeScalingPolicy(instance_count=1),\n",
- " bandwidth_mbps=10)\n",
- "\n",
- "# Single Model Deployment Configuration\n",
- "# Includes info about the deployment type and the model configuration details: \n",
- "# At the moment, only deployment+type='SINGLE_MODEL' is supported. \n",
- "single_model_config = SingleModelDeploymentConfigurationDetails(deployment_type='SINGLE_MODEL',\n",
- " model_configuration_details=model_configuration_details_object)\n",
- "\n",
- "\n",
- "# OPTIONAL - Configuration of the access and predict logs. \n",
- "# Make sure you have the proper policy in place to allow model deployment to emit predict/access logs. For example: \n",
- "# allow any-user to use log-content in tenancy where ALL {request.principal.type = 'datasciencemodeldeployment'}\n",
- "logging_config = False\n",
- "if logging_config: \n",
- "\n",
- " access_log_group_id = \"by the Oracle Cloud Infrastructure Data Science Team
\n", + "\n", + "***" + ] + }, + { + "cell_type": "markdown", + "id": "27e535fd", + "metadata": {}, + "source": [ + "# Introduction \n", + "\n", + "Data Science Jobs allow you to run customized tasks outside of a notebook session. You can have Compute on demand and only pay for the Compute that you need. With jobs, you can run applications that perform tasks such as data preparation, model training, hyperparameter tuning, and batch inference. When the task is complete, the compute automatically terminates. You can use the Logging service to capture output messages. In this notebook, we will use the Accelerated Data Science SDK (ADS) to help us define a Data Science Job to train a transfer learning model to detect pneumonia in patients with X-ray images. Transfer learning uses a pre-trained model as a starting point for training another model, and we are going to use the [VGG-16 model](https://www.robots.ox.ac.uk/~vgg/research/very_deep/). \n", + "\n", + "For more information on using ADS for jobs, you can go to our [documentation](https://docs.oracle.com/en-us/iaas/tools/ads-sdk/latest/user_guide/jobs/index.html)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8521abf3", + "metadata": {}, + "outputs": [], + "source": [ + "from ads.jobs import Job\n", + "from ads.jobs import DataScienceJob, ScriptRuntime\n", + "import ads \n", + "ads.set_auth('resource_principal')" + ] + }, + { + "cell_type": "markdown", + "id": "9a923c45", + "metadata": {}, + "source": [ + "## Infrastructure\n", + "\n", + "Data Science Job infrastructure is defined by a `DataScienceJob` instance. \n", + "Important: If you want to use logging for the job, fill in the `log_group_id` and `log_id` in the cell below. You need to have set up the policies for the logging service. For more information about setting up logs for a job, you can go to our [documentation](https://docs.oracle.com/en-us/iaas/data-science/using/log-about.htm#jobs_about__job-logs)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "149c2781", + "metadata": {}, + "outputs": [], + "source": [ + "infrastructure = (\n", + " DataScienceJob()\n", + " .with_shape_name(\"VM.Standard2.24\")\n", + " .with_block_storage_size(50)\n", + " .with_log_group_id(\"\")\n", + " .with_log_id(\"\")\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "6d5287b3", + "metadata": {}, + "source": [ + "## Job Runtime\n", + "\n", + "`ScriptRuntime` allows you to run Python, Bash, and Java scripts from a single source file (.zip or .tar.gz) or code directory. You can configure a Data Science Conda Environment for running your code." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1d6a7546", + "metadata": {}, + "outputs": [], + "source": [ + "runtime = (\n", + " ScriptRuntime()\n", + " .with_source(\"./training_vgg16.py\")\n", + " .with_service_conda(\"tensorflow27_p37_cpu_v1\")\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "e7e2d826", + "metadata": {}, + "source": [ + "## Define Job\n", + "\n", + "With runtime and infrastructure, you can define a job and give it a name" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "61209c2d", + "metadata": {}, + "outputs": [], + "source": [ + "job = Job(name='vgg16-training').with_infrastructure(infrastructure).with_runtime(runtime)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1bad5e87", + "metadata": {}, + "outputs": [], + "source": [ + "job.to_yaml('training-config.yaml')" + ] + }, + { + "cell_type": "markdown", + "id": "1a629ba7", + "metadata": {}, + "source": [ + "## Create and Run Job\n", + "\n", + "You can call the `create()` method of a job instance to create a job. After the job is created, you can call the `run()` method to create and start a job run. The `run()` method returns a `DataScienceJobRun`. You can monitor the job run output by calling the `watch()` method of the `DataScienceJobRun` instance." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "88e50ec2", + "metadata": {}, + "outputs": [], + "source": [ + "job.create()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "88d0a3a1", + "metadata": {}, + "outputs": [], + "source": [ + "job_run = job.run()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "38f62d2f", + "metadata": {}, + "outputs": [], + "source": [ + "job_run.watch()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python [conda env:tensorflow27_p37_cpu_v1]", + "language": "python", + "name": "conda-env-tensorflow27_p37_cpu_v1-py" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/labs/xray-diagnostics/notebooks/training_vgg16.py b/labs/xray-diagnostics/notebooks/training_vgg16.py new file mode 100644 index 00000000..7869ec5a --- /dev/null +++ b/labs/xray-diagnostics/notebooks/training_vgg16.py @@ -0,0 +1,155 @@ +import os +import ocifs +from ocifs import OCIFileSystem +from zipfile import ZipFile +import random +import shutil +from ads.dataset.factory import DatasetFactory + +import tensorflow as tf +from tensorflow import keras +import matplotlib.pyplot as plt + +fs = OCIFileSystem() + +# Creating the local directory +dirpath = f"./data/" +if not os.path.exists(dirpath): + os.makedirs(dirpath) + +# Downloading the data from Object Storage using OCIFS (https://github.com/oracle/ocifs) +if os.path.exists(os.path.join(dirpath, "chest_xrays.zip")): + with ZipFile(os.path.join(dirpath, "chest_xrays.zip"), 'r') as zipf: + zipf.extractall(dirpath) +else: + fs.download('oci://hosted-ds-datasets@bigdatadatasciencelarge/chest-xrays/ChestXRay2017.zip',os.path.join(dirpath, "chest_xrays.zip")) + with ZipFile(os.path.join(dirpath, "chest_xrays.zip"), 'r') as zipf: + zipf.extractall(dirpath) + +train_dir = "./data/chest_xray/train/" +test_dir = "./data/chest_xray/test/" +valid_dir = f"./data/chest_xray/validation/" +if not os.path.exists(valid_dir): + os.makedirs(valid_dir) + +normal_train = "./data/chest_xray/train/NORMAL/" +pneumonia_train = "./data/chest_xray/train/PNEUMONIA/" + +normal_images = os.listdir(normal_train) +pneumonia_images = os.listdir(pneumonia_train) + +valid_dir_normal = os.path.join(valid_dir,"NORMAL") +if not os.path.exists(valid_dir_normal): + os.makedirs(valid_dir_normal) + +valid_dir_pneumonia = os.path.join(valid_dir,"PNEUMONIA") +if not os.path.exists(valid_dir_pneumonia): + os.makedirs(valid_dir_pneumonia) + +# validation sample: +nb_validation_normal = 8 +nb_validation_pneumonia = 8 + +validation_normal_files = random.sample(normal_images, k=nb_validation_normal) +validation_pneumonia_files = random.sample(pneumonia_images, k=nb_validation_pneumonia) + +for x in validation_normal_files: + shutil.move(os.path.join(normal_train,x),os.path.join(valid_dir_normal,x)) + +for x in validation_pneumonia_files: + shutil.move(os.path.join(pneumonia_train,x),os.path.join(valid_dir_pneumonia,x)) + +f_pneumonia_training = len(os.listdir(pneumonia_train)) / (len(os.listdir(pneumonia_train)) + len(os.listdir(normal_train))) +f_normal_training = 1.0 - f_pneumonia_training +print(f'fraction pneumonia in training dataset : {f_pneumonia_training}') +print(f'fraction normal in training dataset : {f_normal_training}') + +from keras.preprocessing.image import ImageDataGenerator + +image_generator = ImageDataGenerator( + rotation_range=20, + width_shift_range=0.1, + shear_range=0.1, + zoom_range=0.1, + samplewise_center=True, + samplewise_std_normalization=True +) + +train = image_generator.flow_from_directory(train_dir, + batch_size=8, + shuffle=True, + class_mode='binary', + target_size=(180, 180)) + +validation = image_generator.flow_from_directory(valid_dir, + batch_size=1, + shuffle=False, + class_mode='binary', + target_size=(180, 180)) + + +test = image_generator.flow_from_directory(test_dir, + batch_size=1, + shuffle=False, + class_mode='binary', + target_size=(180, 180)) + + +from tensorflow.keras.models import Sequential +from tensorflow.keras.layers import GlobalAveragePooling2D +from tensorflow.keras.applications import VGG16, InceptionV3 +from tensorflow.keras.layers import Dense, Conv2D, MaxPool2D, Dropout, Flatten, BatchNormalization +from tensorflow.keras.metrics import Accuracy, Precision, Recall +from tensorflow.keras.optimizers import Adam + +vgg16_base_model = VGG16(input_shape=(180,180,3), + include_top=False, + weights='imagenet') + +vgg16_model = Sequential([ + vgg16_base_model, + GlobalAveragePooling2D(), + Dense(512, activation="relu"), + BatchNormalization(), + Dropout(0.6), + Dense(128, activation="relu"), + BatchNormalization(), + Dropout(0.4), + Dense(64,activation="relu"), + BatchNormalization(), + Dropout(0.3), + Dense(1,activation="sigmoid") + ]) + + +optimizer = Adam(learning_rate=0.001) +METRICS = ['accuracy', + Precision(name='precision'), + Recall(name='recall')] + +vgg16_model.compile(optimizer=optimizer, + loss='binary_crossentropy', + metrics=METRICS) + +class_weight = {0: f_pneumonia_training, 1: f_normal_training} + +r = vgg16_model.fit(train, + epochs=10, + validation_data=validation, + class_weight=class_weight, + steps_per_epoch=100, + validation_steps=25) + +evaluation =vgg16_model.evaluate(test) +print(f"Test Accuracy: {evaluation[1] * 100:.2f}%") + +evaluation = vgg16_model.evaluate(train) +print(f"Train Accuracy: {evaluation[1] * 100:.2f}%") + +vgg16_model.save("./vgg16.tf",save_format='tf') + +print('uploading model to object storage') + +fs.upload("./vgg16.tf", "oci://ds-models@bigdatadatasciencelarge/vgg16.tf", recursive=True) + +print('uploaded the model to object storage') \ No newline at end of file