notebooks/EDA/interacting_with_ceph.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Open Data Hub and Object Storage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The intent of this notebook is to provide examples of how data engineers/scientist can use Open Data Hub and object storage, specifically, Ceph object storage, much in the same way they are accustomed to interacting with Amazon Simple Storage Service (S3). This is made possible because Ceph's object storage gateway offers excellent fidelity with the modalities of Amazon S3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with Boto"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boto is an integrated interface to current and future infrastructural services offered by Amazon Web Services. Among the services it provides interfaces for is Amazon S3. For lightweight analysis of data using python tools like numpy or pandas, it is handy to interact with data stored in object storage using pure python. This is where Boto shines. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://s3.upshift.redhat.com\n",
      "mcliffor\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import boto3\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "s3_endpoint_url = os.environ['S3_ENDPOINT_URL']\n",
    "s3_access_key = os.environ['AWS_ACCESS_KEY_ID']\n",
    "s3_secret_key = os.environ['AWS_SECRET_ACCESS_KEY']\n",
    "s3_bucket_name = os.environ['JUPYTERHUB_USER']\n",
    "\n",
    "print(s3_endpoint_url)\n",
    "print(s3_bucket_name)\n",
    "s3 = boto3.client('s3','us-east-1', endpoint_url= s3_endpoint_url,\n",
    "                       aws_access_key_id = s3_access_key,\n",
    "                       aws_secret_access_key = s3_secret_key)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interacting with S3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a bucket, uploading an object (put), and listing the bucket.\n",
    "\n",
    "In the cell below we will use our boto3 connection, `s3`, to do the following: Create an S3 bucket, upload an object, and then display all of the contents of that bucket.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "forestmnist.1.tgz\n",
      "kube-metrics/operationinfo.csv/_SUCCESS\n",
      "kube-metrics/operationinfo.csv/part-00000-1de3723d-a2d5-4f64-9726-d5e0f640fca6-c000.csv\n",
      "kube-metrics/operationinfo.csv/part-00001-1de3723d-a2d5-4f64-9726-d5e0f640fca6-c000.csv\n",
      "kube-metrics/operationinfo.csv/part-00002-1de3723d-a2d5-4f64-9726-d5e0f640fca6-c000.csv\n",
      "kube-metrics/operationinfo.csv/part-00003-1de3723d-a2d5-4f64-9726-d5e0f640fca6-c000.csv\n",
      "kube-metrics/operationinfo.csv/part-00004-1de3723d-a2d5-4f64-9726-d5e0f640fca6-c000.csv\n",
      "kube-metrics/operationinfo.csv/part-00005-1de3723d-a2d5-4f64-9726-d5e0f640fca6-c000.csv\n",
      "new_data\n",
      "new_data.csv\n",
      "object\n",
      "somefolder/new_data.csv\n",
      "trip_report.tsv/_SUCCESS\n",
      "trip_report.tsv/part-00000-3549378a-5714-4808-8ffa-a591faa64ff4-c000.csv\n"
     ]
    }
   ],
   "source": [
    "#s3.create_bucket(Bucket=s3_bucket_name)\n",
    "#s3.put_object(Bucket=s3_bucket_name,Key='object',Body='data')\n",
    "for key in s3.list_objects(Bucket=s3_bucket_name)['Contents']:\n",
    "    print(key['Key'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise #1: Manage Remote Storage\n",
    "\n",
    "Let's do something slightly more more complicated and upload a small file to our new bucket. \n",
    "\n",
    "Below we have used pandas to generate a small csv file for you. Run the below cell, and then upload it to your S3 bucket. Then Display the contents of your bucket like we did above. \n",
    "\n",
    "This resource may be helpful: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-uploading-files.html\n",
    "\n",
    "#### Objective\n",
    "\n",
    "1) Upload a csv file to your s3 bucket using `s3.upload_file()`\n",
    "\n",
    "2) List the objects currently in your bucket using `s3.list_objects()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Create and save a small pandas dataframe and save it locally as a .csv file\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "x = [1,2,3,4]\n",
    "y = [4,5,6,7]\n",
    "\n",
    "df  = pd.DataFrame([x,y])\n",
    "df.to_csv('new_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Upload a csv file to your s3 bucket using s3.upload_file()\n",
    "\n",
    "s3.upload_file(Filename='new_data.csv',Bucket=s3_bucket_name, Key='somefolder/new_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "forestmnist.1.tgz\n",
      "kube-metrics/operationinfo.csv/_SUCCESS\n",
      "kube-metrics/operationinfo.csv/part-00000-1de3723d-a2d5-4f64-9726-d5e0f640fca6-c000.csv\n",
      "kube-metrics/operationinfo.csv/part-00001-1de3723d-a2d5-4f64-9726-d5e0f640fca6-c000.csv\n",
      "kube-metrics/operationinfo.csv/part-00002-1de3723d-a2d5-4f64-9726-d5e0f640fca6-c000.csv\n",
      "kube-metrics/operationinfo.csv/part-00003-1de3723d-a2d5-4f64-9726-d5e0f640fca6-c000.csv\n",
      "kube-metrics/operationinfo.csv/part-00004-1de3723d-a2d5-4f64-9726-d5e0f640fca6-c000.csv\n",
      "kube-metrics/operationinfo.csv/part-00005-1de3723d-a2d5-4f64-9726-d5e0f640fca6-c000.csv\n",
      "new_data\n",
      "new_data.csv\n",
      "object\n",
      "somefolder/new_data.csv\n",
      "trip_report.tsv/_SUCCESS\n",
      "trip_report.tsv/part-00000-3549378a-5714-4808-8ffa-a591faa64ff4-c000.csv\n"
     ]
    }
   ],
   "source": [
    "# 2. List the objects currently in your bucket using s3.list_objects()\n",
    "\n",
    "for key in s3.list_objects(Bucket=s3_bucket_name)['Contents']:\n",
    "    print(key['Key'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets read our data from Ceph back into our notbook! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  0  1  2  3\n",
       "0           0  1  2  3  4\n",
       "1           1  4  5  6  7"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "obj = s3.get_object(Bucket='mcliffor', Key = 'somefolder/new_data.csv')\n",
    "df = pd.read_csv(obj['Body'])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, now you know how to interact with and manage your data store with simple data types.  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}