python-jupyter-apache-kafka-ksql-tensorflow-keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Apache Kafka Integration + Preprocessing / Interactive Analysis with KSQL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook uses the combination of Python, Apache Kafka, KSQL for Machine Learning infrastructures. \n",
    "\n",
    "It includes code examples using ksql-python and other widespread components from Python’s machine learning ecosystem, like Numpy, pandas, TensorFlow and Keras. \n",
    "\n",
    "The use case is fraud detection for credit card payments. We use a test data set from Kaggle as foundation to train an unsupervised autoencoder to detect anomalies and potential fraud in payments. Focus of this example is not just model training, but the whole Machine Learning infrastructure including data ingestion, data preprocessing, model training, model deployment and monitoring. All of this needs to be scalable, reliable and performant.\n",
    "\n",
    "If you want to learn more about the relation between the Apache Kafka open source ecosystem and Machine Learning, please check out these two blog posts:\n",
    "\n",
    "- [How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka](https://www.confluent.io/blog/build-deploy-scalable-machine-learning-production-apache-kafka/)\n",
    "- [https://www.confluent.io/blog/using-apache-kafka-drive-cutting-edge-machine-learning](https://www.confluent.io/blog/using-apache-kafka-drive-cutting-edge-machine-learning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start Backend Services (Zookeeper, Kafka, KSQL)\n",
    "\n",
    "The only server requirement is a local KSQL server running (with Kafka broker ZK node). If you don't have it running, just use Confluent CLI:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This CLI is intended for development only, not for production\n",
      "https://docs.confluent.io/current/cli/index.html\n",
      "\n",
      "Using CONFLUENT_CURRENT: /var/folders/0s/0xdkb9n12yqdb3fs71926z3c0000gp/T/confluent.RK2J2sUD\n",
      "Starting zookeeper\n",
      "zookeeper is [\u001b[0;32mUP\u001b[0m]\n",
      "Starting kafka\n",
      "kafka is [\u001b[0;32mUP\u001b[0m]\n",
      "Starting schema-registry\n",
      "schema-registry is [\u001b[0;32mUP\u001b[0m]\n",
      "Starting ksql-server\n",
      "ksql-server is [\u001b[0;32mUP\u001b[0m]\n"
     ]
    }
   ],
   "source": [
    "# Shows correct startup but does not work 100% yet. Better start from Terminal!\n",
    "#! confluent start ksql-server"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Integration and Preprocessing with Python and KSQL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, create the Kafka Topic 'creditcardfraud_source' if it does not exist already:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.\n",
      "Created topic \"creditcardfraud_source\".\n"
     ]
    }
   ],
   "source": [
    "! kafka-topics --zookeeper localhost:2181 --create --topic creditcardfraud_source --partitions 3 --replication-factor 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then load KSQL library and initiate connection to KSQL server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ksql import KSQLAPI\n",
    "client = KSQLAPI('http://localhost:8088')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consume source data from Kafka Topic \"creditcardfraud_source\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.create_stream(table_name='creditcardfraud_source',\n",
    "                     columns_type=['Id bigint', 'Timestamp varchar', 'User varchar', 'Time int', 'V1 double', 'V2 double', 'V3 double', 'V4 double', 'V5 double', 'V6 double', 'V7 double', 'V8 double', 'V9 double', 'V10 double', 'V11 double', 'V12 double', 'V13 double', 'V14 double', 'V15 double', 'V16 double', 'V17 double', 'V18 double', 'V19 double', 'V20 double', 'V21 double', 'V22 double', 'V23 double', 'V24 double', 'V25 double', 'V26 double', 'V27 double', 'V28 double', 'Amount double', 'Class string'],\n",
    "                     topic='creditcardfraud_source',\n",
    "                     value_format='DELIMITED')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preprocessing: \n",
    "\n",
    "- Filter columns which are not needed \n",
    "- Filter messages where column 'class' is empty\n",
    "- Change data format to Avro for more convenient further processing\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.create_stream_as(table_name='creditcardfraud_preprocessed_avro',\n",
    "                     select_columns=['Time', 'V1', 'V2', 'V3', 'V4', 'V5', 'V6', 'V7', 'V8', 'V9', 'V10', 'V11', 'V12', 'V13', 'V14', 'V15', 'V16', 'V17', 'V18', 'V19', 'V20', 'V21', 'V22', 'V23', 'V24', 'V25', 'V26', 'V27', 'V28', 'Amount', 'Class'],\n",
    "                     src_table='creditcardfraud_source',\n",
    "                     conditions='Class IS NOT NULL',\n",
    "                     kafka_topic='creditcardfraud_preprocessed_avro',\n",
    "                     value_format='AVRO')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the creates KSQL Streams:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.ksql('show streams')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the metadata of the KSQL Stream:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.ksql('describe CREDITCARDFRAUD_PREPROCESSED_AVRO')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interactive query statement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = client.query('SELECT * FROM CREDITCARDFRAUD_PREPROCESSED_AVRO LIMIT 1')\n",
    "\n",
    "for item in query: \n",
    "    print(item)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO How to embed ' ' in Python ???\n",
    "# See https://github.com/bryanyang0528/ksql-python/issues/54\n",
    "# client.ksql('SET \\'auto.offset.reset\\'=\\'earliest\\'');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional (optional) analysis and preprocessing examples\n",
    "\n",
    "Some more examples for possible data wrangling and preprocessing with KSQL:\n",
    "\n",
    "- Anonymization\n",
    "- Augmentation\n",
    "- Merge / Join data frames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = client.query('SELECT Id, MASK_LEFT(User, 2) FROM creditcardfraud_source LIMIT 1')\n",
    "\n",
    "for item in query: \n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = client.query('SELECT Id, IFNULL(Class, \\'-1\\') FROM creditcardfraud_source LIMIT 1')\n",
    "\n",
    "for item in query: \n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stream-Table-Join\n",
    "\n",
    "For the STREAM-TABLE-JOIN, you first need to create a Kafka Topic 'Users' (for the corresponding KSQL TABLE 'Users):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created topic \"users\".\r\n"
     ]
    }
   ],
   "source": [
    "! kafka-topics --zookeeper localhost:2181 --create --topic users --partitions 3 --replication-factor 1 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then create the KSQL Table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.create_table(table_name='users',\n",
    "                     columns_type=['userid varchar', 'gender varchar', 'regionid varchar'],\n",
    "                     topic='users',\n",
    "                     key='userid',\n",
    "                     value_format='AVRO')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'@type': 'currentStatus',\n",
       "  'statementText': \"CREATE STREAM creditcardfraud_per_user WITH (VALUE_FORMAT='AVRO', KAFKA_TOPIC='creditcardfraud_per_user') AS SELECT Time, Amount, Class FROM creditcardfraud_source c INNER JOIN USERS u on c.user = u.userid WHERE u.USERID = 1;\",\n",
       "  'commandId': 'stream/CREDITCARDFRAUD_PER_USER/create',\n",
       "  'commandStatus': {'status': 'SUCCESS',\n",
       "   'message': 'Stream created and running'}}]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.ksql(\"CREATE STREAM creditcardfraud_per_user WITH (VALUE_FORMAT='AVRO', KAFKA_TOPIC='creditcardfraud_per_user') AS SELECT Time, Amount, Class FROM creditcardfraud_source c INNER JOIN USERS u on c.user = u.userid WHERE u.USERID = 1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TODO => The following section will be fixed soon\n",
    "# Mapping from KSQL to Numpy / Pandas for Machine Learning Tasks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "a = \"Kai\"\n",
    "#print(\"BEFORE \" + a)\n",
    "query = client.query('select * from creditcardfraud_source LIMIT 1')\n",
    "#print(list(query))\n",
    "\n",
    "for item in query: \n",
    "    print(item)\n",
    "    a = item\n",
    "    print(type(a))\n",
    "#    print(\"AFTER \" + a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This above command returns a generator. It can be printed e.g. by reading its values via next(query) or a for loop.\n",
    "\n",
    "TODO Get KSQL list into correct Python structure (list? dict?) for Numpy and TensorFlow\n",
    "\n",
    "https://towardsdatascience.com/23-great-pandas-codes-for-data-scientists-cca5ed9d8a38\n",
    "https://pandas.pydata.org/pandas-docs/stable/10min.html\n",
    "https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.iterrows.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataframe = pd.DataFrame(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(dataframe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing with Pandas + Model Training with TensorFlow / Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This part only includes the steps required for model training of the Autoencoder with Keras and TensorFlow. \n",
    "\n",
    "If you want to get a better understanding of the model, take a look at the other notebook [Python Tensorflow Keras Fraud Detection Autoencoder.ipynb](http://localhost:8888/notebooks/Python%20Tensorflow%20Keras%20Fraud%20Detection%20Autoencoder.ipynb) which includes many more details, plots and explanations.\n",
    "\n",
    "[Kudos to David Ellison](https://www.datascience.com/blog/fraud-detection-with-tensorflow).\n",
    "\n",
    "[The credit card fraud data set is available at Kaggle](https://www.kaggle.com/mlg-ulb/creditcardfraud/data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO Copy%Paste from the other Notebook (Python Tensorflow Keras Fraud Detection Autoencoder.ipynb)\n",
    "# Will be done after Magnus helped to fix the mapping from KSQL Generator to Pandas dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import packages\n",
    "# matplotlib inline\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy import stats\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import pickle\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix, precision_recall_curve\n",
    "from sklearn.metrics import recall_score, classification_report, auc, roc_curve\n",
    "from sklearn.metrics import precision_recall_fscore_support, f1_score\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from pylab import rcParams\n",
    "from keras.models import Model, load_model\n",
    "from keras.layers import Input, Dense\n",
    "from keras.callbacks import ModelCheckpoint, TensorBoard\n",
    "from keras import regularizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO => Replace data frame 'df' with data from KSQL query from above to show integration between Kafka and Pandas...\n",
    "\n",
    "# \"data/creditcard_small.csv\" is a very small data set (just for quick demo purpose to get a model binary)\n",
    "# => replace with \"data/creditcard.csv\" to use a real data set to train a model with good accuracy\n",
    "df = pd.read_csv(\"data/creditcard_small.csv\") \n",
    "df.head(n=5) #just to check you imported the dataset properly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#set random seed and percentage of test data\n",
    "RANDOM_SEED = 314 #used to help randomly select the data points\n",
    "TEST_PCT = 0.2 # 20% of the data\n",
    "\n",
    "#set up graphic style in this case I am using the color scheme from xkcd.com\n",
    "rcParams['figure.figsize'] = 14, 8.7 # Golden Mean\n",
    "LABELS = [\"Normal\",\"Fraud\"]\n",
    "#col_list = [\"cerulean\",\"scarlet\"]# https://xkcd.com/color/rgb/\n",
    "#sns.set(style='white', font_scale=1.75, palette=sns.xkcd_palette(col_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "normal_df = df[df.Class == 0] #save normal_df observations into a separate df\n",
    "fraud_df = df[df.Class == 1] #do the same for frauds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#data = df.drop(['Time'], axis=1) #if you think the var is unimportant\n",
    "df_norm = df\n",
    "df_norm['Time'] = StandardScaler().fit_transform(df_norm['Time'].values.reshape(-1, 1))\n",
    "df_norm['Amount'] = StandardScaler().fit_transform(df_norm['Amount'].values.reshape(-1, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x, test_x = train_test_split(df_norm, test_size=TEST_PCT, random_state=RANDOM_SEED)\n",
    "train_x = train_x[train_x.Class == 0] #where normal transactions\n",
    "train_x = train_x.drop(['Class'], axis=1) #drop the class column\n",
    "\n",
    "test_y = test_x['Class'] #save the class column for the test set\n",
    "test_x = test_x.drop(['Class'], axis=1) #drop the class column\n",
    "\n",
    "train_x = train_x.values #transform to ndarray\n",
    "test_x = test_x.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reduce number of epochs and batch_size if your Jupyter crashes (due to memory issues)\n",
    "# nb_epoch = 100\n",
    "# batch_size = 128\n",
    "nb_epoch = 5\n",
    "batch_size = 32\n",
    "\n",
    "input_dim = train_x.shape[1] #num of columns, 30\n",
    "encoding_dim = 14\n",
    "hidden_dim = int(encoding_dim / 2) #i.e. 7\n",
    "learning_rate = 1e-7\n",
    "\n",
    "input_layer = Input(shape=(input_dim, ))\n",
    "encoder = Dense(encoding_dim, activation=\"tanh\", activity_regularizer=regularizers.l1(learning_rate))(input_layer)\n",
    "encoder = Dense(hidden_dim, activation=\"relu\")(encoder)\n",
    "decoder = Dense(hidden_dim, activation='tanh')(encoder)\n",
    "decoder = Dense(input_dim, activation='relu')(decoder)\n",
    "autoencoder = Model(inputs=input_layer, outputs=decoder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "autoencoder.compile(metrics=['accuracy'],\n",
    "                    loss='mean_squared_error',\n",
    "                    optimizer='adam')\n",
    "\n",
    "cp = ModelCheckpoint(filepath=\"models/autoencoder_fraud.h5\",\n",
    "                               save_best_only=True,\n",
    "                               verbose=0)\n",
    "\n",
    "tb = TensorBoard(log_dir='./logs',\n",
    "                histogram_freq=0,\n",
    "                write_graph=True,\n",
    "                write_images=True)\n",
    "\n",
    "history = autoencoder.fit(train_x, train_x,\n",
    "                    epochs=nb_epoch,\n",
    "                    batch_size=batch_size,\n",
    "                    shuffle=True,\n",
    "                    validation_data=(test_x, test_x),\n",
    "                    verbose=1,\n",
    "                    callbacks=[cp, tb]).history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "autoencoder = load_model('models/autoencoder_fraud.h5')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_x_predictions = autoencoder.predict(test_x)\n",
    "mse = np.mean(np.power(test_x - test_x_predictions, 2), axis=1)\n",
    "error_df = pd.DataFrame({'Reconstruction_error': mse,\n",
    "                        'True_class': test_y})\n",
    "error_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The binary 'models/autoencoder_fraud.h5' is the trained model which can then be deployed anywhere to do prediction on new incoming events in real time. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Deployment\n",
    "\n",
    "This demo focuses on the combination of Python and KSQL for data preprocessing and model training. If you want to understand the relation between Apache Kafka, KSQL and Python-related Machine Learning tools for model deployment and monitoring, please check out my other Github projects:\n",
    "\n",
    "Some examples of model deployment in Kafka environments:\n",
    "\n",
    "- [Analytic models (TensorFlow, Keras, H2O and Deeplearning4j) embedded in Kafka Streams microservices](https://github.com/kaiwaehner/kafka-streams-machine-learning-examples)\n",
    "- [Anomaly detection of IoT sensor data with a model embedded into a KSQL UDF](https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot)\n",
    "- [RPC communication between Kafka Streams application and model server (TensorFlow Serving)](https://github.com/kaiwaehner/tensorflow-serving-java-grpc-kafka-streams)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix: Pandas analysis with above Fraud Detection Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"data/creditcard.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Amount']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[0:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[1,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Takes a minute or two (big CSV file)...\n",
    "#df.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}