mnist_keras_maixpy_colab.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "mnist_keras.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ashitani/jupyter_examples/blob/master/mnist_keras_maixpy_colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z8HQTxoHRaGO",
        "colab_type": "text"
      },
      "source": [
        "# Keras/MNIST/Maixpy\n",
        "\n",
        "KerasでMNISTモデルを学習してMaixpyにデプロイするまでを記載します。\n",
        "\n",
        "# 環境構築\n",
        "\n",
        "- Maix_Toolboxはもろもろ便利ツール一式です。\n",
        "- そこから参照されているncc ( TensorflowLiteの出力をK210のKPUで実行できるkmodel形式に変換するツール）をインストール。Linux用のバイナリが提供されていて、Google Colabで直接実行できます。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rm5RqgUfRi-4",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 394
        },
        "outputId": "3b949635-d952-4ab4-e1e3-cedde06771bb"
      },
      "source": [
        "!git clone https://github.com/sipeed/Maix_Toolbox\n",
        "!mkdir Maix_Toolbox/workspace\n",
        "!mkdir Maix_Toolbox/ncc\n",
        "%cd /content/Maix_Toolbox/ncc\n",
        "!wget https://github.com/kendryte/nncase/releases/download/v0.1.0-rc5/ncc-linux-x86_64.tar.xz\n",
        "!tar Jxf ncc-linux-x86_64.tar.xz\n",
        "%cd /content/Maix_Toolbox/workspace"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "fatal: destination path 'Maix_Toolbox' already exists and is not an empty directory.\n",
            "mkdir: cannot create directory ‘Maix_Toolbox/workspace’: File exists\n",
            "mkdir: cannot create directory ‘Maix_Toolbox/ncc’: File exists\n",
            "/content/Maix_Toolbox/ncc\n",
            "--2019-07-21 12:45:21--  https://github.com/kendryte/nncase/releases/download/v0.1.0-rc5/ncc-linux-x86_64.tar.xz\n",
            "Resolving github.com (github.com)... 140.82.118.4\n",
            "Connecting to github.com (github.com)|140.82.118.4|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/128056991/86526300-8233-11e9-91ac-884e08be60de?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190721%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190721T124521Z&X-Amz-Expires=300&X-Amz-Signature=c0bfc3e2d6f156daf236b2452a9dc13e3ed1fdf9b8082dd765402786ef382842&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dncc-linux-x86_64.tar.xz&response-content-type=application%2Foctet-stream [following]\n",
            "--2019-07-21 12:45:21--  https://github-production-release-asset-2e65be.s3.amazonaws.com/128056991/86526300-8233-11e9-91ac-884e08be60de?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190721%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190721T124521Z&X-Amz-Expires=300&X-Amz-Signature=c0bfc3e2d6f156daf236b2452a9dc13e3ed1fdf9b8082dd765402786ef382842&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dncc-linux-x86_64.tar.xz&response-content-type=application%2Foctet-stream\n",
            "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.131.43\n",
            "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.131.43|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 54220128 (52M) [application/octet-stream]\n",
            "Saving to: ‘ncc-linux-x86_64.tar.xz.1’\n",
            "\n",
            "ncc-linux-x86_64.ta 100%[===================>]  51.71M  22.2MB/s    in 2.3s    \n",
            "\n",
            "2019-07-21 12:45:24 (22.2 MB/s) - ‘ncc-linux-x86_64.tar.xz.1’ saved [54220128/54220128]\n",
            "\n",
            "/content/Maix_Toolbox/workspace\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eFuX6zmHRkpH",
        "colab_type": "text"
      },
      "source": [
        "# 学習\n",
        "\n",
        "[kerasのmnist_cnn](https://github.com/keras-team/keras/blob/master/examples/mnist_cnn.py)をいくつか修正しました。\n",
        "\n",
        "- Conv2Dはpadding=\"same\"でないとnncで変換できないのでオプションを追記しました。\n",
        "- もとのモデルだとDenseのパラメータサイズが大きすぎてメモリオーバーになるので、前段にMaxPoolをもう一弾かませたり、Denseの次元を128->32に減らしたりしました。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vJ7k4fwQSY4O",
        "colab_type": "code",
        "outputId": "5b66e4e7-9fea-4661-d6ce-f017b2a6c016",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 921
        }
      },
      "source": [
        "\n",
        "from __future__ import print_function\n",
        "import keras\n",
        "from keras.datasets import mnist\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Dropout, Flatten\n",
        "from keras.layers import Conv2D, MaxPooling2D\n",
        "from keras import backend as K\n",
        "\n",
        "batch_size = 128\n",
        "num_classes = 10\n",
        "epochs = 12\n",
        "\n",
        "# input image dimensions\n",
        "img_rows, img_cols = 28, 28\n",
        "\n",
        "# the data, split between train and test sets\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "\n",
        "if K.image_data_format() == 'channels_first':\n",
        "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
        "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
        "    input_shape = (1, img_rows, img_cols)\n",
        "else:\n",
        "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
        "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
        "    input_shape = (img_rows, img_cols, 1)\n",
        "\n",
        "x_train = x_train.astype('float32')\n",
        "x_test = x_test.astype('float32')\n",
        "x_train /= 255\n",
        "x_test /= 255\n",
        "print('x_train shape:', x_train.shape)\n",
        "print(x_train.shape[0], 'train samples')\n",
        "print(x_test.shape[0], 'test samples')\n",
        "\n",
        "# convert class vectors to binary class matrices\n",
        "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
        "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Conv2D(32, kernel_size=(3, 3),\n",
        "                 activation='relu',\n",
        "                 padding='same', # nncase supports only padding==same\n",
        "                 input_shape=input_shape))\n",
        "model.add(MaxPooling2D(pool_size=(2, 2))) # added because model size is too large\n",
        "model.add(Conv2D(64, (3, 3), \n",
        "                 padding='same', # nncase supports only padding==same\n",
        "                 activation='relu'))\n",
        "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "model.add(Dropout(0.25))\n",
        "model.add(Flatten())\n",
        "model.add(Dense(32, activation='relu')) # modified because model size is too large\n",
        "#model.add(Dense(128, activation='relu'))\n",
        "model.add(Dropout(0.5))\n",
        "model.add(Dense(num_classes, activation='softmax'))\n",
        "\n",
        "model.compile(loss=keras.losses.categorical_crossentropy,\n",
        "              optimizer=keras.optimizers.Adadelta(),\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.fit(x_train, y_train,\n",
        "          batch_size=batch_size,\n",
        "          epochs=epochs,\n",
        "          verbose=1,\n",
        "          validation_data=(x_test, y_test))\n",
        "score = model.evaluate(x_test, y_test, verbose=0)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])\n",
        "\n",
        "model.save(\"mnist.h5\")"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n",
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0721 12:45:34.219817 140617481860992 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
            "\n",
            "W0721 12:45:34.237364 140617481860992 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
            "\n",
            "W0721 12:45:34.240609 140617481860992 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
            "\n",
            "W0721 12:45:34.261916 140617481860992 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3976: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.\n",
            "\n",
            "W0721 12:45:34.314332 140617481860992 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
            "\n",
            "W0721 12:45:34.329217 140617481860992 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "x_train shape: (60000, 28, 28, 1)\n",
            "60000 train samples\n",
            "10000 test samples\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "W0721 12:45:34.432064 140617481860992 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
            "\n",
            "W0721 12:45:34.442806 140617481860992 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead.\n",
            "\n",
            "W0721 12:45:34.564304 140617481860992 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Train on 60000 samples, validate on 10000 samples\n",
            "Epoch 1/12\n",
            "60000/60000 [==============================] - 8s 128us/step - loss: 0.5158 - acc: 0.8321 - val_loss: 0.0880 - val_acc: 0.9717\n",
            "Epoch 2/12\n",
            "60000/60000 [==============================] - 6s 96us/step - loss: 0.2274 - acc: 0.9305 - val_loss: 0.0589 - val_acc: 0.9801\n",
            "Epoch 3/12\n",
            "60000/60000 [==============================] - 6s 95us/step - loss: 0.1842 - acc: 0.9437 - val_loss: 0.0462 - val_acc: 0.9846\n",
            "Epoch 4/12\n",
            "60000/60000 [==============================] - 6s 94us/step - loss: 0.1667 - acc: 0.9498 - val_loss: 0.0392 - val_acc: 0.9874\n",
            "Epoch 5/12\n",
            "60000/60000 [==============================] - 6s 92us/step - loss: 0.1483 - acc: 0.9542 - val_loss: 0.0398 - val_acc: 0.9881\n",
            "Epoch 6/12\n",
            "60000/60000 [==============================] - 6s 92us/step - loss: 0.1403 - acc: 0.9571 - val_loss: 0.0372 - val_acc: 0.9886\n",
            "Epoch 7/12\n",
            "60000/60000 [==============================] - 6s 93us/step - loss: 0.1325 - acc: 0.9592 - val_loss: 0.0358 - val_acc: 0.9881\n",
            "Epoch 8/12\n",
            "60000/60000 [==============================] - 6s 94us/step - loss: 0.1250 - acc: 0.9605 - val_loss: 0.0337 - val_acc: 0.9891\n",
            "Epoch 9/12\n",
            "60000/60000 [==============================] - 6s 92us/step - loss: 0.1216 - acc: 0.9626 - val_loss: 0.0331 - val_acc: 0.9890\n",
            "Epoch 10/12\n",
            "60000/60000 [==============================] - 5s 91us/step - loss: 0.1158 - acc: 0.9636 - val_loss: 0.0327 - val_acc: 0.9899\n",
            "Epoch 11/12\n",
            "60000/60000 [==============================] - 6s 92us/step - loss: 0.1142 - acc: 0.9646 - val_loss: 0.0306 - val_acc: 0.9906\n",
            "Epoch 12/12\n",
            "60000/60000 [==============================] - 6s 92us/step - loss: 0.1053 - acc: 0.9675 - val_loss: 0.0290 - val_acc: 0.9916\n",
            "Test loss: 0.02897006175173501\n",
            "Test accuracy: 0.9916\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m4A0tqqCSjTd",
        "colab_type": "text"
      },
      "source": [
        "# モデル変換\n",
        "\n",
        "- 下準備\n",
        "    - kmodelへの変換時にテスト画像がimagesフォルダにないとだめらしい(これを使って量子化の際のダイナミックレンジ推定を行うらしい)ので作成します。最終的にはuint8が突っ込まれるので、uint8にしないとだめです。\n",
        "    - flash-list.jsonを用意([こちら](http://blog.sipeed.com/p/390.html)が詳しいです)\n",
        "- tflite_convertでh5->TensorflowLite に変換します。\n",
        "- tflite2kmodel.shでkmodelに変換します。\n",
        "- [こちら](https://github.com/sipeed/LicheeDan_K210_examples/tree/master/src/mnist)を参考に、kmodelをflash-list.jsonとともにtarして、kpkgファイルを作成します。\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7x7DqCiHTh2i",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "9401757a-0ee5-4ff5-e81e-869bbca30572"
      },
      "source": [
        "%cd /content/Maix_Toolbox/\n",
        "!mkdir images\n",
        "import numpy as np\n",
        "import cv2\n",
        "batch_num=100\n",
        "batch = x_train[0:batch_num]\n",
        "imgs=batch.reshape((batch_num,28,28))*255\n",
        "imgs=imgs.astype(np.uint8)\n",
        "for i,img in enumerate(imgs):\n",
        "  cv2.imwrite(\"images/%03d.jpg\"%i, img)\n",
        "\n",
        "jsontext=\"\"\"\n",
        "{\n",
        "  \"version\": \"0.1.0\",\n",
        "  \"files\": [\n",
        "    {\n",
        "      \"address\": 0x00300000,\n",
        "      \"bin\": \"mnist.kmodel\",\n",
        "      \"sha256Prefix\": false\n",
        "    }\n",
        "  ]\n",
        "}\n",
        "\"\"\"\n",
        "with open(\"workspace/flash-list.json\",\"w\") as f:\n",
        "  f.write(jsontext)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/Maix_Toolbox\n",
            "mkdir: cannot create directory ‘images’: File exists\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-npPJC9oe5wR",
        "colab_type": "code",
        "outputId": "d5e5717b-0b0e-40d1-e1b0-f19a0135c9cc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "%cd /content/Maix_Toolbox/\n",
        "!tflite_convert  --output_file=/content/Maix_Toolbox/workspace/mnist.tflite --keras_model_file=/content/Maix_Toolbox/workspace/mnist.h5\n",
        "!./tflite2kmodel.sh workspace/mnist.tflite\n",
        "!./ncc/ncc -i tflite -o k210model --dataset /content/Maix_Toolbox/images /content/Maix_Toolbox/workspace/mnist.tflite /content/Maix_Toolbox/workspace/mnist.kmodel\n",
        "%cd /content/Maix_Toolbox/workspace\n",
        "!zip -r mnist_at_0x300000.kfpkg mnist.kmodel flash-list.json"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/Maix_Toolbox\n",
            "2019-07-21 12:46:52.369473: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz\n",
            "2019-07-21 12:46:52.369741: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x562274bfc140 executing computations on platform Host. Devices:\n",
            "2019-07-21 12:46:52.369780: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>\n",
            "2019-07-21 12:46:52.371873: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1\n",
            "2019-07-21 12:46:52.434538: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:52.435069: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56227533f180 executing computations on platform CUDA. Devices:\n",
            "2019-07-21 12:46:52.435107: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla K80, Compute Capability 3.7\n",
            "2019-07-21 12:46:52.435383: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:52.435810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: \n",
            "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n",
            "pciBusID: 0000:00:04.0\n",
            "2019-07-21 12:46:52.436099: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0\n",
            "2019-07-21 12:46:52.437437: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0\n",
            "2019-07-21 12:46:52.438999: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0\n",
            "2019-07-21 12:46:52.439355: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0\n",
            "2019-07-21 12:46:52.441064: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0\n",
            "2019-07-21 12:46:52.442535: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0\n",
            "2019-07-21 12:46:52.446691: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7\n",
            "2019-07-21 12:46:52.446814: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:52.447331: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:52.447699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0\n",
            "2019-07-21 12:46:52.447759: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0\n",
            "2019-07-21 12:46:52.448862: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
            "2019-07-21 12:46:52.448894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 \n",
            "2019-07-21 12:46:52.448911: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N \n",
            "2019-07-21 12:46:52.449241: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:52.449854: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:52.450257: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:40] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
            "2019-07-21 12:46:52.450343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10446 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
            "2019-07-21 12:46:53.423493: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:53.423921: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1\n",
            "2019-07-21 12:46:53.424077: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session\n",
            "2019-07-21 12:46:53.424976: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:53.425387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: \n",
            "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n",
            "pciBusID: 0000:00:04.0\n",
            "2019-07-21 12:46:53.425454: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0\n",
            "2019-07-21 12:46:53.425490: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0\n",
            "2019-07-21 12:46:53.425522: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0\n",
            "2019-07-21 12:46:53.425570: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0\n",
            "2019-07-21 12:46:53.425606: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0\n",
            "2019-07-21 12:46:53.425635: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0\n",
            "2019-07-21 12:46:53.425665: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7\n",
            "2019-07-21 12:46:53.425765: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:53.426220: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:53.426644: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0\n",
            "2019-07-21 12:46:53.426692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
            "2019-07-21 12:46:53.426714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 \n",
            "2019-07-21 12:46:53.426733: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N \n",
            "2019-07-21 12:46:53.427040: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:53.427507: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:53.427862: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10446 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
            "2019-07-21 12:46:53.436365: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: graph_to_optimize\n",
            "2019-07-21 12:46:53.436399: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   function_optimizer: function_optimizer did nothing. time = 0.002ms.\n",
            "2019-07-21 12:46:53.436414: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   function_optimizer: function_optimizer did nothing. time = 0ms.\n",
            "W0721 12:46:53.441457 139908598962048 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/lite/python/util.py:238: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.compat.v1.graph_util.convert_variables_to_constants`\n",
            "W0721 12:46:53.441708 139908598962048 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/graph_util_impl.py:270: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.compat.v1.graph_util.extract_sub_graph`\n",
            "I0721 12:46:53.455049 139908598962048 graph_util_impl.py:311] Froze 8 variables.\n",
            "I0721 12:46:53.457128 139908598962048 graph_util_impl.py:364] Converted 8 variables to const ops.\n",
            "2019-07-21 12:46:53.459370: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:53.459758: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1\n",
            "2019-07-21 12:46:53.459848: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session\n",
            "2019-07-21 12:46:53.460432: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:53.460811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: \n",
            "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n",
            "pciBusID: 0000:00:04.0\n",
            "2019-07-21 12:46:53.460861: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0\n",
            "2019-07-21 12:46:53.460895: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0\n",
            "2019-07-21 12:46:53.460930: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0\n",
            "2019-07-21 12:46:53.460962: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0\n",
            "2019-07-21 12:46:53.460998: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0\n",
            "2019-07-21 12:46:53.461030: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0\n",
            "2019-07-21 12:46:53.461061: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7\n",
            "2019-07-21 12:46:53.461162: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:53.461650: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:53.462000: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0\n",
            "2019-07-21 12:46:53.462039: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
            "2019-07-21 12:46:53.462061: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 \n",
            "2019-07-21 12:46:53.462080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N \n",
            "2019-07-21 12:46:53.462400: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:53.462842: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-21 12:46:53.463208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10446 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
            "2019-07-21 12:46:53.465362: E tensorflow/core/grappler/grappler_item_builder.cc:637] Init node conv2d_1/kernel/Assign doesn't exist in graph\n",
            "uasge: ./tflite2kmodel.sh xxx.tflite\n",
            "2019-07-21 12:46:57.697760: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA\n",
            "0: InputLayer -> 1x1x28x28\n",
            "1: K210Conv2d 1x1x28x28 -> 1x32x14x14\n",
            "2: K210Conv2d 1x32x14x14 -> 1x64x7x7\n",
            "3: Dequantize 1x64x7x7 -> 1x64x7x7\n",
            "4: TensorflowFlatten 1x64x7x7 -> 1x3136\n",
            "5: FullyConnected 1x3136 -> 1x32\n",
            "6: Quantize 1x32 -> 1x32\n",
            "7: K210AddPadding 1x32 -> 1x32x4x4\n",
            "8: K210Conv2d 1x32x4x4 -> 1x10x4x4\n",
            "9: K210RemovePadding 1x10x4x4 -> 1x10\n",
            "10: Dequantize 1x10 -> 1x10\n",
            "11: Softmax 1x10 -> 1x10\n",
            "12: OutputLayer 1x10\n",
            "KPU memory usage: 2097152 B\n",
            "Main memory usage: 28224 B\n",
            "2019-07-21 12:47:00.590814: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA\n",
            "0: InputLayer -> 1x1x28x28\n",
            "1: K210Conv2d 1x1x28x28 -> 1x32x14x14\n",
            "2: K210Conv2d 1x32x14x14 -> 1x64x7x7\n",
            "3: Dequantize 1x64x7x7 -> 1x64x7x7\n",
            "4: TensorflowFlatten 1x64x7x7 -> 1x3136\n",
            "5: FullyConnected 1x3136 -> 1x32\n",
            "6: Quantize 1x32 -> 1x32\n",
            "7: K210AddPadding 1x32 -> 1x32x4x4\n",
            "8: K210Conv2d 1x32x4x4 -> 1x10x4x4\n",
            "9: K210RemovePadding 1x10x4x4 -> 1x10\n",
            "10: Dequantize 1x10 -> 1x10\n",
            "11: Softmax 1x10 -> 1x10\n",
            "12: OutputLayer 1x10\n",
            "KPU memory usage: 2097152 B\n",
            "Main memory usage: 28224 B\n",
            "/content/Maix_Toolbox/workspace\n",
            "updating: mnist.kmodel (deflated 7%)\n",
            "updating: flash-list.json (deflated 24%)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-HEykstOTxo2",
        "colab_type": "text"
      },
      "source": [
        "# デバイスへの転送と実行\n",
        "\n",
        "mnist_at_0x300000.kfpkgをダウンロードします。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WksRhutIf7bP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import files\n",
        "files.download('mnist_at_0x300000.kfpkg')\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zaEb8UZ6UPFq",
        "colab_type": "text"
      },
      "source": [
        "kflash.pyで焼く場合は\n",
        "\n",
        "```\n",
        "python3 kflash.py -p /dev/＊＊＊  mnist.kfpkg\n",
        "```\n",
        "\n",
        "等々でOkです。もちろんkflash_guiを使ってもよいでしょう。\n",
        "\n",
        "ファームは、この程度のサイズのモデルならデフォルト（maixpy_v0.3.2_full.bin）でOkでしたが、大きいモデルを使いたければそのぶん小さなファームを使うといいでしょう。\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zRbmIhtfUamw",
        "colab_type": "text"
      },
      "source": [
        "maixpyに接続して、Ctrl-Eを入力します。下記をコピペしてCtrl-Dで起動します。\n",
        "vflip/hmirrorのあたりは環境に合わせて修正するといいと思います。自分の環境（秋月で買ったMaix Bit Suit)ではvflipが必要でした。\n",
        "\n",
        "\n",
        "```\n",
        "import sensor,lcd,image\n",
        "import KPU as kpu\n",
        "lcd.init()\n",
        "sensor.reset()\n",
        "sensor.set_auto_gain(0,24) # auto gain disable and 24dB\n",
        "sensor.set_pixformat(sensor.RGB565)\n",
        "sensor.set_framesize(sensor.QVGA)\n",
        "sensor.set_windowing((224, 224))    #set to 224x224 input\n",
        "sensor.set_vflip(True)                    #flip camera\n",
        "#sensor.set_hmirror(0)               #flip camera\n",
        "task = kpu.load(0x300000)           #load model from flash address 0x300000\n",
        "sensor.run(1)\n",
        "while True:\n",
        "    img = sensor.snapshot()\n",
        "    lcd.display(img,oft=(0,0))      #display large picture\n",
        "    img1=img.to_grayscale(1)        #convert to gray\n",
        "    img2=img1.resize(28,28)         #resize to mnist input 28x28\n",
        "    a=img2.invert()                 #invert picture as mnist need\n",
        "    a=img2.strech_char(1)           #preprocessing pictures, eliminate dark corner\n",
        "    lcd.display(img2,oft=(224,32))  #display small 28x28 picture\n",
        "    a=img2.pix_to_ai();             #generate data for ai\n",
        "    fmap=kpu.forward(task,img2)     #run neural network model \n",
        "    plist=fmap[:]                   #get result (10 digit's probability)\n",
        "    pmax=max(plist)                 #get max probability\n",
        "    print(plist)\n",
        "    max_index=plist.index(pmax)     #get the digit\n",
        "    lcd.draw_string(224,0,\"%d: %.3f\"%(max_index,pmax),lcd.WHITE,lcd.BLACK)  #show result\n",
        "```\n"
      ]
    }
  ]
}