Section_4_hls4ml.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.datasets import mnist\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "from qkeras.utils import _add_supported_quantized_objects\n",
    "import hls4ml\n",
    "from sklearn.metrics import accuracy_score\n",
    "import numpy as np\n",
    "\n",
    "co = {}\n",
    "_add_supported_quantized_objects(co)\n",
    "\n",
    "def get_train_test_set():\n",
    "    (x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "    \n",
    "    x_train = x_train / 256.0\n",
    "    x_test = x_test / 256.0\n",
    "    \n",
    "    x_train = x_train.reshape(x_train.shape[0], -1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], -1)\n",
    "        \n",
    "    y_train = to_categorical(y_train, 10)\n",
    "    y_test = to_categorical(y_test, 10)\n",
    "    \n",
    "    return (x_train, y_train), (x_test, y_test)\n",
    "\n",
    "def print_dict(d, indent=0):\n",
    "    align=20\n",
    "    for key, value in d.items():\n",
    "        print('  ' * indent + str(key), end='')\n",
    "        if isinstance(value, dict):\n",
    "            print()\n",
    "            print_dict(value, indent+1)\n",
    "        else:\n",
    "            print(':' + ' ' * (20 - len(key) - 2 * indent) + str(value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# hls4ml\n",
    "\n",
    "The goal of hls4ml is to perform fast, efficient, customisable hardware inference of Deep Learning models. The workflow for hls4ml is illustrated below. \n",
    "\n",
    "The user (you) train a model. There is a \"loop\" shown here since we can do some optimization of the model we train to make it efficient at inference time. One of these optimizations is to use QKeras!\n",
    "\n",
    "You then convert your model using hls4ml. At this point, you specify a configuration to use. This will be covered in more detail, but one of the configuration possibilities is to change the data types. When using QKeras trained models, this can be done automatically.\n",
    "\n",
    "hls4ml maps your model and configuration onto its library of optimized HLS (high level synthesis) implementations of NN inference components. You then run the vendor tools to synthesize and evaluate the inference results. At present, hls4ml targets Xilinx FPGAs with Vivado HLS, while there are ongoing developments targeting Intel FPGAs with Quartus HLS, as well as other FPGA vendors or custom ASICs with Catapult HLS. \n",
    "\n",
    "<img src=\"files/hls4ml_flow.jpg\" alt=\"hls4ml flow\" style=\"width: 700px;\"/>\n",
    "\n",
    "\n",
    "## Reuse Factor\n",
    "The \"reuse factor\" is the simple handle that hls4ml provides to tune the inference parallelism of your model. When running on custom hardware, we can control how the operations of our NN map onto the available resources. `ReuseFactor=1` means that all operations take place in parallel. This mode uses the most resources, but achieves the lowest latency and highest throughput. `ReuseFactor=4` (in the case of the simple Dense layer in the cartoon) means that all operations take place sequentially on the same hardware. This will use fewer resources, but correspondingly take longer latency and have lower throughput.\n",
    "\n",
    "<img src=\"files/hls4ml_reuse.png\" alt=\"hls4ml flow\" style=\"width: 500px;\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load model\n",
    "Load the QKeras model we trained in Section 2 on the MNIST dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.models.load_model('./section2_model_0.h5', custom_objects=co)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convert the model to FPGA firmware with hls4ml\n",
    "Now we will go through the steps to convert the model we trained to a low-latency optimized FPGA firmware with hls4ml.\n",
    "First, we will evaluate its classification performance to make sure we haven't lost accuracy using the fixed-point data types. \n",
    "Then we will synthesize the model with Vivado HLS and check the metrics of latency and FPGA resource usage.\n",
    "\n",
    "## Make an hls4ml config\n",
    "The hls4ml Neural Network inference library is controlled through a configuration dictionary.\n",
    "For QKeras models, when creating the configuration using `granularity='name'`, the data types are automatically set according to the quantizers of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cfg = hls4ml.utils.config_from_keras_model(model, granularity='name')\n",
    "print_dict(cfg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to make some extra tweaks to maintain good performance with this low-precision model:\n",
    "- We set the Model level \"`Strategy`\" to `Resource`, targeting the the inference implementation for larger layers.\n",
    "- We modify the `ReuseFactor` of the first layer of the network, which is by far the largest.\n",
    "- We set the `Strategy` of the second dense layer to `Latency`, since it is much smaller than the first layer\n",
    "- We use `Stratgey : Stable` for the output Softmax layer, which is important for models with high accuracy. For models with lower accuracy, the default `Strategy : Resource` is good enough, and is a bit faster.\n",
    "- Finally, we also need to use rounding, rather than the HLS default truncation for our low-precision activation layers. This is set using an hls4ml Optimizer pass. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cfg['Model'] = {'Precision' : 'ap_fixed<16,6>', 'ReuseFactor' : 1, 'Strategy' : 'Resource'}\n",
    "cfg['LayerName']['q_dense']['ReuseFactor'] = 112\n",
    "cfg['LayerName']['q_dense_1']['Strategy'] = 'Latency'\n",
    "cfg['LayerName']['activation_2']['Strategy'] = 'Stable'\n",
    "print_dict(cfg)\n",
    "# Change the rounding behaviour of activation layers:\n",
    "hls4ml.model.optimizer.OutputRoundingSaturationMode.layers = ['Activation']\n",
    "hls4ml.model.optimizer.OutputRoundingSaturationMode.rounding_mode = 'AP_RND'\n",
    "hls4ml.model.optimizer.OutputRoundingSaturationMode.saturation_mode = 'AP_SAT'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert & Compile\n",
    "Now we convert our QKeras model to an `HLSModel` object, applying our generated configuration. We then `compile` the model, which writes out the HLS project and compiles the fixed-point emulation library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "hls_model = hls4ml.converters.convert_from_keras_model(model,\n",
    "                                                       hls_config=cfg,\n",
    "                                                       output_dir='section2_hls4ml_prj_0',\n",
    "                                                       fpga_part='xcu250-figd2104-2L-e')\n",
    "hls_model.compile()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Profile\n",
    "\n",
    "We can visualise the weights and activations of our model, and check that the configuration makes sense. In the plot, the box-and-whisker show the range of value for the quantity (weight or activation) on the y-axis. The values on the x-axis are presented on logarithmic axis with base-2, so that we can easily read off the number of bits. The grey boxes show the data types given by our hls4ml configuration settings. Since we generated our configuration above automatically from the QKeras model, we shouldn't have to change anything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "(x_train, y_train), (x_test, y_test) = get_train_test_set()\n",
    "hls4ml.model.profiling.numerical(keras_model=model, hls_model=hls_model, X=x_test[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate\n",
    "Now we can run `hls_model.predict` to execute the bit-accurate fixed-point emulation of the FPGA inference code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_hls = hls_model.predict(x_test)\n",
    "y_qke = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accuracy\n",
    "Then let's print the accuracy of the QKeras model as well as the FPGA emulation. If all went well, they should be very similar!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"QKeras Accuracy: {}\".format(accuracy_score(np.argmax(y_test, axis=1), np.argmax(y_qke, axis=1))))\n",
    "print(\"hls4ml Accuracy: {}\".format(accuracy_score(np.argmax(y_test, axis=1), np.argmax(y_hls, axis=1))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Synthesize\n",
    "If we have Xilinx's Vivado software installed, we can synthesize the HLS inference code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hls_model.build(csim=False, synth=True, vsynth=True, export=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Report\n",
    "Now we can look at the synthesis reports. Since many people may not have an installation of the software, we provided a report that we can read. Take particular note of the latency, and the usage of FPGA resources. The first file is the report from Vivado HLS C Synthesis. This fixes the latency in clock-cycles of the design. Often the estimates of other resources can be a little high, so the second file is the report from Vivado logic synthesis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cat files/section2_hls4ml_prj_0/myproject_prj/solution1/syn/report/myproject_csynth.rpt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cat files/section2_hls4ml_prj_0/vivado_synth.rpt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load model\n",
    "Now let's load the second QKeras model we trained in Section 2 on the MNIST dataset. This one used `kernel_quantizer=quantized_po2(4,1)` for the weights of the first layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.models.load_model('./section2_model_1.h5', custom_objects=co)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert & Compile\n",
    "Let's create another hls4ml configuration, again using the QKeras model. Now we convert our QKeras model to an `HLSModel` object, applying our generated configuration. We then `compile` the model, which writes out the HLS project and compiles the fixed-point emulation library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cfg = hls4ml.utils.config_from_keras_model(model, granularity='name')\n",
    "cfg['Model'] = {'Precision' : 'ap_fixed<16,6>', 'ReuseFactor' : 1, 'Strategy' : 'Resource'}\n",
    "cfg['LayerName']['q_dense_2']['ReuseFactor'] = 112\n",
    "cfg['LayerName']['q_dense_3']['Strategy'] = 'Latency'\n",
    "cfg['LayerName']['activation_3']['Strategy'] = 'Stable'\n",
    "print_dict(cfg)\n",
    "# Change the rounding behaviour of activation layers:\n",
    "hls4ml.model.optimizer.OutputRoundingSaturationMode.layers = ['Activation']\n",
    "hls4ml.model.optimizer.OutputRoundingSaturationMode.rounding_mode = 'AP_RND'\n",
    "hls4ml.model.optimizer.OutputRoundingSaturationMode.saturation_mode = 'AP_SAT'\n",
    "hls_model = hls4ml.converters.convert_from_keras_model(model,\n",
    "                                                       hls_config=cfg,\n",
    "                                                       output_dir='section2_hls4ml_prj_1',\n",
    "                                                       fpga_part='xcu250-figd2104-2L-e')\n",
    "hls_model.compile()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Profiling\n",
    "Let's again check the profiling plot, showing the distribution of weights and activations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hls4ml.model.profiling.numerical(keras_model=model, hls_model=hls_model, X=x_test[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate\n",
    "And now evaluate the model performance using both Keras and hls4ml."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_hls = hls_model.predict(x_test)\n",
    "y_qke = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"QKeras Accuracy: {}\".format(accuracy_score(np.argmax(y_test, axis=1), np.argmax(y_qke, axis=1))))\n",
    "print(\"hls4ml Accuracy: {}\".format(accuracy_score(np.argmax(y_test, axis=1), np.argmax(y_hls, axis=1))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Synthesize\n",
    "If we have Xilinx's Vivado software installed, we can synthesize the HLS inference code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hls_model.build(csim=False, synth=True, vsynth=True, export=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Report\n",
    "Now we can look at the synthesis reports. Since many people may not have an installation of the software, we provided a report that we can read. Take particular note of the latency, and the usage of FPGA resources. The first file is the report from Vivado HLS C Synthesis. This fixes the latency in clock-cycles of the design. Often the estimates of other resources can be a little high, so the second file is the report from Vivado logic synthesis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cat files/section2_hls4ml_prj_1/myproject_prj/solution1/syn/report/myproject_csynth.rpt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cat files/section2_hls4ml_prj_1/vivado_synth.rpt"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}