examples/04_managing_log_output.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Log output\n",
    "\n",
    "We use the same example as in the quickstart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "using MIPVerify\n",
    "using HiGHS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist = MIPVerify.read_datasets(\"MNIST\")\n",
    "n1 = MIPVerify.get_example_network_params(\"MNIST.n1\")\n",
    "sample_image = MIPVerify.get_image(mnist.test.images, 1);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Viewing Log Output\n",
    "\n",
    "Our package logs to STDOUT at the `debug`, `info`, `notice` and `warn` levels. \n",
    "The default output level is to show logs only at the `notice` level and above, which is also the recommended level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n",
      "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n",
      "Running HiGHS 1.4.2 [date: 1970-01-01, git hash: f797c1ab6]\n",
      "Copyright (c) 2022 ERGO-Code under MIT licence terms\n",
      "Presolving model\n",
      "1779 rows, 1627 cols, 56716 nonzeros\n",
      "1779 rows, 1627 cols, 56716 nonzeros\n",
      "\n",
      "Solving MIP model with:\n",
      "   1779 rows\n",
      "   1627 cols (29 binary, 0 integer, 0 implied int., 1598 continuous)\n",
      "   56716 nonzeros\n",
      "\n",
      "        Nodes      |    B&B Tree     |            Objective Bounds              |  Dynamic Constraints |       Work      \n",
      "     Proc. InQueue |  Leaves   Expl. | BestBound       BestSol              Gap |   Cuts   InLp Confl. | LpIters     Time\n",
      "\n",
      "         0       0         0   0.00%   0               inf                  inf        0      0      0         0     0.0s\n",
      "         0       0         0   0.00%   0               inf                  inf        0      0     13      1616     0.1s\n",
      "         0       0         0   0.00%   0               inf                  inf     3942     12     99     30330    14.6s\n",
      " T      15       0         3   1.61%   0               0.0498354344     100.00%     3951     12    109     39558    16.3s\n",
      " T      27       1        10   3.42%   0               0.0474230913     100.00%     3974     12    131     45590    17.4s\n",
      " T      45       4        20  77.27%   0.0381825316    0.0460846816      17.15%     4563     15    162     58011    20.1s\n",
      "\n",
      "Solving report\n",
      "  Status            Optimal\n",
      "  Primal bound      0.0460846815889\n",
      "  Dual bound        0.0460846815889\n",
      "  Gap               0% (tolerance: 0.01%)\n",
      "  Solution status   feasible\n",
      "                    0.0460846815889 (objective)\n",
      "                    0 (bound viol.)\n",
      "                    0 (int. viol.)\n",
      "                    0 (row viol.)\n",
      "  Timing            21.37 (total)\n",
      "                    0.02 (presolve)\n",
      "                    0.00 (postsolve)\n",
      "  Nodes             53\n",
      "  LP iterations     60163 (total)\n",
      "                    19015 (strong br.)\n",
      "                    707 (separation)\n",
      "                    28049 (heuristics)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Dict{Any, Any} with 11 entries:\n",
       "  :TargetIndexes      => [10]\n",
       "  :SolveTime          => 21.369\n",
       "  :TotalTime          => 39.5667\n",
       "  :Perturbation       => [_[1] _[2] … _[27] _[28];;; _[29] _[30] … _[55] _[56];…\n",
       "  :PerturbedInput     => [_[785] _[786] … _[811] _[812];;; _[813] _[814] … _[83…\n",
       "  :TighteningApproach => \"lp\"\n",
       "  :PerturbationFamily => linf-norm-bounded-0.05\n",
       "  :SolveStatus        => OPTIMAL\n",
       "  :Model              => A JuMP Model…\n",
       "  :Output             => JuMP.AffExpr[-0.012063867412507534 _[1601] + 0.6606525…\n",
       "  :PredictedIndex     => 8"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "MIPVerify.find_adversarial_example(\n",
    "    n1, \n",
    "    sample_image, \n",
    "    10, \n",
    "    HiGHS.Optimizer,\n",
    "    Dict(),\n",
    "    pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.05),\n",
    "    norm_order = Inf,\n",
    "    tightening_algorithm = lp,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More information is available if we set the log level to `debug`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m[debug | Memento]: Logger(root) is already registered and force=false.\u001b[39m\n",
      "\u001b[36m[notice | MIPVerify]: Attempting to find adversarial example. Neural net predicted label is 8, target labels are [10]\u001b[39m\n",
      "\u001b[36m[notice | MIPVerify]: Determining upper and lower bounds for the input to each non-linear unit.\u001b[39m\n",
      "\u001b[32m[info | MIPVerify]: Applying Flatten() ... \u001b[39m\n",
      "\u001b[32m[info | MIPVerify]: Applying Linear(784 -> 40) ... \u001b[39m\n",
      "\u001b[32m[info | MIPVerify]: Applying ReLU() ...\u001b[39m\n",
      "\u001b[32m[info | MIPVerify]: Applying Linear(40 -> 20) ... \u001b[39m\n",
      "\u001b[32m[info | MIPVerify]: Applying ReLU() ...\u001b[39m\n",
      "\u001b[32m[info | MIPVerify]: Applying Linear(20 -> 10) ... \u001b[39m\n",
      "Running HiGHS 1.4.2 [date: 1970-01-01, git hash: f797c1ab6]\n",
      "Copyright (c) 2022 ERGO-Code under MIT licence terms\n",
      "Presolving model\n",
      "1779 rows, 1627 cols, 56716 nonzeros\n",
      "1779 rows, 1627 cols, 56716 nonzeros\n",
      "\n",
      "Solving MIP model with:\n",
      "   1779 rows\n",
      "   1627 cols (29 binary, 0 integer, 0 implied int., 1598 continuous)\n",
      "   56716 nonzeros\n",
      "\n",
      "        Nodes      |    B&B Tree     |            Objective Bounds              |  Dynamic Constraints |       Work      \n",
      "     Proc. InQueue |  Leaves   Expl. | BestBound       BestSol              Gap |   Cuts   InLp Confl. | LpIters     Time\n",
      "\n",
      "         0       0         0   0.00%   0               inf                  inf        0      0      0         0     0.0s\n",
      "         0       0         0   0.00%   0               inf                  inf        0      0     13      1616     0.1s\n",
      "         0       0         0   0.00%   0               inf                  inf     3942     12     99     30330    14.3s\n",
      " T      15       0         3   1.61%   0               0.0498354344     100.00%     3951     12    109     39558    16.1s\n",
      " T      27       1        10   3.42%   0               0.0474230913     100.00%     3974     12    131     45590    17.1s\n",
      " T      45       4        20  77.27%   0.0381825316    0.0460846816      17.15%     4563     15    162     58011    20.2s\n",
      "\n",
      "Solving report\n",
      "  Status            Optimal\n",
      "  Primal bound      0.0460846815889\n",
      "  Dual bound        0.0460846815889\n",
      "  Gap               0% (tolerance: 0.01%)\n",
      "  Solution status   feasible\n",
      "                    0.0460846815889 (objective)\n",
      "                    0 (bound viol.)\n",
      "                    0 (int. viol.)\n",
      "                    0 (row viol.)\n",
      "  Timing            21.52 (total)\n",
      "                    0.02 (presolve)\n",
      "                    0.00 (postsolve)\n",
      "  Nodes             53\n",
      "  LP iterations     60163 (total)\n",
      "                    19015 (strong br.)\n",
      "                    707 (separation)\n",
      "                    28049 (heuristics)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Dict{Any, Any} with 11 entries:\n",
       "  :TargetIndexes      => [10]\n",
       "  :SolveTime          => 21.5182\n",
       "  :TotalTime          => 22.2792\n",
       "  :Perturbation       => [_[1] _[2] … _[27] _[28];;; _[29] _[30] … _[55] _[56];…\n",
       "  :PerturbedInput     => [_[785] _[786] … _[811] _[812];;; _[813] _[814] … _[83…\n",
       "  :TighteningApproach => \"lp\"\n",
       "  :PerturbationFamily => linf-norm-bounded-0.05\n",
       "  :SolveStatus        => OPTIMAL\n",
       "  :Model              => A JuMP Model…\n",
       "  :Output             => JuMP.AffExpr[-0.012063867412507534 _[1601] + 0.6606525…\n",
       "  :PredictedIndex     => 8"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "MIPVerify.set_log_level!(\"debug\")\n",
    "\n",
    "MIPVerify.find_adversarial_example(\n",
    "    n1, \n",
    "    sample_image, \n",
    "    10, \n",
    "    HiGHS.Optimizer,\n",
    "    Dict(),\n",
    "    pp = MIPVerify.LInfNormBoundedPerturbationFamily(0.05),\n",
    "    norm_order = Inf,\n",
    "    tightening_algorithm = lp,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handling log output\n",
    "\n",
    "We use the logger provided by the [`Memento`](https://github.com/invenia/Memento.jl) package. Rather than just having logs written to STDOUT, you can choose to capture the output (for example, to write logs to file) by adding your own handlers to `MIPVerify.LOGGER`. See the [Memento documentation](https://invenia.github.io/Memento.jl/latest/) for more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Memento.Logger"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "typeof(MIPVerify.LOGGER)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.7.1",
   "language": "julia",
   "name": "julia-1.7"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}