docs/notebooks/neuralnet/mnist_example_dense.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example: Optimal adversaries for dense MNIST model\n",
    "\n",
    "This notebook gives an example where OMLT is used to find adversarial examples for a trained dense neural network. We follow the below steps:<br>\n",
    "1.) A neural network with ReLU activation functions is trained to classify images from the MNIST dataset <br>\n",
    "2.) OMLT is used to generate a mixed-integer encoding of the trained model using the big-M formulation <br>\n",
    "3.) The model is optimized to find the maximum classification error (defined by an \"adversarial\" label) over a small input region <br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Library Setup\n",
    "This notebook assumes you have a working PyTorch environment to train the neural network for classification. The neural network is then formulated in Pyomo using OMLT which therefore requires working Pyomo and OMLT installations.\n",
    "\n",
    "The required Python libraries used this notebook are as follows: <br>\n",
    "- `numpy`: used for manipulate input data <br>\n",
    "- `torch`: the machine learning language we use to train our neural network\n",
    "- `torchvision`: a package containing the MNIST dataset\n",
    "- `pyomo`: the algebraic modeling language for Python, it is used to define the optimization model passed to the solver\n",
    "- `onnx`: used to express trained neural network models\n",    
    "- `omlt`: the package this notebook demonstates. OMLT can formulate machine learning models (such as neural networks) within Pyomo\n",
    "\n",
    "**NOTE:** This notebook also assumes you have a working MIP solver executable (e.g., CBC, Gurobi) to solve optimization problems in Pyomo. The open-source solver CBC is called by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import requisite packages\n",
    "#data manipulation\n",
    "import numpy as np\n",
    "import tempfile\n",
    "\n",
    "#pytorch for training neural network\n",
    "import torch, torch.onnx\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torchvision import datasets, transforms\n",
    "from torch.optim.lr_scheduler import StepLR\n",
    "\n",
    "#pyomo for optimization\n",
    "import pyomo.environ as pyo\n",
    "\n",
    "#omlt for interfacing our neural network with pyomo\n",
    "from omlt import OmltBlock\n",
    "from omlt.neuralnet import FullSpaceNNFormulation\n",
    "from omlt.io.onnx import write_onnx_model_with_bounds, load_onnx_neural_network_with_bounds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the Data and Train a Neural Network\n",
    "\n",
    "We begin by loading the MNIST dataset as `DataLoader` objects with pre-set training and testing batch sizes:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ../data/MNIST/raw/train-images-idx3-ubyte.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100.0%\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ../data/MNIST/raw/train-images-idx3-ubyte.gz to ../data/MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to ../data/MNIST/raw/train-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "102.8%\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ../data/MNIST/raw/train-labels-idx1-ubyte.gz to ../data/MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to ../data/MNIST/raw/t10k-images-idx3-ubyte.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100.0%\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ../data/MNIST/raw/t10k-images-idx3-ubyte.gz to ../data/MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to ../data/MNIST/raw/t10k-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "112.7%"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ../data/MNIST/raw/t10k-labels-idx1-ubyte.gz to ../data/MNIST/raw\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "#set training and test batch sizes\n",
    "train_kwargs = {'batch_size': 64}\n",
    "test_kwargs = {'batch_size': 1000}\n",
    "\n",
    "#build DataLoaders for training and test sets\n",
    "dataset1 = datasets.MNIST('../data', train=True, download=True, transform=transforms.ToTensor())\n",
    "dataset2 = datasets.MNIST('../data', train=False, transform=transforms.ToTensor())\n",
    "train_loader = torch.utils.data.DataLoader(dataset1,**train_kwargs, shuffle=True)\n",
    "test_loader = torch.utils.data.DataLoader(dataset2, **test_kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we define the structure of the dense neural network model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "hidden_size = 50\n",
    "\n",
    "class Net(nn.Module):\n",
    "    #define layers of neural network\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.hidden1  = nn.Linear(784, hidden_size)\n",
    "        self.hidden2  = nn.Linear(hidden_size, hidden_size)\n",
    "        self.output  = nn.Linear(hidden_size, 10)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.softmax = nn.LogSoftmax(dim=1)\n",
    "\n",
    "    #define forward pass of neural network\n",
    "    def forward(self, x):\n",
    "        x = self.hidden1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.hidden2(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.output(x)\n",
    "        x = self.softmax(x)      \n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We next define simple functions for training and testing the neural network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#training function computes loss and its gradient on batch, and prints status after every 200 batches\n",
    "def train(model, train_loader, optimizer, epoch):\n",
    "    model.train(); criterion = nn.NLLLoss()\n",
    "    for batch_idx, (data, target) in enumerate(train_loader):\n",
    "        optimizer.zero_grad()\n",
    "        output = model(data.view(-1, 28*28))\n",
    "        loss = criterion(output, target)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        if batch_idx % 200  == 0:\n",
    "            print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n",
    "                epoch, batch_idx * len(data), len(train_loader.dataset),\n",
    "                100. * batch_idx / len(train_loader), loss.item()))\n",
    "\n",
    "#testing function computes loss and prints overall model accuracy on test set\n",
    "def test(model, test_loader):\n",
    "    model.eval(); criterion = nn.NLLLoss( reduction='sum')\n",
    "    test_loss = 0; correct = 0\n",
    "    with torch.no_grad():\n",
    "        for data, target in test_loader:\n",
    "            output = model(data.view(-1, 28*28))\n",
    "            test_loss += criterion(output, target).item()  \n",
    "            pred = output.argmax(dim=1, keepdim=True) \n",
    "            correct += pred.eq(target.view_as(pred)).sum().item()\n",
    "    test_loss /= len(test_loader.dataset)\n",
    "    print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n",
    "        test_loss, correct, len(test_loader.dataset), 100. * correct / len(test_loader.dataset)))            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we train the neural network on the dataset.\n",
    "Training here is performed using the `Adadelta` optimizer for five epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Epoch: 0 [0/60000 (0%)]\tLoss: 2.312474\n",
      "Train Epoch: 0 [12800/60000 (21%)]\tLoss: 0.433773\n",
      "Train Epoch: 0 [25600/60000 (43%)]\tLoss: 0.337540\n",
      "Train Epoch: 0 [38400/60000 (64%)]\tLoss: 0.466846\n",
      "Train Epoch: 0 [51200/60000 (85%)]\tLoss: 0.088567\n",
      "\n",
      "Test set: Average loss: 0.1634, Accuracy: 9508/10000 (95%)\n",
      "\n",
      "Train Epoch: 1 [0/60000 (0%)]\tLoss: 0.137867\n",
      "Train Epoch: 1 [12800/60000 (21%)]\tLoss: 0.057379\n",
      "Train Epoch: 1 [25600/60000 (43%)]\tLoss: 0.045729\n",
      "Train Epoch: 1 [38400/60000 (64%)]\tLoss: 0.377446\n",
      "Train Epoch: 1 [51200/60000 (85%)]\tLoss: 0.218694\n",
      "\n",
      "Test set: Average loss: 0.1208, Accuracy: 9630/10000 (96%)\n",
      "\n",
      "Train Epoch: 2 [0/60000 (0%)]\tLoss: 0.133075\n",
      "Train Epoch: 2 [12800/60000 (21%)]\tLoss: 0.137646\n",
      "Train Epoch: 2 [25600/60000 (43%)]\tLoss: 0.026231\n",
      "Train Epoch: 2 [38400/60000 (64%)]\tLoss: 0.020423\n",
      "Train Epoch: 2 [51200/60000 (85%)]\tLoss: 0.073325\n",
      "\n",
      "Test set: Average loss: 0.1031, Accuracy: 9677/10000 (97%)\n",
      "\n",
      "Train Epoch: 3 [0/60000 (0%)]\tLoss: 0.037360\n",
      "Train Epoch: 3 [12800/60000 (21%)]\tLoss: 0.119995\n",
      "Train Epoch: 3 [25600/60000 (43%)]\tLoss: 0.018661\n",
      "Train Epoch: 3 [38400/60000 (64%)]\tLoss: 0.071436\n",
      "Train Epoch: 3 [51200/60000 (85%)]\tLoss: 0.048075\n",
      "\n",
      "Test set: Average loss: 0.0930, Accuracy: 9713/10000 (97%)\n",
      "\n",
      "Train Epoch: 4 [0/60000 (0%)]\tLoss: 0.031118\n",
      "Train Epoch: 4 [12800/60000 (21%)]\tLoss: 0.022899\n",
      "Train Epoch: 4 [25600/60000 (43%)]\tLoss: 0.052135\n",
      "Train Epoch: 4 [38400/60000 (64%)]\tLoss: 0.047121\n",
      "Train Epoch: 4 [51200/60000 (85%)]\tLoss: 0.053384\n",
      "\n",
      "Test set: Average loss: 0.0881, Accuracy: 9728/10000 (97%)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#define model and optimizer\n",
    "model = Net()\n",
    "optimizer = optim.Adadelta(model.parameters(), lr=1)\n",
    "scheduler = StepLR(optimizer, step_size=1, gamma=0.7)\n",
    "\n",
    "#train neural network for five epochs\n",
    "for epoch in range(5):\n",
    "    train(model, train_loader, optimizer, epoch)\n",
    "    test(model, test_loader)\n",
    "    scheduler.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a MIP Formulation for the Trained Neural Network\n",
    "\n",
    "We are now ready to use OMLT to formulate the trained model within a Pyomo optimization model. We use a  binary variable representation to represent the ReLU activation logic which is created automatically using the `FullSpaceNNFormulation` object (one could also simply use the `ReluBigFormulation` object)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we define a neural network without the final `LogSoftmax` activation. Although this activation helps greatly in training the neural network model, it is not trivial to encode in the optimization model. The ranking of the output labels remains the same without the activation, so it can be omitted when finding optimal adversaries. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class NoSoftmaxNet(nn.Module):\n",
    "    #define layers of neural network\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.hidden1  = nn.Linear(784, hidden_size)\n",
    "        self.hidden2  = nn.Linear(hidden_size, hidden_size)\n",
    "        self.output  = nn.Linear(hidden_size, 10)\n",
    "        self.relu = nn.ReLU()\n",
    "\n",
    "    #define forward pass of neural network\n",
    "    def forward(self, x):\n",
    "        x = self.hidden1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.hidden2(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.output(x)\n",
    "        return x\n",
    "\n",
    "#create neural network without LogSoftmax and load parameters from existing model\n",
    "model2 = NoSoftmaxNet()\n",
    "model2.load_state_dict(model.state_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we define an instance of the optimal adversary problem. We formulate the optimization problem as: <br>\n",
    "\n",
    "$\n",
    "\\begin{align*} \n",
    "& \\max_x \\ y_k - y_j \\\\\n",
    "& s.t. y_k = N_k(x) \\\\ \n",
    "&\\quad |x - \\bar{x}|_\\infty \\leq 0.05\n",
    "\\end{align*}\n",
    "$\n",
    "\n",
    "where $\\bar{x}$ corresponds to an image in the test dataset with true label `j`, and $N_k(x)$ is the value of the neural network output corresponding to adversarial label `k` given input `x`. PyTorch needs to trace the model execution to export it to ONNX, so we also define a dummy input tensor `x_temp`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#load image and true label from test set with index 'problem_index'\n",
    "problem_index = 0\n",
    "image = dataset2[problem_index][0].view(-1,28*28).detach().numpy()\n",
    "label = dataset2[problem_index][1]\n",
    "\n",
    "#define input region defined by infinity norm\n",
    "epsilon_infty = 5e-2\n",
    "lb = np.maximum(0, image - epsilon_infty)\n",
    "ub = np.minimum(1, image + epsilon_infty)\n",
    "\n",
    "#save input bounds as dictionary\n",
    "input_bounds = {}\n",
    "for i in range(28*28):\n",
    "    input_bounds[i] = (float(lb[0][i]), float(ub[0][i])) \n",
    "    \n",
    "#define dummy input tensor    \n",
    "x_temp = dataset2[problem_index][0].view(-1,28*28)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now export the PyTorch model as an ONNX model and use `load_onnx_neural_network_with_bounds` to load it into OMLT."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tempfile.NamedTemporaryFile(suffix='.onnx', delete=False) as f:\n",
    "    #export neural network to ONNX\n",
    "    torch.onnx.export(\n",
    "        model2,\n",
    "        x_temp,\n",
    "        f,\n",
    "        input_names=['input'],\n",
    "        output_names=['output'],\n",
    "        dynamic_axes={\n",
    "            'input': {0: 'batch_size'},\n",
    "            'output': {0: 'batch_size'}\n",
    "        }\n",
    "    )\n",
    "    #write ONNX model and its bounds using OMLT\n",
    "    write_onnx_model_with_bounds(f.name, None, input_bounds)\n",
    "    #load the network definition from the ONNX model\n",
    "    network_definition = load_onnx_neural_network_with_bounds(f.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a sanity check before creating the optimization model, we can print the properties of the neural network layers from `network_definition`. This allows us to check input/output sizes, as well as activation functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\tInputLayer(input_size=[784], output_size=[784])\tlinear\n",
      "1\tDenseLayer(input_size=[784], output_size=[50])\trelu\n",
      "2\tDenseLayer(input_size=[50], output_size=[50])\trelu\n",
      "3\tDenseLayer(input_size=[50], output_size=[10])\tlinear\n"
     ]
    }
   ],
   "source": [
    "for layer_id, layer in enumerate(network_definition.layers):\n",
    "    print(f\"{layer_id}\\t{layer}\\t{layer.activation}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can load `network_definition` as a full-space `FullSpaceNNFormulation` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "formulation = FullSpaceNNFormulation(network_definition)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solve Optimal Adversary Problem in Pyomo\n",
    "\n",
    "We now encode the trained neural network in a Pyomo model from the `FullSpaceNNFormulation` object. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create pyomo model\n",
    "m = pyo.ConcreteModel()\n",
    "\n",
    "#create an OMLT block for the neural network and build its formulation\n",
    "m.nn = OmltBlock()\n",
    "m.nn.build_formulation(formulation) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we define an adversarial label as the true label plus one (or zero if the true label is nine), as well as the objective function for optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "adversary = (label + 1) % 10\n",
    "m.obj = pyo.Objective(expr=(-(m.nn.outputs[adversary]-m.nn.outputs[label])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we solve the optimal adversary problem using a mixed-integer solver."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Welcome to the CBC MILP Solver \n",
      "Version: 2.10.5 \n",
      "Build Date: Oct 15 2020 \n",
      "\n",
      "command line - /home/jhjalvi/anaconda3/envs/tensorflow/bin/cbc -printingOptions all -import /tmp/tmpdwk9ljju.pyomo.lp -stat=1 -solve -solu /tmp/tmpdwk9ljju.pyomo.soln (default strategy 1)\n",
      "Option for printingOptions changed from normal to all\n",
      "Presolve 332 (-1777) rows, 1029 (-1664) columns and 31506 (-14801) elements\n",
      "Statistics for presolved model\n",
      "Original problem has 100 integers (100 of which binary)\n",
      "Presolved problem has 71 integers (71 of which binary)\n",
      "==== 979 zero objective 51 different\n",
      "==== absolute objective values 51 different\n",
      "==== for integers 71 zero objective 1 different\n",
      "71 variables have objective of 0\n",
      "==== for integers absolute objective values 1 different\n",
      "71 variables have objective of 0\n",
      "===== end objective counts\n",
      "\n",
      "\n",
      "Problem has 332 rows, 1029 columns (50 with objective) and 31506 elements\n",
      "Column breakdown:\n",
      "0 of type 0.0->inf, 759 of type 0.0->up, 0 of type lo->inf, \n",
      "199 of type lo->up, 0 of type free, 0 of type fixed, \n",
      "0 of type -inf->0.0, 0 of type -inf->up, 71 of type 0.0->1.0 \n",
      "Row breakdown:\n",
      "0 of type E 0.0, 0 of type E 1.0, 0 of type E -1.0, \n",
      "87 of type E other, 0 of type G 0.0, 0 of type G 1.0, \n",
      "0 of type G other, 174 of type L 0.0, 0 of type L 1.0, \n",
      "71 of type L other, 0 of type Range 0.0->1.0, 0 of type Range other, \n",
      "0 of type Free \n",
      "Continuous objective value is -8.70513 - 0.03 seconds\n",
      "Cgl0003I 0 fixed, 0 tightened bounds, 54 strengthened rows, 0 substitutions\n",
      "Cgl0003I 0 fixed, 0 tightened bounds, 1 strengthened rows, 0 substitutions\n",
      "Cgl0004I processed model has 258 rows, 955 columns (63 integer (63 of which binary)) and 52403 elements\n",
      "Cbc0038I Initial state - 45 integers unsatisfied sum - 16.9175\n",
      "Cbc0038I Pass   1: suminf.    6.67376 (26) obj. 8.1543 iterations 369\n",
      "Cbc0038I Pass   2: suminf.    0.00000 (0) obj. 10.7826 iterations 1051\n",
      "Cbc0038I Solution found of 10.7826\n",
      "Cbc0038I Relaxing continuous gives 7.01412\n",
      "Cbc0038I Before mini branch and bound, 18 integers at bound fixed and 356 continuous\n",
      "Cbc0038I Full problem 258 rows 955 columns, reduced to 195 rows 558 columns - 6 fixed gives 189, 552 - still too large\n",
      "Cbc0038I Full problem 258 rows 955 columns, reduced to 167 rows 535 columns - too large\n",
      "Cbc0038I Mini branch and bound did not improve solution (0.35 seconds)\n",
      "Cbc0038I Round again with cutoff of 5.95662\n",
      "Cbc0038I Pass   3: suminf.    7.02236 (29) obj. 5.95662 iterations 60\n",
      "Cbc0038I Pass   4: suminf.    1.95668 (13) obj. 5.95662 iterations 412\n",
      "Cbc0038I Pass   5: suminf.    0.37202 (1) obj. 5.95662 iterations 778\n",
      "Cbc0038I Pass   6: suminf.    0.20711 (1) obj. 5.95662 iterations 35\n",
      "Cbc0038I Pass   7: suminf.    1.75379 (6) obj. 5.95662 iterations 390\n",
      "Cbc0038I Pass   8: suminf.    0.24439 (2) obj. 5.95662 iterations 172\n",
      "Cbc0038I Pass   9: suminf.    0.31716 (1) obj. 5.95662 iterations 129\n",
      "Cbc0038I Pass  10: suminf.    0.15007 (1) obj. 5.95662 iterations 46\n",
      "Cbc0038I Pass  11: suminf.    2.30947 (9) obj. 5.95662 iterations 233\n",
      "Cbc0038I Pass  12: suminf.    0.21814 (1) obj. 5.95662 iterations 221\n",
      "Cbc0038I Pass  13: suminf.    0.37586 (1) obj. 5.95662 iterations 44\n",
      "Cbc0038I Pass  14: suminf.    3.21707 (16) obj. 5.95662 iterations 266\n",
      "Cbc0038I Pass  15: suminf.    2.95868 (15) obj. 5.95662 iterations 20\n",
      "Cbc0038I Pass  16: suminf.    2.95331 (15) obj. 5.95662 iterations 22\n",
      "Cbc0038I Pass  17: suminf.    0.31716 (1) obj. 5.95662 iterations 812\n",
      "Cbc0038I Pass  18: suminf.    0.15007 (1) obj. 5.95662 iterations 39\n",
      "Cbc0038I Pass  19: suminf.    3.13470 (12) obj. 5.95662 iterations 403\n",
      "Cbc0038I Pass  20: suminf.    2.54532 (10) obj. 5.95662 iterations 94\n",
      "Cbc0038I Pass  21: suminf.    5.93299 (26) obj. 5.95662 iterations 340\n",
      "Cbc0038I Pass  22: suminf.    0.28804 (1) obj. 5.95662 iterations 562\n",
      "Cbc0038I Pass  23: suminf.    0.10862 (1) obj. 5.95662 iterations 48\n",
      "Cbc0038I Pass  24: suminf.    2.17321 (10) obj. 5.95662 iterations 298\n",
      "Cbc0038I Pass  25: suminf.    0.28804 (1) obj. 5.95662 iterations 289\n",
      "Cbc0038I Pass  26: suminf.    0.10862 (1) obj. 5.95662 iterations 48\n",
      "Cbc0038I Pass  27: suminf.    3.49937 (14) obj. 5.95662 iterations 289\n",
      "Cbc0038I Pass  28: suminf.    0.37176 (1) obj. 5.95662 iterations 208\n",
      "Cbc0038I Pass  29: suminf.    0.20074 (1) obj. 5.95662 iterations 26\n",
      "Cbc0038I Pass  30: suminf.    0.91276 (5) obj. 5.95662 iterations 120\n",
      "Cbc0038I Pass  31: suminf.    2.70655 (11) obj. 5.95662 iterations 250\n",
      "Cbc0038I Pass  32: suminf.    2.64846 (13) obj. 5.95662 iterations 32\n",
      "Cbc0038I Rounding solution of 6.24461 is better than previous of 7.01412\n",
      "\n",
      "Cbc0038I Before mini branch and bound, 3 integers at bound fixed and 353 continuous\n",
      "Cbc0038I Full problem 258 rows 955 columns, reduced to 200 rows 566 columns - 10 fixed gives 190, 556 - still too large\n",
      "Cbc0038I Full problem 258 rows 955 columns, reduced to 163 rows 534 columns - too large\n",
      "Cbc0038I Mini branch and bound did not improve solution (1.01 seconds)\n",
      "Cbc0038I Round again with cutoff of 4.05313\n",
      "Cbc0038I Pass  32: suminf.    7.70811 (30) obj. 4.05313 iterations 7\n",
      "Cbc0038I Pass  33: suminf.    4.89653 (23) obj. 4.05313 iterations 121\n",
      "Cbc0038I Pass  34: suminf.    4.88223 (23) obj. 4.05313 iterations 23\n",
      "Cbc0038I Pass  35: suminf.    1.20216 (3) obj. 4.05313 iterations 1192\n",
      "Cbc0038I Pass  36: suminf.    0.91596 (3) obj. 4.05313 iterations 61\n",
      "Cbc0038I Pass  37: suminf.    0.90941 (3) obj. 4.05313 iterations 27\n",
      "Cbc0038I Pass  38: suminf.    2.77624 (12) obj. 4.05313 iterations 160\n",
      "Cbc0038I Pass  39: suminf.    1.18048 (3) obj. 4.05313 iterations 356\n",
      "Cbc0038I Pass  40: suminf.    0.86798 (2) obj. 4.05313 iterations 73\n",
      "Cbc0038I Pass  41: suminf.    0.85671 (3) obj. 4.05313 iterations 39\n",
      "Cbc0038I Pass  42: suminf.    2.64239 (10) obj. 4.05313 iterations 103\n",
      "Cbc0038I Pass  43: suminf.    1.20892 (3) obj. 4.05313 iterations 119\n",
      "Cbc0038I Pass  44: suminf.    0.90215 (3) obj. 4.05313 iterations 59\n",
      "Cbc0038I Pass  45: suminf.    0.89043 (3) obj. 4.05313 iterations 43\n",
      "Cbc0038I Pass  46: suminf.    4.06118 (14) obj. 4.05313 iterations 330\n",
      "Cbc0038I Pass  47: suminf.    3.86419 (13) obj. 4.05313 iterations 33\n",
      "Cbc0038I Pass  48: suminf.    0.92252 (3) obj. 4.05313 iterations 242\n",
      "Cbc0038I Pass  49: suminf.    4.29719 (14) obj. 4.05313 iterations 350\n",
      "Cbc0038I Pass  50: suminf.    4.17818 (15) obj. 4.05313 iterations 53\n",
      "Cbc0038I Pass  51: suminf.    0.91809 (3) obj. 4.05313 iterations 188\n",
      "Cbc0038I Pass  52: suminf.    0.89043 (3) obj. 4.05313 iterations 61\n",
      "Cbc0038I Pass  53: suminf.    1.21047 (3) obj. 4.05313 iterations 106\n",
      "Cbc0038I Pass  54: suminf.    0.89965 (3) obj. 4.05313 iterations 72\n",
      "Cbc0038I Pass  55: suminf.    2.95986 (11) obj. 4.05313 iterations 232\n",
      "Cbc0038I Pass  56: suminf.    2.80081 (11) obj. 4.05313 iterations 64\n",
      "Cbc0038I Pass  57: suminf.    0.97287 (3) obj. 4.05313 iterations 198\n",
      "Cbc0038I Pass  58: suminf.    0.95556 (3) obj. 4.05313 iterations 53\n",
      "Cbc0038I Pass  59: suminf.    1.23943 (3) obj. 4.05313 iterations 110\n",
      "Cbc0038I Pass  60: suminf.    0.96379 (3) obj. 4.05313 iterations 73\n",
      "Cbc0038I Pass  61: suminf.    3.20098 (12) obj. 4.05313 iterations 340\n",
      "Cbc0038I No solution found this major pass\n",
      "Cbc0038I Before mini branch and bound, 6 integers at bound fixed and 376 continuous\n",
      "Cbc0038I Full problem 258 rows 955 columns, reduced to 196 rows 538 columns - 6 fixed gives 190, 532 - still too large\n",
      "Cbc0038I Full problem 258 rows 955 columns, reduced to 164 rows 512 columns - too large\n",
      "Cbc0038I Mini branch and bound did not improve solution (1.49 seconds)\n",
      "Cbc0038I After 1.49 seconds - Feasibility pump exiting with objective of 6.24461 - took 1.35 seconds\n",
      "Cbc0012I Integer solution of 6.2446143 found by feasibility pump after 0 iterations and 0 nodes (1.54 seconds)\n",
      "Cbc0038I Full problem 258 rows 955 columns, reduced to 215 rows 912 columns - 30 fixed gives 184, 881 - still too large\n",
      "Cbc0031I 34 added rows had average density of 383.97059\n",
      "Cbc0013I At root node, 34 cuts changed objective from -3.5608111 to 0.11409384 in 100 passes\n",
      "Cbc0014I Cut generator 0 (Probing) - 132 row cuts average 52.4 elements, 0 column cuts (0 active)  in 0.083 seconds - new frequency is -100\n",
      "Cbc0014I Cut generator 1 (Gomory) - 1147 row cuts average 815.8 elements, 0 column cuts (0 active)  in 1.221 seconds - new frequency is -100\n",
      "Cbc0014I Cut generator 2 (Knapsack) - 3 row cuts average 3.7 elements, 0 column cuts (0 active)  in 0.073 seconds - new frequency is -100\n",
      "Cbc0014I Cut generator 3 (Clique) - 0 row cuts average 0.0 elements, 0 column cuts (0 active)  in 0.003 seconds - new frequency is -100\n",
      "Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 3392 row cuts average 281.4 elements, 0 column cuts (0 active)  in 0.717 seconds - new frequency is 1\n",
      "Cbc0014I Cut generator 5 (FlowCover) - 0 row cuts average 0.0 elements, 0 column cuts (0 active)  in 1.634 seconds - new frequency is -100\n",
      "Cbc0014I Cut generator 6 (TwoMirCuts) - 290 row cuts average 367.1 elements, 0 column cuts (0 active)  in 0.132 seconds - new frequency is 1\n",
      "Cbc0010I After 0 nodes, 1 on tree, 6.2446143 best solution, best possible 0.11409384 (9.72 seconds)\n",
      "Cbc0038I Full problem 258 rows 955 columns, reduced to 215 rows 910 columns - 18 fixed gives 195, 890 - still too large\n",
      "Cbc0038I Full problem 258 rows 955 columns, reduced to 212 rows 907 columns - 18 fixed gives 192, 887 - still too large\n",
      "Cbc0001I Search completed - best objective 6.244614291599583, took 69539 iterations and 156 nodes (29.56 seconds)\n",
      "Cbc0032I Strong branching done 1052 times (53050 iterations), fathomed 1 nodes and fixed 0 variables\n",
      "Cbc0035I Maximum depth 31, 2 variables fixed on reduced cost\n",
      "Cuts at root node changed objective from -3.56081 to 0.114094\n",
      "Probing was tried 100 times and created 132 cuts of which 0 were active after adding rounds of cuts (0.083 seconds)\n",
      "Gomory was tried 100 times and created 1147 cuts of which 0 were active after adding rounds of cuts (1.221 seconds)\n",
      "Knapsack was tried 100 times and created 3 cuts of which 0 were active after adding rounds of cuts (0.073 seconds)\n",
      "Clique was tried 100 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.003 seconds)\n",
      "MixedIntegerRounding2 was tried 446 times and created 11432 cuts of which 0 were active after adding rounds of cuts (3.153 seconds)\n",
      "FlowCover was tried 100 times and created 0 cuts of which 0 were active after adding rounds of cuts (1.634 seconds)\n",
      "TwoMirCuts was tried 446 times and created 290 cuts of which 0 were active after adding rounds of cuts (0.406 seconds)\n",
      "ZeroHalf was tried 1 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)\n",
      "\n",
      "Result - Optimal solution found\n",
      "\n",
      "Objective value:                6.24461429\n",
      "Enumerated nodes:               156\n",
      "Total iterations:               69539\n",
      "Time (CPU seconds):             29.77\n",
      "Time (Wallclock seconds):       31.00\n",
      "\n",
      "Total time (CPU seconds):       29.80   (Wallclock seconds):       31.04\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'Problem': [{'Name': 'unknown', 'Lower bound': 6.24461429, 'Upper bound': 6.24461429, 'Number of objectives': 1, 'Number of constraints': 332, 'Number of variables': 1029, 'Number of binary variables': 100, 'Number of integer variables': 100, 'Number of nonzeros': 50, 'Sense': 'minimize'}], 'Solver': [{'Status': 'ok', 'User time': -1.0, 'System time': 29.8, 'Wallclock time': 31.04, 'Termination condition': 'optimal', 'Termination message': 'Model was solved to optimality (subject to tolerances), and an optimal solution is available.', 'Statistics': {'Branch and bound': {'Number of bounded subproblems': 156, 'Number of created subproblems': 156}, 'Black box': {'Number of iterations': 69539}}, 'Error rc': 0, 'Time': 31.065782070159912}], 'Solution': [OrderedDict([('number of solutions', 0), ('number of solutions displayed', 0)])]}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyo.SolverFactory('cbc').solve(m, tee=True)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "b8f31a8284ce774e9ad8d309790c576c984c0620550967f9ef361ac8e66f487d"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}