example/Example_StarDiscr.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0baf691e-b0d5-4acd-8057-a4c4e99ddb49",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cb84e58-8b54-4d61-b5ff-08c845b13d4f",
   "metadata": {},
   "source": [
    "This notebook illustrates how to access the Submodular problems using IOHexperimenter.\n",
    "To install IOHexperimenter, the following command can be used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56f53a36-e40c-4ff3-9c08-12b8745524a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "#%pip install ioh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1d6d79b-8aa4-419e-80c7-15b7fd0c2492",
   "metadata": {},
   "source": [
    "Once installed, simply import the package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3bfc62ad-24cd-4dc6-b3d1-cd9d2293631b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ioh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b509ddaa-446d-4629-b118-f4dbc08ab44f",
   "metadata": {},
   "source": [
    "# Accessing problems"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d87a4346-c909-4c6e-8de8-6f80821fce69",
   "metadata": {},
   "source": [
    "In ioh, everything revolves around the problem class, so we start by creating an individual submodular problem.\n",
    "To see which problems are available, we can check:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f295e32c-aa60-45d9-88e3-fad61ca35dc5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{30: 'UniformStarDiscrepancy10',\n",
       " 31: 'UniformStarDiscrepancy25',\n",
       " 32: 'UniformStarDiscrepancy50',\n",
       " 33: 'UniformStarDiscrepancy100',\n",
       " 34: 'UniformStarDiscrepancy150',\n",
       " 35: 'UniformStarDiscrepancy200',\n",
       " 36: 'UniformStarDiscrepancy250',\n",
       " 37: 'UniformStarDiscrepancy500',\n",
       " 38: 'UniformStarDiscrepancy750',\n",
       " 39: 'UniformStarDiscrepancy1000',\n",
       " 40: 'SobolStarDiscrepancy10',\n",
       " 41: 'SobolStarDiscrepancy25',\n",
       " 42: 'SobolStarDiscrepancy50',\n",
       " 43: 'SobolStarDiscrepancy100',\n",
       " 44: 'SobolStarDiscrepancy150',\n",
       " 45: 'SobolStarDiscrepancy200',\n",
       " 46: 'SobolStarDiscrepancy250',\n",
       " 47: 'SobolStarDiscrepancy500',\n",
       " 48: 'SobolStarDiscrepancy750',\n",
       " 49: 'SobolStarDiscrepancy1000',\n",
       " 50: 'HaltonStarDiscrepancy10',\n",
       " 51: 'HaltonStarDiscrepancy25',\n",
       " 52: 'HaltonStarDiscrepancy50',\n",
       " 53: 'HaltonStarDiscrepancy100',\n",
       " 54: 'HaltonStarDiscrepancy150',\n",
       " 55: 'HaltonStarDiscrepancy200',\n",
       " 56: 'HaltonStarDiscrepancy250',\n",
       " 57: 'HaltonStarDiscrepancy500',\n",
       " 58: 'HaltonStarDiscrepancy750',\n",
       " 59: 'HaltonStarDiscrepancy1000'}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ioh.ProblemType.STAR.problems"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea607fe2-305f-4745-bf52-22f222e12886",
   "metadata": {},
   "source": [
    "We can then construct a problem as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a8a575fc-2eed-4591-88a1-8b7df038dbc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "f = ioh.get_problem(30, instance=1, dimension=5, problem_type=ioh.ProblemType.STAR)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01d7975c-490e-4a19-8b33-2b0e8f1903fd",
   "metadata": {},
   "source": [
    "This problem contains all kinds of information about the way it is constructed. It also keeps track of all evaluations, which can be seen as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6af1db0b-2ac5-4b51-8ced-1a03bb81272f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<MetaData: UniformStarDiscrepancy10 id: 30 iid: 1 dim: 5>\n",
      "<State evaluations: 0 optimum_found: false current_best: <Solution x: [2.247116418577895e+308, 2.247116418577895e+308, 2.247116418577895e+308, 2.247116418577895e+308, 2.247116418577895e+308] y: -inf>>\n"
     ]
    }
   ],
   "source": [
    "print(f.meta_data)\n",
    "print(f.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89426fbd-bf71-4442-9c7d-b535fb30bbfc",
   "metadata": {},
   "source": [
    "Since this problem can be evaluated as a common python function, it should work directly with any optimizer. We make a basic random search to illustrate this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e4879f7e-1df2-4bf4-b5e8-7319b851c20d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "class RandomSearch:\n",
    "    'Simple random search algorithm'\n",
    "    def __init__(self, n: int, length: float = 0.0):\n",
    "        self.n: int = n\n",
    "        self.length: float = length\n",
    "        \n",
    "    def __call__(self, problem: ioh.problem.RealSingleObjective) -> None:\n",
    "        'Evaluate the problem n times with a randomly generated solution'\n",
    "        \n",
    "        for _ in range(self.n):\n",
    "            # We can use the problems bounds accessor to get information about the problem bounds\n",
    "            x = np.random.uniform(problem.bounds.lb, problem.bounds.ub)\n",
    "            self.length = np.linalg.norm(x)\n",
    "            \n",
    "            problem(x)   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a541dde-9cf3-4af4-a660-5091e70b62e0",
   "metadata": {},
   "source": [
    "If we want to run this algorithm on our problem, we can do the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a38de893-a238-4ab4-bfe9-d3590d4fed02",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<State evaluations: 10 optimum_found: false current_best: <Solution x: [0.4354764066933351, 0.9529410471322448, 0.9894592265289566, 0.7367521046229838, 0.2676026847808851] y: 0.019045607346369953>>\n"
     ]
    }
   ],
   "source": [
    "r = RandomSearch(10)\n",
    "r(f)\n",
    "print(f.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79320baa-fc59-450c-8ecd-ef00da3a0121",
   "metadata": {},
   "source": [
    "To run multiple independent runs on the same problem, we can reset the state as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "063cf170-024a-402c-b578-0865643da891",
   "metadata": {},
   "outputs": [],
   "source": [
    "f.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34a5275f-c80d-4c3c-9f0a-670a5f32499c",
   "metadata": {},
   "source": [
    "# Logging"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ffb6ea2-0ccd-4ae1-86de-eb872648b292",
   "metadata": {},
   "source": [
    "The default usage of IOHExperimenter is in generating logs of benchmarking experiments which can be analyzed in IOHAnalyzer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84a76cbe-520a-44e4-9b1a-525eb4cfe4c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "logger = ioh.logger.Analyzer(\n",
    "    root=os.getcwd(),                  # Store data in the current working directory\n",
    "    folder_name=\"my-experiment\",       # in a folder named: 'my-experiment'\n",
    "    algorithm_name=\"random-search\",    # meta-data for the algorithm used to generate these results\n",
    "    store_positions=False               # disable storing x-variables in the logged files\n",
    ")\n",
    "\n",
    "# this automatically creates a folder 'my-experiment' in the current working directory\n",
    "# if the folder already exists, it will given an additional number to make the name unique\n",
    "logger"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cb27e5f-4cca-4707-afbb-54b9987aba6e",
   "metadata": {},
   "source": [
    "We can add this logger to a problem so we can store the data when running our algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a306a7c-773e-4251-a3e3-05ac752dac8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "f.attach_logger(logger)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "416f61ff-1a16-4d29-b167-957eb657ccda",
   "metadata": {},
   "outputs": [],
   "source": [
    "r = RandomSearch(100)\n",
    "r(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95a7cf21-0a70-4bf9-a325-82f0c79b6851",
   "metadata": {},
   "source": [
    "Once finished with the run, we can close the logger to force it to write the data (happens automatically when running as a python script)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a060dec-a11a-43ca-9f76-a695f8d04bac",
   "metadata": {},
   "outputs": [],
   "source": [
    "logger.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ee1d180-1b7f-4b28-b127-3e6eeae41ab4",
   "metadata": {},
   "source": [
    "# Experiment class"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "203e6d30-4de2-497c-b9b3-2ed202dec8bb",
   "metadata": {},
   "source": [
    "In Python, we provide the Experiment class which can be used to easily run a given algorithm over a larger number of problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0886d3e-2d58-41a9-ab65-775d56aa19f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment = ioh.Experiment(\n",
    "    algorithm = RandomSearch(10), # An algorithm instance\n",
    "    fids = list(ioh.ProblemType.STAR.problems.keys())[:3],               # the id's of the problems we want to test\n",
    "    dims = [2,5],                 # the dimensions of the problems we want to test\n",
    "    iids = [1,2,3],               # the instance id's of the problems we want to test\n",
    "    reps = 1,                     # the number of runs,\n",
    "    problem_type=ioh.ProblemType.STAR, #the problem type\n",
    "    zip_output = True       \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28ff45b6-9067-4b85-9c66-226cffcff507",
   "metadata": {},
   "source": [
    "Running this experiment creates a zip-file, which can directly be processed by IOHanalyzer (https://iohanalyzer.liacs.nl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74265079-9b37-47f6-897a-2aaa6af3052d",
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33c20e04-a75a-4646-ad39-49702976f072",
   "metadata": {},
   "source": [
    "For more background and other information on using IOHexperimenter, we refer to https://iohprofiler.github.io/IOHexp/\n",
    "\n",
    "If you have any questions about the usage of the star discrepancy problems, or about IOHprofiler in general, please make an issue on https://github.com/IOHprofiler/IOHexperimenter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2a30781-e702-458a-811e-5a63d302d9b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# cleanup\n",
    "from shutil import rmtree\n",
    "import os\n",
    "import glob\n",
    "\n",
    "def clean():\n",
    "    for name in (\"my-experiment\", \"ioh_data\"):\n",
    "        for path in glob.glob(f\"{name}*\"):\n",
    "            if os.path.isfile(path):\n",
    "                os.remove(path)\n",
    "            if os.path.isdir(path):\n",
    "                rmtree(path, ignore_errors=True)\n",
    "\n",
    "def ls(p=\"./\"):\n",
    "    for obj in os.listdir(os.path.normpath(p)):\n",
    "        print(obj)\n",
    "\n",
    "def cat(f):\n",
    "    with open(os.path.normpath(f)) as h:\n",
    "        print(h.read())\n",
    "\n",
    "clean()\n",
    "\n",
    "rmtree(\"my-experiment\", ignore_errors=True)\n",
    "rmtree(\"ioh_data\", ignore_errors=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}