SimOpt is a testbed of simulation-optimization problems and solvers. Its purpose is to encourage the development and constructive comparison of simulation-optimization (SO) solvers (algorithms). We are particularly interested in the finite-time performance of solvers, rather than the asymptotic results that one often finds in related literature.
For the purposes of this project, we define simulation as a very general technique for estimating statistical measures of complex systems. A system is modeled as if the probability distributions of the underlying random variables were known. Realizations of these random variables are then drawn randomly from these distributions. Each replication gives one observation of the system response, i.e., an evaluation of the objective function or stochastic constraints. By simulating a system in this fashion for multiple replications and aggregating the responses, one can compute statistics and use them for evaluation and design.
Several papers have discussed the development of SimOpt and experiments run on the testbed:
- Eckman et al. (2024) studies feasibility metrics for stochastically constrained simulation-optimization problems in preparation for introducing related metrics in SimOpt.
- Shashaani et al. (2024) conducts a large data-farming experiment over solver factors to learn relationships between their settings and a solver's finite-time performance.
- Eckman et al. (2023) is the most up-to-date publication about SimOpt and describes the code architecture and how users can interact with the library.
- Eckman et al. (2023) introduces the design of experiments for comparing solvers; this design has been implemented in the latest Python version of SimOpt. For detailed description of the terminology used in the library, e.g., factors, macroreplications, post-processing, solvability plots, etc., see this paper.
- Eckman et al. (2019) describes in detail changes to the architecture of the MATLAB version of SimOpt and the control of random number streams.
- Dong et al. (2017) conducts an experimental comparison of several solvers in SimOpt and analyzes their relative performance.
- Pasupathy and Henderson (2011) describes an earlier interface for MATLAB implementations of problems and solvers.
- Pasupathy and Henderson (2006) explains the original motivation for the testbed.
- The
master branchcontains the source code for the latest stable release of the testbed - The
development branchcontains the latest code for the testbed, but may contain more bugs than the master branch
⚠️ MATLAB versions of this testbed are no longer supported
- The
matlab branchcontains a previous stable version of the testbed written in MATLAB
Full documentation for the source code can be found on our readthedocs page.
- Miniconda or Anaconda
- If you already have a compatible IDE (such as VS Code), we've found that Miniconda will work fine at 1/10 of the size of Anaconda. Otherwise, you may need the Spyder IDE that comes with the full Anaconda distribution.
- It is highly recommended to check the box during installation to add Python/Miniconda/Anaconda to your system PATH.
- If you know you have Python installed but are getting a
Command not founderror when trying to use Python commands, then you may need to add Python to your PATH.
- VS Code (optional)
- This is a lightweight IDE that is compatible with Miniconda.
- Git (optional)
- If you don't have Git installed, you can download the code as a zip file instead
There are two ways to download a copy of the source code onto your machine:
-
Download the code in a zip file by clicking the green
<> Codebutton above repo contents and clicking theDownload ZIPoption, then unzip the code to a folder on your computer. This does not requiregitto be installed but makes downloading updates to the repository more challenging.
-
Clone the branch you'd like to download to a folder on your computer. This requires
gitto be installed but makes downloading updates to the repository much easier.
If you do not need the source code for SimOpt, you may install the library as a Python package instead. See the Package and Basic Example sections for more details about this option.
The notebooks folder contains short, self-contained Jupyter notebooks (each runs in under a minute) that walk through the library. Each notebook is paired with a .py script via Jupytext, so you can either run the scripts as standalone programs or open the notebooks in JupyterLab or VS Code. Older demo notebooks are kept in archive/notebooks.
| Notebook | Description |
|---|---|
01_model_and_problem |
Run replications of a simulation model, evaluate a problem at a solution, and estimate stochastic constraints |
02_single_experiment |
Run one solver on one problem, inspect the results, and plot progress curves, solvability CDFs, terminal progress, and area under the curve |
03_comparing_solvers |
Benchmark several solvers on a set of problem variants with solvability profiles, difference profiles, and terminal scatter plots |
04_constrained_problems |
Compare solver variants on a problem with stochastic constraints using feasibility progress and terminal feasibility plots |
05_data_farming |
Build designs over model, solver, and problem factors, run them, and load designs from a file |
06_extending_simopt |
Use data-driven input models, run discrete solvers, and supply a known optimal solution for normalization |
07_standalone_solvers |
Describe your own problem with simopt_dsl, run one solver on it with simopt_solvers.solve, and plot its recommendation history |
After downloading the source code, you will need to configure the conda environment to run the code. This can be done by running the following command in the terminal:
setup_simopt.batcmd /c setup_simopt.batchmod +x setup_simopt.sh && ./setup_simopt.shThis script will create a new conda environment called simopt and install all necessary packages. To activate the environment, run the following command in the terminal:
conda activate simoptIf you wish to update the environment with the latest compatible packages, you can simply rerun the setup script.
To open the web interface, run simopt web (or python -m simopt web) and open http://localhost:8000.
From the web interface you can build and run problem-solver experiments, generate plots, and create data-farming designs for solvers, problems, and models.
The previous Tkinter GUI has been removed; it is preserved on the archive/tkinter branch.
The simoptlib package is available to download through the Python Packaging Index (PyPI) and can be installed from the terminal with the following command:
python -m pip install simoptlib
After installing simoptlib, the package's main modules can be imported from the Python console (or in code):
import simopt
from simopt import models, solvers, experiment_base
The following snippet of code will run 10 macroreplications of the Random Search solver ("RNDSRCH") on the Continuous Newsvendor problem ("CNTNEWS-1"):
myexperiment = simopt.experiment_base.ProblemSolver("RNDSRCH", "CNTNEWS-1")
myexperiment.run(n_macroreps=10)The results will be saved to a .pickle file in a folder called experiments/outputs. To post-process the results, by taking, for example 200 postreplications at each recommended solution, run the following:
myexperiment.post_replicate(n_postreps=200)
simopt.experiment_base.post_normalize([myexperiment], n_postreps_init_opt=200)A .txt file summarizing the progress of the solver on each macroreplication can be produced:
myexperiment.log_experiment_results()A .txt file called RNDSRCH_on_CNTNEWS-1_experiment_results.txt will be saved in a folder called experiments/logs.
One can then plot the mean progress curve of the solver (with confidence intervals) with the objective function values shown on the y-axis:
simopt.experiment_base.plot_progress_curves(
experiments=[myexperiment],
plot_type=simopt.experiment_base.PlotType.MEAN,
normalize=False,
)The Python scripts in the notebooks folder provide more guidance on how to run common experiments using the library.
One can also use the SimOpt graphical user interface by running the following from the terminal:
python -m simoptYou can contribute problems and solvers to SimOpt (or fix other coding bugs) by forking the repository and initiating pull requests in GitHub to request that your changes be integrated.
The core development team currently consists of
- David Eckman (Texas A&M University)
- Sara Shashaani (North Carolina State University)
- Shane Henderson (Cornell University)
- Cen Wang (Texas A&M University)
- Emily Liu (Cornell University)
Previous maintainer:
- William Grochocinski (North Carolina State University)
To cite this work, please use the CITATION.cff file or use the built-in citation generator:
An earlier website for SimOpt (http://www.simopt.org) was developed through work supported by the following grants:
- National Science Foundation
Recent work on the development of SimOpt has been supported by the following grants
- National Science Foundation
- Air Force Office of Scientific Research
- FA9550-12-1-0200
- FA9550-15-1-0038
- FA9550-16-1-0046
- Army Research Office
- W911NF-17-1-0094
Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation (NSF).