-
Notifications
You must be signed in to change notification settings - Fork 0
Confident Local Optimization for Noisy Black-Box Parameter Tuning
License
godmoves/CLOP
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Welcome to CLOP, a generic tool for automatic parameter optimization Author: Rémi Coulom, Université Lille 3 Home: http://remi.coulom.free.fr/CLOP/ This software is licensed under the terms of the GNU General Public License http://www.gnu.org/licenses/gpl.html Building CLOP ============= Linux ----- Required Ubuntu packages: sudo apt-get install ccache g++ swig python-dev libgsl0-dev libboost-dev libboost-thread-dev libboost-filesystem-dev libboost-date-time-dev gnuplot libqt4-dev cd programs/clop/compgcc make cd programs/clop/compqt/clop-console qmake && make cd programs/clop/compqt/clop-gui qmake && make Windows & Mac ------------- Get the Qt SDK. Project files in programs/clop/compqt/* can be opened in Qt Creator and compiled. Optimizing a program with CLOP ============================== First step: connection script ----------------------------- The first step to connect a program to CLOP consists in writing a connection script. A connection script takes parameter values as input, plays one game, and returns the game outcome as output. For more information, run "Dummy.exe" in the Windows distribution, or programs/clop/script/real/DummyScript.py in Linux. For Go-playing programs using the GTP protocol, a Python script based on gogui tools is also provided in programs/clop/script/real/GoguiScript.py. You'll have to customize this GoguiScript.py for your program. Second step: experiment description in .clop file ------------------------------------------------ Experiment settings such as the list of processors and parameter ranges are defined in a separate file with a .clop extension. More details can be found in programs/clop/script/real/DummyExperiment.clop The experiment can be run by opening such a .clop file with the clop program. Third step: run the .clop file ----------------------------- You can either run it with the clop-gui program, or from the command line with "clop-console c <myscript.clop". Running the experiment will produce a log file and a data file. You can find error message and other information in the log. Every game result is written to the data file. If you run the same experiment again, clop will find the previous data file, and continue the experiment where it stopped. Tip: choosing parameter ranges ------------------------------ In general, it is a good idea to use intervals that are very wide. CLOP should be clever enough to focus on good values by itself. One problem is that if you have many variables, and very wide intervals, CLOP may have to explore uniformly at random for a long time before it finds a victory. In that case, you can prime the experiment by using very narrow intervals around reasonable values, so that a few wins are collected. Play a few games in this setting, then continue the experiment with wide intervals. Another problem is that CLOP only performs local optimization, and may miss a better optimum. If you know better parameters, you can also use the same mechanism to start CLOP in a small area, and widen it later. Choosing parameter type (ie, GammaParameters vs LinearParameter) ---------------------------------------------------------------- The more quadratic the function to be optimized, the better clop works. So you may wish to perform a non-linear transformation of parameters to make the function more quadratic. LinearParameter will scale the parameter range linearly to normalize it between -1 and +1. GammaParameter will take the logarithm of the parameter before scaling it. GammaParameter must take positive values only. If a parameter cannot take negative values, and multiplying it by a factor has a similar effect as dividing it by the same factor, then GammaParameter should be used instead of LinearParameter. Interpreting statistics: clop does not estimate strength accurately ------------------------------------------------------------------- Win rates displayed in clop-gui are biased. The win rate over samples with w(x)=1 ("Central" column) is often too optimistic. The win rate over all samples ("All" column) is pessimistic. clop cannot estimate accurately how strong the program is at the maximum. Implementing your own optimization algorithm ============================================ If you wish to implement your own optimization algorithm, the base classes you need to implement are CSamplingPolicy and CMaxEstimator. You will also probably have to create a CObserver in order to collect the outcome of games. You can look at CBAST.{h,cpp} for an example. This class implements all 3 at the same time. CLOP is designed that way in order to allow asynchronous results. When testing on a cluster, next game may start before we know the result of the previous game. If you only want to run games strictly in sequence, you can assume that OnOutcome() is called right after NextSample(), for the same sample. With this assumption, you can write simpler code. This assumption works with artificial problems, but it does not work with real experiments distributed over more than one processor. In order to compile with make, you should add your classes to programs/clop/compgcc/clop.mk. You'll probably want to put your classes in a different directory, so you'll have to add to -I and vpath %cpp. Then, if you wish to access your classes with a python script, you must add them to programs/clop/swig/clop_swig.i. Just add the relevant #include and %include by analogy with the other ones. If some of your classes keep references to objects created in python, you should tell python with %pythonappend, so that python's memory manager does not free them too early. You don't have to #include/%include classes that won't be used from python. Those classes should be in clop.mk only. That's all ========== So far, I wrote very little documentation. Take a look at the scripts, the code, and ask questions. I'll write more documentation as necessary.
About
Confident Local Optimization for Noisy Black-Box Parameter Tuning
Resources
License
Stars
Watchers
Forks
Releases
No releases published