Skip to content

Applying OEDA to CrowdNav

Jump to bottom
Ali Naci Uysal edited this page Jul 5, 2018 · 15 revisions

This page shows how one can use OEDA to optimize CrowdNav, a complex traffic navigation scenario.

The results of this evaluation have been published on a separate research paper--for more details you can either read the paper or check directly the online Appendix accompanying the paper.

In the rest of this page, we will explain how to use OEDA on performing experiments for optimizing CrowdNav.

Preliminaries

Step-by-step Instructions

  • Navigate to Target Systems section and click 'Create Target System' button

  • Select CrowdNav from pop-up

  • Add at least one data type and changeable variable
  • Click 'Use Configuration' button
  • Changeable variables are explained here [1]

  • Provide a unique target system name
  • Examine retrieved information and change if necessary, e.g. a different port for data provider(s)
  • Change optimization criteria, min, max and default values of changeable variables located in right side of the page if necessary
  • Select one of the data providers as "Primary Data Provider", precedence will be given to this type of data provider while retrieving data from different providers during experimentation
  • Click 'Save Target System' button

  • Navigate to Experiments section in left side
  • Click 'Create Experiment' button

  • Select CrowdNav as target system
  • Provide an experiment name
  • Provide one data type to be considered for experimentation,
  • Provide aggregation function for this data type, which will be used to calculate overall result of an experiment stage. You can see different aggregation functions supported by OEDA via dropdown list

  • Scroll down in the same page, default values required for different steps have already been populated.
  • These values can be changed according to logic of respective steps, e.g. you cannot provide a single variable and single value for Factorial Analysis of Variance (ANOVA) step. An error will pop-up for erroneous values and cases.
  • Provide values for alpha value and maximum number of significant interactions for ANOVA. Latter one will be used to select significant interactions that can be obtained by ANOVA.
  • Select library to be used for Bayesian Optimization step. OEDA supports Scikit Optimizer and mlrMBO. To use latter one, make sure R server was set-up and running in the background, see this guide. Experienced users can also change Acquisiton Methods of respective libraries.
  • Provide number of iterations to be performed. If you provide a small number that is not enough for the optimization library, OEDA will try to find the smallest number depending on the selected configuration according to ANOVA result and restrictions of optimization library.
  • Provide number of samples to be collected for each configuration of optimization process
  • Provide alpha, minimum effect size, and number of samples to be collected for t-test. See this page to determine proper effect size.
  • Click 'Save Experiment' button in top-right of this page

  • You will be redirected to this page where you can see current & already-finished experiments
  • Select newly-created experiment to see its process

  • In this page, you can see current status of experiment, its steps, scatter plots and histograms of respective stages. These are updated in real-time.

  • After some time new step(s) and data points will be retrieved and you will be able to see ANOVA result by clicking 'Show ANOVA Details' button.
  • If ANOVA cannot find significant interactions between changeable variables, then experiment will be terminated.
  • If ANOVA can find significant interactions, it will select the most significant interaction(s) based on values defined while creating the experiment.
  • Then it will perform optimization run(s) and you will be able to see respective results.
  • It will perform another experiment based on optimized configuration, which is the result of previous Bayesian optimization step, and default configuration of target system.

  • At the end, t-test will be performed to check statistical significance based on p-value and effect size of data points obtained during the last two experiments, where best and default configuration of target system have been used, respectively.

References

[1] Gerostathopoulos, Ilias, Christian Prehofer, and Tomas Bures. "Adapting a System with Noisy Outputs with Statistical Guarantees." Proc. of SEAMS 2018, to appear (2018)

Clone this wiki locally