Rename study options without breaking changes #138

c-bata · 2020-07-25T14:30:16Z

StudyOptionSet... is redundant and not consistent.

sile/kurobako-go doesn't use these options.
kubeflow/katib uses these options.

$ git grep StudyOptionSet ./pkg/suggestion/
pkg/suggestion/v1alpha3/goptuna/converter.go:   studyOpts = append(studyOpts, goptuna.StudyOptionSetDirection(direction))
pkg/suggestion/v1beta1/goptuna/converter.go:    studyOpts = append(studyOpts, goptuna.StudyOptionSetDirection(direction))

…nel with deprecation

github-actions · 2020-07-25T14:35:57Z

Benchmark result of sigopt/evalset/Ackley problem

Report ID: a1cd9cad1615247e6d930b1263037812a6ef192a4c2112656d530e884836c9b0
Kurobako Version: 0.1.10
Number of Solvers: 6
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Goptuna (BIPOP-CMA-ES)	0	1
Goptuna (CMA-ES)	0	1
Goptuna (IPOP-CMA-ES)	0	1
Goptuna (TPE)	0	1
Optuna (TPE)	0	1
Random	0	1

Individual Results

(1) Problem: sigopt/evalset/Ackley(dim=10)

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	Goptuna (TPE) (study)	5.116923 +- 0.561424	8145.139 +- 274.914	42.879 +- 6.544
1	Optuna (TPE) (study)	6.326946 +- 1.945140	8739.853 +- 1732.156	113.138 +- 31.210
1	Goptuna (BIPOP-CMA-ES) (study)	0.050740 +- 0.039694	3875.009 +- 339.454	1.333 +- 0.263
1	Goptuna (IPOP-CMA-ES) (study)	0.050740 +- 0.039694	3875.009 +- 339.454	1.216 +- 0.203
1	Goptuna (CMA-ES) (study)	0.050740 +- 0.039694	3875.009 +- 339.454	1.300 +- 0.278
1	Random (study)	15.476572 +- 0.916833	16117.231 +- 457.140	0.005 +- 0.004

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "bipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (BIPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (IPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "tpe"
    ]
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "tpe"
    ]
  }
}

specification:

{
  "name": "Optuna (TPE)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 93492282fbba0e83fba906eaf579809327a6e263940fb4fd42cc3abc9a155beb

recipe:

{
  "sigopt": {
    "name": "ACKLEY",
    "dim": 10
  }
}

specification:

{
  "name": "sigopt/evalset/Ackley(dim=10)",
  "attrs": {
    "github": "https://github.com/sigopt/evalset",
    "paper": "Dewancker, Ian, et al. \"A strategy for ranking optimization methods using multiple criteria.\" Workshop on Automatic Machine Learning. 2016.",
    "version": "kurobako_problems=0.1.7"
  },
  "params_domain": [
    {
      "name": "p0",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p1",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p2",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p3",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p4",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p5",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p6",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p7",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p8",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p9",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Objective Value",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: f0fa2ceb80845ca22ffa2479f45c1ff82f4870c1bf20f556b5155edd6387cda8

problem: sigopt/evalset/Ackley(dim=10)
solver: Goptuna (BIPOP-CMA-ES)
budget: 1000
repeats: 5
concurrency: 1

ID: 1d5f43ce7d06f0ecc20f649b9a660458e0e55148ceb2ff28705e0a123f802638

problem: sigopt/evalset/Ackley(dim=10)
solver: Goptuna (CMA-ES)
budget: 1000
repeats: 5
concurrency: 1

ID: 6f8c38733e62ecf7d89a3669fe8739e12eee2d4828c035728ff6e509aac633af

problem: sigopt/evalset/Ackley(dim=10)
solver: Goptuna (IPOP-CMA-ES)
budget: 1000
repeats: 5
concurrency: 1

ID: 66298c57166677e2f5d11b0595998342167947d2100a4d34abe76b2d73bdbd0a

problem: sigopt/evalset/Ackley(dim=10)
solver: Goptuna (TPE)
budget: 1000
repeats: 5
concurrency: 1

ID: 135c5259d0a821bd89b8eb14637fe9ff1e1028162a8a083ac07acff879672620

problem: sigopt/evalset/Ackley(dim=10)
solver: Optuna (TPE)
budget: 1000
repeats: 5
concurrency: 1

ID: ac65437313c4ab7ce99a7ecb88d98d40e124c8e950a546486d06df3b17d4e0ec

problem: sigopt/evalset/Ackley(dim=10)
solver: Random
budget: 1000
repeats: 5
concurrency: 1

c-bata added 2 commits July 25, 2020 23:35

Rename StudyOptionSetDirection to StudyOptionDirection with deprecation

64e3554

Rename StudyOptionSetTrialNotifyChannel to StudyOptionTrialNotifyChan…

e714e25

…nel with deprecation

c-bata force-pushed the rename-study-options branch from db64937 to e714e25 Compare July 25, 2020 14:35

c-bata merged commit d2d7798 into master Jul 25, 2020

c-bata deleted the rename-study-options branch July 25, 2020 14:39

c-bata changed the title ~~Rename study options with backward compatibility.~~ Rename study options without breaking changes Jul 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rename study options without breaking changes #138

Rename study options without breaking changes #138

c-bata commented Jul 25, 2020 •

edited

github-actions bot commented Jul 25, 2020 •

edited

Table of Contents

Overall Results

Individual Results

(1) Problem: sigopt/evalset/Ackley(dim=10)

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

Problems

ID: 93492282fbba0e83fba906eaf579809327a6e263940fb4fd42cc3abc9a155beb

Studies

ID: f0fa2ceb80845ca22ffa2479f45c1ff82f4870c1bf20f556b5155edd6387cda8

ID: 1d5f43ce7d06f0ecc20f649b9a660458e0e55148ceb2ff28705e0a123f802638

ID: 6f8c38733e62ecf7d89a3669fe8739e12eee2d4828c035728ff6e509aac633af

ID: 66298c57166677e2f5d11b0595998342167947d2100a4d34abe76b2d73bdbd0a

ID: 135c5259d0a821bd89b8eb14637fe9ff1e1028162a8a083ac07acff879672620

ID: ac65437313c4ab7ce99a7ecb88d98d40e124c8e950a546486d06df3b17d4e0ec

Rename study options without breaking changes #138

Rename study options without breaking changes #138

Conversation

c-bata commented Jul 25, 2020 • edited

github-actions bot commented Jul 25, 2020 • edited

Benchmark result of sigopt/evalset/Ackley problem

Table of Contents

Overall Results

Individual Results

(1) Problem: sigopt/evalset/Ackley(dim=10)

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

Problems

ID: 93492282fbba0e83fba906eaf579809327a6e263940fb4fd42cc3abc9a155beb

Studies

ID: f0fa2ceb80845ca22ffa2479f45c1ff82f4870c1bf20f556b5155edd6387cda8

ID: 1d5f43ce7d06f0ecc20f649b9a660458e0e55148ceb2ff28705e0a123f802638

ID: 6f8c38733e62ecf7d89a3669fe8739e12eee2d4828c035728ff6e509aac633af

ID: 66298c57166677e2f5d11b0595998342167947d2100a4d34abe76b2d73bdbd0a

ID: 135c5259d0a821bd89b8eb14637fe9ff1e1028162a8a083ac07acff879672620

ID: ac65437313c4ab7ce99a7ecb88d98d40e124c8e950a546486d06df3b17d4e0ec

c-bata commented Jul 25, 2020 •

edited

github-actions bot commented Jul 25, 2020 •

edited