Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor constants definition #85

Merged
merged 1 commit into from
Nov 26, 2020
Merged

Refactor constants definition #85

merged 1 commit into from
Nov 26, 2020

Conversation

c-bata
Copy link
Collaborator

@c-bata c-bata commented Nov 26, 2020

  • Define _EPS as a module global variable instead of instance variable.
  • Define _FLT_MAX instead of sys.float_info.max because 1e32 is enough.

@github-actions
Copy link

Benchmark of Six-Hump Camel function

plot curve image

  • Report ID: e84a761f4215efc947740794201be6822b6bf265f503a088459294415d9bd5fd
  • Kurobako Version: 0.2.0
  • Number of Solvers: 4
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Random 0 1
cmaes 0 1
pycma 0 1
sep-cmaes 0 1

Individual Results

(1) Problem: Six-Hump Camel Function

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 pycma (study) -1.031628 +- 0.000000 66767.817 +- 53781.414 30.886 +- 5.879
1 Random (study) -0.479527 +- 0.615616 45706.409 +- 44149.296 0.000 +- 0.000
1 cmaes (study) -1.031628 +- 0.000000 57520.176 +- 53897.363 0.909 +- 0.028
1 sep-cmaes (study) -1.031628 +- 0.000000 57511.033 +- 53896.077 0.818 +- 0.014

Solvers

ID: adba9490e43fe66d939f067852b174c90815eb5f7d736b45282d07bd324579d4

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.2.0"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: c7777d227fa50c528488b4e6ecf537645442f079d03ea21d950f781998063d41

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.3.0, kurobako-py=0.1.8"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 8dc7ee97c31472d10ffe598a54aaf9f294853af9b88bc7bdfab42caf69f32990

recipe:

{
  "name": "pycma",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "pycma"
    ]
  }
}

specification:

{
  "name": "pycma",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.3.0, kurobako-py=0.1.8"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: ccf3028ab5a2b7021af83a9dea79822a5ffab2df0dd6be4a911af2eb51a3d6d8

recipe:

{
  "name": "sep-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "sep-cmaes"
    ]
  }
}

specification:

{
  "name": "sep-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.3.0, kurobako-py=0.1.8"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

Problems

ID: 6467a2061b1fa5c028c60874a64fabe9c1b0ab7d776af0da0df6cfb4dfd02ba8

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_six_hump_camel.py"
    ]
  }
}

specification:

{
  "name": "Six-Hump Camel Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Six-Hump Camel",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: dc79d0d48060e98c17898b92f060ed68fa679cd629d89a78da642c88bf3d4091

ID: 28854888427397b293bda486b10e17e017be24578ab9319347e7dd022f92a59f

ID: 05dde2ca86149df42349d19c23f1d7d9eb1c38731f1249c925c3dca45dfcf83d

ID: 9cc7feb3dcd1bef4dfef863f26bd42cea5d7c159b679df37af54a7065be0d04e

@github-actions
Copy link

Benchmark of Rastrigin function

plot curve image

  • Report ID: 44a01462d69146af8715002452d0f492725b51243d4fbe5ffd5f5fe44dfb46d9
  • Kurobako Version: 0.2.0
  • Number of Solvers: 5
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Random 0 0
cmaes 0 0
ipop-cmaes 3 1
ipop-sep-cmaes 1 1
sep-cmaes 0 0

Individual Results

(1) Problem: Rastrigin (dim=2)

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 ipop-cmaes (study) 0.081636 +- 0.184330 1498.069 +- 808.511 21.782 +- 2.145
1 ipop-sep-cmaes (study) 0.181123 +- 0.346026 2310.735 +- 1138.421 20.306 +- 1.932
2 cmaes (study) 0.824359 +- 0.676621 2448.751 +- 1720.725 22.757 +- 2.253
2 sep-cmaes (study) 1.403603 +- 1.018936 3953.814 +- 2548.705 20.576 +- 2.681
3 Random (study) 1.392878 +- 0.512078 5576.501 +- 1551.925 0.005 +- 0.000

Solvers

ID: adba9490e43fe66d939f067852b174c90815eb5f7d736b45282d07bd324579d4

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.2.0"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: c7777d227fa50c528488b4e6ecf537645442f079d03ea21d950f781998063d41

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.3.0, kurobako-py=0.1.8"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 11fa479af683ab4e806598ccb939912151efc3b946ddc5a21f21fcc87798396c

recipe:

{
  "name": "ipop-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "ipop-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.3.0, kurobako-py=0.1.8"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: b3cc05e78949fecc693aabbd8efbde389a7b27e005c02428b15cfc054f930943

recipe:

{
  "name": "ipop-sep-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "ipop-sep-cmaes"
    ]
  }
}

specification:

{
  "name": "ipop-sep-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.3.0, kurobako-py=0.1.8"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: ccf3028ab5a2b7021af83a9dea79822a5ffab2df0dd6be4a911af2eb51a3d6d8

recipe:

{
  "name": "sep-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "sep-cmaes"
    ]
  }
}

specification:

{
  "name": "sep-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.3.0, kurobako-py=0.1.8"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

Problems

ID: ff691b25f0d05f9eb9d7bbb633075253f1a3cbbc24e9558e8ac9b8126882a326

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_rastrigin.py",
      "2"
    ]
  }
}

specification:

{
  "name": "Rastrigin (dim=2)",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rastrigin",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 5118943d4271e745a6a6193ae884ca79ec0862b890c7b5fba550476a3df6f70a

ID: dc3f1d6b8d5954d079132e03e58cca4d3ac383739bb917cba58378a3dda26418

ID: 321e2062474131d946d288ea4709fcd5ea9ffcf24d4c13c5b80c421ce126d2b7

ID: 6eb8f4ef44d78a94da999cca3a9a1ae987e2d14a7d8028fad009550afd524c15

ID: 7e2a913a75a94b59a5365fd541767357f76c9dff11b9f3588cd58d2713ef29a0

@c-bata c-bata merged commit 982fe2e into main Nov 26, 2020
@c-bata c-bata deleted the constants branch November 26, 2020 21:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@c-bata