Fix overflow errors uncovered by Coverage-guided Fuzzing. #104

c-bata · 2021-04-10T16:24:23Z

github-actions · 2021-04-10T16:26:30Z

Tuning LightGBM for Kaggle Toxic Challenge (Surrogate)

Kurobako Version: 0.2.9
Number of Solvers: 2
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
cmaes	0	0
ws-cmaes	1	1

Individual Results

(1) Problem: wscmaes-toxic-target with warm starting

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	ws-cmaes (study)	0.017000 +- 0.000126	0.511 +- 0.007	0.717 +- 0.147
2	cmaes (study)	0.018301 +- 0.000472	0.571 +- 0.012	0.229 +- 0.068

Solvers

ID: 1388a6de65200d045a60b5a1d1913437b57ea1d2a9eb6e07656e13729503bdee

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: ef6eefd4e69f619390fab64533530b7badd4391fc6709167a0d7144c94df45d4

recipe:

{
  "name": "ws-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "ws-cmaes",
      "--warm-starting-trials",
      "100"
    ]
  }
}

specification:

{
  "name": "ws-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 4b8c2bfbea634e8b32601e81057749fa9185971643acf6d1cf36906867e163ce

recipe:

{
  "warm_starting": {
    "source": {
      "surrogate": {
        "model": "./toxic-lightgbm-warm-start/wscmaes-toxic-source/"
      }
    },
    "target": {
      "surrogate": {
        "model": "./toxic-lightgbm-warm-start/wscmaes-toxic-target/"
      }
    }
  }
}

specification:

{
  "name": "wscmaes-toxic-target with warm starting",
  "attrs": {},
  "params_domain": [
    {
      "name": "bagging_fraction",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.1,
        "high": 0.9
      },
      "distribution": "UNIFORM"
    },
    {
      "name": "feature_fraction",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.1,
        "high": 0.9
      },
      "distribution": "UNIFORM"
    },
    {
      "name": "lambda_l1",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.1,
        "high": 10.0
      },
      "distribution": "LOG_UNIFORM"
    },
    {
      "name": "lambda_l2",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.1,
        "high": 10.0
      },
      "distribution": "LOG_UNIFORM"
    },
    {
      "name": "learning_rate",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.001,
        "high": 1.0
      },
      "distribution": "LOG_UNIFORM"
    },
    {
      "name": "num_leaves",
      "range": {
        "type": "DISCRETE",
        "low": 8,
        "high": 129
      },
      "distribution": "UNIFORM"
    }
  ],
  "values_domain": [
    {
      "name": "value",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM"
    }
  ],
  "steps": [
    0,
    1
  ]
}

Studies

ID: 69024f37aa0af2fe15a01e67fa10ab7ecff875a5ffa8e75e2bea7f6fdb1e32bb

problem: wscmaes-toxic-target with warm starting
solver: cmaes
budget: 30
repeats: 12
concurrency: 1

ID: ccbbaa37867579491ed3bb37921689f8e429cda0d133b064397e28a0e5c0ac52

problem: wscmaes-toxic-target with warm starting
solver: ws-cmaes
budget: 30
repeats: 12
concurrency: 1

github-actions · 2021-04-10T16:26:52Z

Benchmark of Six-Hump Camel function

Kurobako Version: 0.2.9
Number of Solvers: 4
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Random	0	1
cmaes	0	1
pycma	0	1
sep-cmaes	0	1

Individual Results

(1) Problem: Six-Hump Camel Function

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	cmaes (study)	-1.031628 +- 0.000001	17707.279 +- 33403.962	1.000 +- 0.028
1	Random (study)	-0.246249 +- 0.411201	26043.434 +- 41591.883	0.000 +- 0.000
1	pycma (study)	-1.031628 +- 0.000000	17680.699 +- 34535.049	32.441 +- 6.517
1	sep-cmaes (study)	-0.953502 +- 0.069965	17922.753 +- 33305.250	1.247 +- 0.106

Solvers

ID: a939db22692860645c92c8d357743db4cecbd97108212d5bc1d04b594aa830a9

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.2.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 1388a6de65200d045a60b5a1d1913437b57ea1d2a9eb6e07656e13729503bdee

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: e826da9b1a6e098f9b9d3ab4aeb3ed55546b179e8a3cf7bc59493c7a37cc15d7

recipe:

{
  "name": "pycma",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "pycma"
    ]
  }
}

specification:

{
  "name": "pycma",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: f93f4880bcd7c97253921a95feff5726f9224363eae3684d639511f4a77a436e

recipe:

{
  "name": "sep-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "sep-cmaes"
    ]
  }
}

specification:

{
  "name": "sep-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 2c63bdd44db7d921185285d58afb2eae15d3eb87414c3923a59bde0c615175de

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_six_hump_camel.py"
    ]
  }
}

specification:

{
  "name": "Six-Hump Camel Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM"
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM"
    }
  ],
  "values_domain": [
    {
      "name": "Six-Hump Camel",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM"
    }
  ],
  "steps": 1
}

Studies

ID: 937df77d401862d742f5cfd9ae768c94307c01a1898d4665f8fcc55b00fd958f

problem: Six-Hump Camel Function
solver: Random
budget: 300
repeats: 5
concurrency: 1

ID: 221cde865b208c8d7e515b243219fd596396c4c4167d8b0ff8e97d9baaa7183f

problem: Six-Hump Camel Function
solver: cmaes
budget: 300
repeats: 5
concurrency: 1

ID: 5f6f4881d6c431d88acd8aab8e11cc8e7591e25b97d13a24512fde0437dff095

problem: Six-Hump Camel Function
solver: pycma
budget: 300
repeats: 5
concurrency: 1

ID: 9210a1c5d00ae58fbde9c72038de4359f818639d62c3f18eff07d8b011504e34

problem: Six-Hump Camel Function
solver: sep-cmaes
budget: 300
repeats: 5
concurrency: 1

github-actions · 2021-04-10T16:29:22Z

Benchmark of Rastrigin function

Kurobako Version: 0.2.9
Number of Solvers: 5
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Random	0	0
cmaes	0	1
ipop-cmaes	1	1
ipop-sep-cmaes	1	1
sep-cmaes	1	1

Individual Results

(1) Problem: Rastrigin (dim=2)

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	cmaes (study)	1.282304 +- 1.061057	3658.082 +- 2517.779	17.495 +- 0.358
1	ipop-cmaes (study)	0.188609 +- 0.377931	2467.974 +- 994.635	16.649 +- 0.691
1	ipop-sep-cmaes (study)	0.410046 +- 0.280275	2153.686 +- 883.489	30.318 +- 3.254
1	sep-cmaes (study)	0.410046 +- 0.280275	2153.686 +- 883.489	28.859 +- 3.424
4	Random (study)	1.502151 +- 0.428810	6051.881 +- 1391.946	0.003 +- 0.000

Solvers

ID: a939db22692860645c92c8d357743db4cecbd97108212d5bc1d04b594aa830a9

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.2.1"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 1388a6de65200d045a60b5a1d1913437b57ea1d2a9eb6e07656e13729503bdee

recipe:

{
  "name": "cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: 5a57a82da232a4ce528dd9a9b6511d01feb022bd94e873b585213e78e61d4bc0

recipe:

{
  "name": "ipop-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "ipop-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: abc48ac81d021098a930e7f7cb2dcc3d8b67a846c88ff1f991f90f881bd34e51

recipe:

{
  "name": "ipop-sep-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "ipop-sep-cmaes"
    ]
  }
}

specification:

{
  "name": "ipop-sep-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

ID: f93f4880bcd7c97253921a95feff5726f9224363eae3684d639511f4a77a436e

recipe:

{
  "name": "sep-cmaes",
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/optuna_solver.py",
      "sep-cmaes"
    ]
  }
}

specification:

{
  "name": "sep-cmaes",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.8.0.dev0, kurobako-py=0.1.12"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 313e939a7e591d92e5be006977daa358d2be4730efd89f70732984f4edb495e4

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/cmaes/cmaes/benchmark/problem_rastrigin.py",
      "2"
    ]
  }
}

specification:

{
  "name": "Rastrigin (dim=2)",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM"
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM"
    }
  ],
  "values_domain": [
    {
      "name": "Rastrigin",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM"
    }
  ],
  "steps": 1
}

Studies

ID: c785cfe8e5f2ad8fd010b3efeead86ae7378974fbc42ce7ceae0688fe486c813

problem: Rastrigin (dim=2)
solver: Random
budget: 2500
repeats: 10
concurrency: 1

ID: c07d8181cf55ecd92131bbd9241cf925a045b28d861fc6a3ae9b45314f732a03

problem: Rastrigin (dim=2)
solver: cmaes
budget: 2500
repeats: 10
concurrency: 1

ID: 9ae8d021c54d33ef43b0e38f98ac3a38333b1415bb6f0ac030aadc2c595639c5

problem: Rastrigin (dim=2)
solver: ipop-cmaes
budget: 2500
repeats: 10
concurrency: 1

ID: d6ab63ab5f33e13302f45ca28ee9294be403819cbbe3672f243cb8dc5b2a43dc

problem: Rastrigin (dim=2)
solver: ipop-sep-cmaes
budget: 2500
repeats: 10
concurrency: 1

ID: 006e847a4161a177605eb5069a7eb6a50778b87e2a5bf932590ecbf01ff3e6f3

problem: Rastrigin (dim=2)
solver: sep-cmaes
budget: 2500
repeats: 10
concurrency: 1

c-bata added 3 commits March 9, 2021 10:44

Add hypothesis testing

b112c0c

Add assertion for too large elements of mean vector

3c1dd0d

Add assertions to avoid overflow errors

4c7b632

Update requirements

f0a7b18

Fix a flake8 error

bab8df2

c-bata merged commit 6f753f3 into main Apr 10, 2021

c-bata deleted the hypothesis branch April 10, 2021 16:35

c-bata mentioned this pull request Apr 10, 2021

Fuzzing for sep-CMA-ES #105

Merged

Fix overflow errors uncovered by Coverage-guided Fuzzing. #104

Fix overflow errors uncovered by Coverage-guided Fuzzing. #104

Conversation

c-bata commented Apr 10, 2021

github-actions bot commented Apr 10, 2021 • edited Loading

Tuning LightGBM for Kaggle Toxic Challenge (Surrogate)

Table of Contents

Overall Results

Individual Results

(1) Problem: wscmaes-toxic-target with warm starting

Solvers

ID: 1388a6de65200d045a60b5a1d1913437b57ea1d2a9eb6e07656e13729503bdee

ID: ef6eefd4e69f619390fab64533530b7badd4391fc6709167a0d7144c94df45d4

Problems

ID: 4b8c2bfbea634e8b32601e81057749fa9185971643acf6d1cf36906867e163ce

Studies

ID: 69024f37aa0af2fe15a01e67fa10ab7ecff875a5ffa8e75e2bea7f6fdb1e32bb

ID: ccbbaa37867579491ed3bb37921689f8e429cda0d133b064397e28a0e5c0ac52

github-actions bot commented Apr 10, 2021 • edited Loading

Benchmark of Six-Hump Camel function

Table of Contents

Overall Results

Individual Results

(1) Problem: Six-Hump Camel Function

Solvers

ID: a939db22692860645c92c8d357743db4cecbd97108212d5bc1d04b594aa830a9

ID: 1388a6de65200d045a60b5a1d1913437b57ea1d2a9eb6e07656e13729503bdee

ID: e826da9b1a6e098f9b9d3ab4aeb3ed55546b179e8a3cf7bc59493c7a37cc15d7

ID: f93f4880bcd7c97253921a95feff5726f9224363eae3684d639511f4a77a436e

Problems

ID: 2c63bdd44db7d921185285d58afb2eae15d3eb87414c3923a59bde0c615175de

Studies

ID: 937df77d401862d742f5cfd9ae768c94307c01a1898d4665f8fcc55b00fd958f

ID: 221cde865b208c8d7e515b243219fd596396c4c4167d8b0ff8e97d9baaa7183f

ID: 5f6f4881d6c431d88acd8aab8e11cc8e7591e25b97d13a24512fde0437dff095

ID: 9210a1c5d00ae58fbde9c72038de4359f818639d62c3f18eff07d8b011504e34

github-actions bot commented Apr 10, 2021 • edited Loading

Benchmark of Rastrigin function

Table of Contents

Overall Results

Individual Results

(1) Problem: Rastrigin (dim=2)

Solvers

ID: a939db22692860645c92c8d357743db4cecbd97108212d5bc1d04b594aa830a9

ID: 1388a6de65200d045a60b5a1d1913437b57ea1d2a9eb6e07656e13729503bdee

ID: 5a57a82da232a4ce528dd9a9b6511d01feb022bd94e873b585213e78e61d4bc0

ID: abc48ac81d021098a930e7f7cb2dcc3d8b67a846c88ff1f991f90f881bd34e51

ID: f93f4880bcd7c97253921a95feff5726f9224363eae3684d639511f4a77a436e

Problems

ID: 313e939a7e591d92e5be006977daa358d2be4730efd89f70732984f4edb495e4

Studies

ID: c785cfe8e5f2ad8fd010b3efeead86ae7378974fbc42ce7ceae0688fe486c813

ID: c07d8181cf55ecd92131bbd9241cf925a045b28d861fc6a3ae9b45314f732a03

ID: 9ae8d021c54d33ef43b0e38f98ac3a38333b1415bb6f0ac030aadc2c595639c5

ID: d6ab63ab5f33e13302f45ca28ee9294be403819cbbe3672f243cb8dc5b2a43dc

ID: 006e847a4161a177605eb5069a7eb6a50778b87e2a5bf932590ecbf01ff3e6f3

github-actions bot commented Apr 10, 2021 •

edited

Loading

github-actions bot commented Apr 10, 2021 •

edited

Loading

github-actions bot commented Apr 10, 2021 •

edited

Loading