Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Categorical parameter does not work #90

Open
Claes1981 opened this issue Sep 2, 2020 · 5 comments · May be fixed by #107
Open

Categorical parameter does not work #90

Claes1981 opened this issue Sep 2, 2020 · 5 comments · May be fixed by #107
Labels
bug Something isn't working Priority: High
Projects
0.7.0

Comments

@Claes1981
Copy link

  • Chess Tuning Tools version: 0.6.0b2
  • Python version: 3.8.3
  • Operating System: OpenSUSE Tumbleweed

Description

Experiment with categorical UCI parameter. The tuner tries to set it to a real number instead of one of specified texts.

What I Did

tune local --tuning-config Test6.json --data-path Test6.npz --logfile Test6.log --plot-path Test6 --verbose 
2020-09-02 15:28:50,540 INFO     Starting iteration 0
2020-09-02 15:28:50,541 INFO     Testing {'TimeManager': '0.38854381999831666'}
2020-09-02 15:28:50,541 DEBUG    engines.json is prepared:
[{'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine1', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}, {'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine2', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}]
2020-09-02 15:28:50,542 INFO     Start experiment
2020-09-02 15:29:29,865 INFO     Experiment finished (39.32348s elapsed).
2020-09-02 15:29:29,865 DEBUG    Raw result:
Indexing opening suite...
Started game 1 of 2 (engine1 vs engine2)
Terminating process of engine engine1(0)
Finished game 1 (engine1 vs engine2): 0-1 {White's connection stalls}
Score of engine1 vs engine2: 0 - 1 - 0  [0.000] 1
Started game 2 of 2 (engine2 vs engine1)
Terminating process of engine engine1(2)
Finished game 2 (engine2 vs engine1): 1-0 {Black's connection stalls}
Score of engine1 vs engine2: 0 - 2 - 0  [0.000] 2
Elo difference: -inf +/- nan, LOS: 7.9 %, DrawRatio: 0.0 %
Finished match


2020-09-02 15:29:30,199 INFO     Got score: 1.021090020413224 +- 2.087254826323683
2020-09-02 15:29:30,200 INFO     Updating model
2020-09-02 15:29:30,200 INFO     GP sampling finished (0.000135s)
2020-09-02 15:29:30,201 DEBUG    GP kernel: None
2020-09-02 15:29:30,241 INFO     Starting iteration 1
2020-09-02 15:29:30,241 INFO     Testing {'TimeManager': '0.7705098312484218'}
2020-09-02 15:29:30,242 DEBUG    engines.json is prepared:
[{'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine1', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}, {'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine2', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}]
2020-09-02 15:29:30,242 INFO     Start experiment
^C
Aborted!

Test6.json:

{
	"engines": [
		{
			"command": "nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden",
			"fixed_parameters": {
                                "WeightsFile": "/partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz",
                                "Backend": "cudnn-fp16",
                                "NNCacheSize": 800000,
                                "Ponder": false,
                                "MoveOverheadMs": 3000
			}
		},
		{
			"command": "nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden",
			"fixed_parameters": {
                                "WeightsFile": "/partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz",
                                "Backend": "cudnn-fp16",
                                "NNCacheSize": 800000,
                                "Ponder": false,
                                "MoveOverheadMs": 3000
			}
		}
	],
	"parameter_ranges": {
       "TimeManager": "('legacy', 'smooth')"
	},
    "engine1_st": "4",
    "engine2_st": "4",
    "rounds": 1,
    "opening_file": "empty.pgn",
    "adjudicate_draws": false,
    "adjudicate_resign": false
}
@AlexisOlson
Copy link
Contributor

Maybe it needs to be specified as categorical like here?

chess-tuning-tools/tune/io.py

Line 112 in c9931ed

# (example Categorical(["a", "b", "c"]))

@kiudee
Copy link
Owner

kiudee commented Sep 2, 2020 •
edited

Could also be that there is a bug regarding categoricals. I will check it tomorrow.
edit: But yes, try AlexisOlson’s suggestion first:

"TimeManager": "Categorical([legacy, smooth])"
or
"TimeManager": "Categorical(['legacy', 'smooth'])"

@Claes1981
Copy link
Author

Claes1981 commented Sep 3, 2020 •
edited

Thanks, tried it now,
"TimeManager": "Categorical([legacy, smooth])":

tune local --tuning-config Test6.json --data-path Test6.npz --logfile Test6.log --plot-path Test6 --verbose 
Traceback (most recent call last):
  File "/home/claes/Chess-tuning-tools/env/bin/tune", line 8, in <module>
    sys.exit(cli())
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/tune/cli.py", line 265, in local
    settings, commands, fixed_params, param_ranges = load_tuning_config(json_dict)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/tune/io.py", line 185, in load_tuning_config
    param_ranges = parse_ranges(json_dict["parameter_ranges"])
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/tune/io.py", line 136, in parse_ranges
    parsed = literal_eval(s)
  File "/usr/lib64/python3.8/ast.py", line 99, in literal_eval
    return _convert(node_or_string)
  File "/usr/lib64/python3.8/ast.py", line 82, in _convert
    return list(map(_convert, node.elts))
  File "/usr/lib64/python3.8/ast.py", line 98, in _convert
    return _convert_signed_num(node)
  File "/usr/lib64/python3.8/ast.py", line 75, in _convert_signed_num
    return _convert_num(node)
  File "/usr/lib64/python3.8/ast.py", line 66, in _convert_num
    _raise_malformed_node(node)
  File "/usr/lib64/python3.8/ast.py", line 63, in _raise_malformed_node
    raise ValueError(f'malformed node or string: {node!r}')
ValueError: malformed node or string: <_ast.Name object at 0x7fd09a625280>

"TimeManager": "(['legacy', 'smooth'])":

tune local --tuning-config Test6.json --data-path Test6.npz --logfile Test6.log --plot-path Test6 --verbose 
2020-09-03 05:51:07,431 INFO     Starting iteration 0
2020-09-03 05:51:07,435 INFO     Testing {'TimeManager': '0.38854381999831666'}
2020-09-03 05:51:07,436 DEBUG    engines.json is prepared:
[{'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine1', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}, {'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine2', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}]
2020-09-03 05:51:07,436 INFO     Start experiment
^C
Aborted!

Before this simple test configuration with just one tuning parameter, I also tried tuning some categorical parameters (two possible text values) and some number parameters together in a multiparameter tune. Then it seemed to work and set the categorical parameters correctly. However, if I paused it, and then resumed, it seemed like it wanted to prune some data (or "reduce_ranges") and stopped with an error doing so.

Just tried to do a similar test:
Test7.json:

	"engines": [
		{
			"command": "nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden",
			"fixed_parameters": {
                                "WeightsFile": "/partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz",
                                "Backend": "cudnn-fp16",
                                "NNCacheSize": 800000,
                                "Ponder": false
			}
		},
		{
			"command": "nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden",
			"fixed_parameters": {
                                "WeightsFile": "/partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz",
                                "Backend": "cudnn-fp16",
                                "NNCacheSize": 800000,
                                "Ponder": false,
                                "MoveOverheadMs": 3000
			}
		}
	],
	"parameter_ranges": {
	   "MoveOverheadMs": "Integer(0, 3000)",
       "TimeManager": "(['legacy', 'smooth','alphazero'])"
	},
    "engine1_st": "4",
    "engine2_st": "4",
    "rounds": 1,
    "opening_file": "empty.pgn",
    "adjudicate_draws": false,
    "adjudicate_resign": false
} 

tune local --tuning-config Test7.json --data-path Test7.npz --logfile Test7.log --plot-path Test7 --verbose 
2020-09-03 20:42:25,791 INFO     Starting iteration 0
2020-09-03 20:42:25,791 INFO     Testing {'MoveOverheadMs': 1734, 'TimeManager': 'legacy'}
2020-09-03 20:42:25,791 DEBUG    engines.json is prepared:
[{'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine1', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False'], 'protocol': 'uci'}, {'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine2', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}]
2020-09-03 20:42:25,792 INFO     Start experiment
2020-09-03 20:44:13,814 INFO     Experiment finished (108.022283s elapsed).
2020-09-03 20:44:13,814 DEBUG    Raw result:
Indexing opening suite...
Started game 1 of 2 (engine1 vs engine2)
Finished game 1 (engine1 vs engine2): 1/2-1/2 {Draw by 3-fold repetition}
Score of engine1 vs engine2: 0 - 0 - 1  [0.500] 1
Started game 2 of 2 (engine2 vs engine1)
Finished game 2 (engine2 vs engine1): 1/2-1/2 {Draw by 3-fold repetition}
Score of engine1 vs engine2: 0 - 0 - 2  [0.500] 2
Elo difference: 0.0 +/- 0.0, LOS: nan %, DrawRatio: 100.0 %
Finished match


2020-09-03 20:44:14,142 INFO     Got score: 0.0 +- 0.8734251875761166
2020-09-03 20:44:14,143 INFO     Updating model
2020-09-03 20:44:14,143 INFO     GP sampling finished (0.000138s)
2020-09-03 20:44:14,144 DEBUG    GP kernel: None
2020-09-03 20:44:14,187 INFO     Starting iteration 1
2020-09-03 20:44:14,187 INFO     Testing {'MoveOverheadMs': 2469, 'TimeManager': 'legacy'}
2020-09-03 20:44:14,188 DEBUG    engines.json is prepared:
[{'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine1', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False'], 'protocol': 'uci'}, {'command': 'nice --5 /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/Lc0-0.26.2/lc0-0262-cuda110221-cudnn765 --show-hidden', 'name': 'engine2', 'initStrings': ['setoption name WeightsFile value /partitions/Sandisk/xfs/media/data/chess/engines/Lc0/weights/Sergio-V/256x20-t40-1541.pb.gz', 'setoption name Backend value cudnn-fp16', 'setoption name NNCacheSize value 800000', 'setoption name Ponder value False', 'setoption name MoveOverheadMs value 3000'], 'protocol': 'uci'}]
2020-09-03 20:44:14,188 INFO     Start experiment
^C
Aborted!
(env) claes@Lenovo-OpenSUSE:~/Chess-tuning-tools> tune local --tuning-config Test7.json --data-path Test7.npz --logfile Test7.log --plot-path Test7 --verbose 
Traceback (most recent call last):
  File "/home/claes/Chess-tuning-tools/env/bin/tune", line 8, in <module>
    sys.exit(cli())
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/tune/cli.py", line 320, in local
    reduction_needed, X_reduced, y_reduced, noise_reduced = reduce_ranges(
  File "/home/claes/Chess-tuning-tools/env/lib64/python3.8/site-packages/tune/local.py", line 363, in reduce_ranges
    if value < lb or value > ub:
TypeError: '<' not supported between instances of 'str' and 'int'

@kiudee kiudee added the bug Something isn't working label Sep 4, 2020
@kiudee
Copy link
Owner

kiudee commented Sep 4, 2020

That message is helpful and it appears that reduce_ranges does not properly account for categorical variables.

@Claes1981
Copy link
Author

Claes1981 commented Sep 4, 2020 •
edited

Should it even process something called "reduce_ranges"? I did not change any settings or ranges before resuming.
(I do not know how the program works though.)

@kiudee kiudee added this to the v0.6.0 milestone Sep 15, 2020
@kiudee kiudee added this to To do in 0.6.0 Sep 20, 2020
@kiudee kiudee added this to To do in 0.7.0 via automation Sep 20, 2020
@kiudee kiudee removed this from To do in 0.6.0 Sep 20, 2020
@kiudee kiudee removed this from the v0.6.0 milestone Sep 20, 2020
@kiudee kiudee moved this from To do to In progress in 0.7.0 Sep 27, 2020
@kiudee kiudee linked a pull request Oct 17, 2020 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working Priority: High
Projects
No open projects
0.7.0
  
In progress
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Switch data format to hdf5 kiudee/chess-tuning-tools
3 participants
@kiudee @AlexisOlson @Claes1981