Feature request: run leela-zero from a remote server

[roy7]

With Sabaki you can run LZ remotely by giving it an ssh command for the go engine as mentioned here:

https://www.reddit.com/r/baduk/comments/6ofhha/how_to_run_leela_on_the_cloud/dkhej3g/

But since leelaz.exe is hard coded into the lizzie jar, that's not really an option here. I wonder how hard it'd be to support (command line option, config file, whatever) a way to run an arbitrary program + params like Sabaki so we might be able to run Lizzie on one machine but LZ itself on a different one?

[featurecat]

so is the idea to run arbitrary bots on lizzie, with custom program arguments and a gtp console?

[roy7]

I guess it could expand to that, but my primary interest was to try replacing leelaz.exe with say ssh.exe and being able to spawn a leelaz on a remote server that is much faster than my desktop.

[featurecat]

Oh I see. that's a really useful feature (for a subset of users). It's not super high on my priority list but I'd accept it as a pr. otherwise I will make it by 0.5 or 0.6

[dfannius]

I just changed leelaz to be a shell script that runs Leela Zero via ssh on a different machine. This is a piece of cake on macOS or Unix but might be more annoying on Windows. I agree that it would be nice to be able to do this with a configuration parameter rather than slyly renaming the supposed leelaz executable.

[dwt]

Just having a shell script called leelaz that then calls out via SSH is also what I do. For me it looks something like this (to be run in Sabaki). I'm still investigating how to get this to work with Lizzie.

#!/usr/bin/env bash

networks_dir=/home/dwt/leela-zero/archived-weights/weights
host=fast.nvidia.card.de
newest_network=$(ssh -n $host ls -tr $networks_dir/|tail -n 1)

leela_zero=/home/dwt/leela-zero/repo/src/leelaz
exec ssh $host $leela_zero --gtp --weights $networks_dir/$newest_network $*

I think it would be great to document this approach for Linux / Mac users that have access to a beefy server.

[dwt]

This is what I'm currently using to run leelaz via ssh on another machine, please feel free to adapt it for the documentation. Some caveats: as I can't use the local weights file, I am currently ignoring all arguments, and replicate them by hand.

% cat leelaz 
#!/usr/bin/env bash

lizzie_options="--lagbuffer 0"

networks_dir=/home/dwt/leela-zero/archived-weights/weights
host=avd
newest_network=$(ssh -n $host ls -tr $networks_dir/|tail -n 1)

leela_zero=/home/dwt/leela-zero/repo-experiments/src/leelaz
exec ssh $host $leela_zero --gtp --weights $networks_dir/$newest_network $lizzie_options

[featurecat]

wow thanks for posting these, I can use them ☺️ I'll add this thread to the documentation so others can find this more easily

[featurecat]

Since the latest version of Leelaz.java uses a more robust process builder algorithm, I think I might just allow the configuration file to read something like:

leelaz_command: "./leelaz -g -b 0 w $network -t $threads"

which would obviously be replaceable with whatever command you like. But not recommended to modify for most users. Maybe we should add inline comments to the configuration file.

[dwt]

Commented configuration files are always a good idea.

I would also like to recommend to use the long command line switches as they are much easier to read and look up fo users.

[featurecat]

Maybe... I think we will actually just have an in-app options menu to set things via the UI, instead of commandline switches. Unless you have a strong argument for them.

[dwt]

Oh I was just referring to the command line switches that lizzie uses to call leelaz. So instead of ./leelaz -g -b 0 w $network -t $threads I would recommend to generate something more along the line of ./leelaz --gtp --lagbuffer 0 --weights $path_to_network --threads $number_of_threads.

Basically it just documents better and is easier to read for users wanting / needing to change that.

[featurecat]

Oh, I see what you mean. Good point!

[OlivierBlanvillain]

@dwt Would you have time to submit a submit document this in a PR? I think it might get more visibility if was an paragraph in the README as opposed to an issue.

[YiDianRen]

The following is my "engine-command" string in the config.txt:

"engine-command": "gcloud compute --project %gcp-project ssh --zone %gcp-zone %gcp-instance --command %gcp-command",
"engine-start-location": ".",
"gcp-command": "/home/YiDianRen/leelaz-016 -g -t 2 -w /home/YiDianRen/-w-elfv1",

However, Lizzie06 outputs an error message:

Did not find LeelaZero, update config.txt or download from LeelaZero homepage

Could someone help me with this problem?
@roy7 @featurecat @dwt @OlivierBlanvillain

[featurecat]

The code responsible for that error message is found here:

lizzie/src/main/java/featurecat/lizzie/analysis/Leelaz.java

Line 128 in 5df5344

// Check if engine is present

This code searches for Leela Zero in your local directory, and tries to output a helpful error message in case you are missing any files. Obviously for a remote command, this is not helpful.

Try including an additional leelaz executable and a weight file named according to the config, in your local directory. That will keep this code happy, I think, and let you run the actual command that you want to run. Let me know if this works, and I'm sorry that the solution is not obvious.

[YiDianRen]

@featurecat Thanks to your reply.
Lizzie can run the local Leelazero without any problem.
The following command, which runs LeelaZero on the Google Cloud Platform, can be executed correctly in the Terminal of OS:
gcloud compute --project %gcp-project ssh --zone %gcp-zone %gcp-instance --command %gcp-command
So, how can Lizzie connect the remote LeelaZero by the above command or other method?

[siulkilulki]

With Lizzie 0.6, @dwt method for running lizzie with leela-zero through ssh doesn't work. I had tried some values for 'engine-start-location' and 'engine-command' but they didn't worked.

Did anybody figure out how to do it with Lizzie 0.6? Lizzie 0.5 works just fine.

EDIT (@YiDianRen):
I have figured it out:
The engine command should look like this:
"engine-command": "/usr/bin/leelaz-remote --weights /usr/share/leela-zero/networks/weights.txt"
The trick is to add --weights argument to engine-command. It's not used by leelaz-remote, but Lizzie.java parses it and checks if it exists.
leelaz-remote content is like @dwt explained.

[featurecat]

Have you tried following my advice? I'm not sure if the OP tried it - include a DUMMY Leelaz executable and a dummy weight file, in your directory, as if you were going to use them. These won't actually be executed, only the engine command will be executed. I think this should work, please try it and let me know.

[dfannius]

I have been using the method that @siulkilulki and @featurecat recommend in their latest comments and it works fine for me. Ideally Lizzie would be aware of the fact that it is running Leela Zero remotely and that there doesn't need to be a weights file locally. Maybe I will try to add such a feature.

[InHill]

Lizzie is now running after 'gcloud' login successfully. Thanks to your great help, @featurecat @siulkilulki @dfannius.
There is another problem the computing in Lizzie interrupts frequently while using the remote engine. To be curious, the computing recovers automatically after serveral minutes of pause. Meanwhile, the network connection works well. Does anyone else encounter the same problem as me? How to solve it?