[remote buildbot-worker setup] How can I setup a Buildbot remote worker (VM)

[sananand007]

Hi team,

My goal is to connect a remove virtual machine as a worker to the Buildbot master. The only options that are presented for the buildbot-worker as per the documents are as below

generally the way I communicate with the Virtual machine is through ssh , I am not understanding how to communicate with a remote host with the command below and what kind of settings would I need to configure to do that. Is there a wiki or a document providing some information about this ??

~ % buildbot-worker create-worker --help
Usage:    buildbot-worker create-worker [options] <basedir> <master> <name> <passwd>

[phillxnet]

@sananand007 I am not a buildbot developer, and only have trivial documentation commits. But I think the missing link here is that you have to install the buildbot worker software on the remove virtual machine. It is then initialised to communicate (connect to) the master via the credentials you give the create-worker command you indicate. The master config must likewise have knowledge of this worker before it will allow it's connection into that master. The master then has access to that remove virtual machine to the extend of the user you run the worker under. I start my KVM virtual machine based workers via systemd using the scripts available in the buildbot-contrib repo here:

https://github.com/buildbot/buildbot-contrib/tree/master/worker/contrib/systemd

There are I believe KVM worker instantiation capabilities now within buildbot but this is not what you appear to be asking.

So in short the remove VM hosts a buildbot worker that is then, once authenticated with the master, client to the master.

Hope that helps.

[sananand007]

@phillxnet First of all thanks again for your time to reply on this.

Before I start my explaination, I think the buildBot worker configuration document assumes that the worker can already communicate with the master and vice versa, in some way like a LAN. All the ways of configuration is mostly defined to add secrets and TLS (auth( during communication and nothing more. It feels as if, the worker hostname and port are already visible to the master and buildbot ducumentation does not tell anything about how to make that possible for different cases such as

• Bare Metal + VM
• Bare Metal + Bare Metal
• VM + VM ( like aws ec2 + aws ec2 )
• other cases

I already have installed all the needed Command Line tools required in the worker node, but unless and untill the worker node is already visible to the master node and vice versa, the secret setup and other steps and not meaningful.

Is there any possible documentation for setting up something of a communication between

• Bare Metal + VM/s ?

Thanks again for the reply

[phillxnet]

@sananand007 Hello again.
Re:

I think the buildBot worker configuration document assumes that the worker can already communicate with the master and vice versa, in some way like a LAN.

I think network setup is clearly beyond the remit of the buildbot documentation, in and of itself. You just need to make sure you can ping by at least the ip number. That should be enough, assuming it is static. Again more in the realm of networking. A prerequisite for setting up services that use networking.

The following doc does look to cover the Buildbot side of things. I know they are also open to pull requests for documentation improvement because, as mentioned, I've submitted a trivial one myself.
https://docs.buildbot.net/latest/manual/installation/worker.html

Regarding the port used by the master, this is also configurable. See:
https://docs.buildbot.net/latest/manual/configuration/global.html#setting-the-pb-port-for-workers
The first paragraph reads:

The buildmaster will listen on a TCP port of your choosing for connections from workers. It can also use this port for connections from remote Change Sources, status clients, and debug tools. This port should be visible to the outside world, and you’ll need to tell your worker admins about your choice.

So in short. you need to establish a prior unused port that you can route, or use the default, and make sure it is not blocked by a firewall on the master. The client then calls in to the master to setup their chit-chat back and forth.

The type of machine, OS, level of virtualisation is all irrelevant here. At least from buildbot's perspective. Reference the documentation of your hypervisor router etc for how you do the network bit. And once you can ping, you can move to establishing the port forward if that's even required. The default port for workers to call in to the master (so open for incomming tcp on master) use to be 9989. Look to your global config in the master to know what is actually setup. Or of course to setup your preferred port, wise to it's other possible uses.

So in short you need only establish an open port on the master and be able to ping the master from the worker. The buildbot config does the rest. They have their own configurable protocol to chat with using tcp.

Buildbot does have a steep on-ramp but that is not uncommon with extremely flexible client server models. It is not in their remit to then educate on network setup: i.e. VPN or port forwarding or routing etc. That would be network setup educational material.

But there could be some cross linking between these two doc sections. Maybe that could be a doc pull request in the making if you fancied.

I found the initial buildbot tuturial here: https://docs.buildbot.net/latest/tutorial/index.html extremely useful.

Just keep in mind that Buildbot doesn't care what or where it is running. It only need tcp access between it's part. And that the workers initiate the connection. You only need to setup tcp between the worker and the master at the commonly agreed port (default safest bet) and have no firewall blocking this connection and you should be good. As long as the credentials are also matched.

Start by being able to ping the master from the worker.
Then ensure master port for worker call-in is configured and known. And not blocked by a firewall.
Match worker definition/credentials when configuring in master and creating the worker.
Watch for issues / error in the master and client logs once you start the worker (assuming the master is running).

Hope that helps.

[sananand007]

@phillxnet I fished around some more and tried to set up something as an experiment to see my understanding.

What I discovered are as follows:

• I wanted to set up a DigitalOcean droplet as a worker, it is exactly similar to an Amazon EC2 instance. It's a VM that is allocated to you. its compute as a Service(CAAS) ( linux instance you can ssh to the instance )
• I wanted to have a physical Linux machine as the buildbot master, I set that up as my own personal linux machine
• I concluded, This kind of worker that I want to set Up here, is called a Latent Worker
• At the end of it, I realized this support is not existing currently in buildbot as per - http://docs.buildbot.net/current/manual/configuration/workers.html#supported-latent-workers
• Basically I believe buildbot needs its workers to be local workers which are directly connected to the buildbot master such that the ports are locally seen by the buildbot master, any kind of remote worker needs special treatment , or rather they need to be setup as latent worker which uses a separate python API for creating that worker and other functionalities that comes with maintaining that worker.

I think this somewhat concludes what I wanted to achieve.

• If I do want to use it for any kind of VM available as Compute as a service, I cannot or rather I do not have the instructions to do that, I would need to use the listed latent-worker support that is available to use, so use a EC2 (Amazon), or use a datacenter (OpenStack), or just use several Docker container spread across the same machine
• I will try to use a EC2 instance and set that up as a worker, as the purpose of this activity is to have something that is scalable and not about owning a datacenter or multiple physical machines

[phillxnet]

@sananand007 Looks like you are gaining familiarity at least with the options available.
Your may also be interested in Python's own distributed use of buildbot across the globe where volunteers and sponsors setup workers to do distributed compute to facilitate the maintenance of Python itself:
https://devguide.python.org/buildworker/
and the workers in action via their open buildbot dashboard:
https://buildbot.python.org/all/#/workers

[sananand007]

Tnx @phillxnet for the reply, I did see the devguide that you have shared which gave me some ideas about latent workers. The workers link is quite helpful, as I might ask some of the admins about the configurations that they have used.

THanks again for following up, I am setting up a doc of my steps to share as I proceed.