multiple local bots on different machines

[kbiz862]

Can i run multiple local bots on different machines with one server on different machine ?

[inferno-chromium]

Yes https://www.youtube.com/watch?v=65aywI2JcLA but that is not a supported usecase and might have scalability issues. You will on your own there. We recommend using Google Cloud for production usecases.

[kbiz862]

In this video he runs bot and server on same VM. I want to run multiple bots on different machines and all of them connected to a single server.

[inferno-chromium]

it should just work as long as you open the following ports listed here. -https://github.com/google/clusterfuzz/blob/master/src/local/butler/constants.py#L59

[kbiz862]

Hi, i tried to connect remote bots to the server. it shows the list of remote bots but when i submit the job, remote jobs seems to be not taking the job.

I followed the steps mentioned in https://www.youtube.com/watch?v=65aywI2JcLA. Using these steps i am able to upload the jobs from other machines as well(where server is not running).

Server constant.py (vm-cfz1 is my server hostname)
DEV_APPSERVER_PORT = 9000
DEV_APPSERVER_HOST = 'vm-cfz1:' + str(DEV_APPSERVER_PORT)

CRON_SERVICE_PORT = 9009
CRON_SERVICE_HOST = 'vm-cfz1:' + str(CRON_SERVICE_PORT)

DEV_APPSERVER_ADMIN_PORT = 9002

DATASTORE_EMULATOR_PORT = 9004
DATASTORE_EMULATOR_HOST = 'vm-cfz1:' + str(DATASTORE_EMULATOR_PORT)

PUBSUB_EMULATOR_PORT = 9006
PUBSUB_EMULATOR_HOST = 'localhost:' + str(PUBSUB_EMULATOR_PORT)

LOCAL_GCS_SERVER_PORT = 9008
LOCAL_GCS_SERVER_HOST = 'http://vm-cfz1:' + str(LOCAL_GCS_SERVER_PORT)

remote bot(whose hostname is vm-cfz2) constant.py

DEV_APPSERVER_PORT = 9000
DEV_APPSERVER_HOST = 'vm-cfz1:' + str(DEV_APPSERVER_PORT)

CRON_SERVICE_PORT = 9009
CRON_SERVICE_HOST = 'vm-cfz1:' + str(CRON_SERVICE_PORT)

DEV_APPSERVER_ADMIN_PORT = 9002

DATASTORE_EMULATOR_PORT = 9004
DATASTORE_EMULATOR_HOST = 'vm-cfz1:' + str(DATASTORE_EMULATOR_PORT)

PUBSUB_EMULATOR_PORT = 9006
PUBSUB_EMULATOR_HOST = 'vm-cfz1:' + str(PUBSUB_EMULATOR_PORT)

LOCAL_GCS_SERVER_PORT = 9008
LOCAL_GCS_SERVER_HOST = 'http://vm-cfz1:' + str(LOCAL_GCS_SERVER_PORT)

command to run server:

python3.7 butler.py run_server --storage-path /prj/clusterfuzz --bootstrap

command to run bot:
python3.7 butler.py run_bot --server-storage-path /prj/clusterfuzz/ --name remote_bot ./remote_bot/

Note: /prj/clusterfuzz is shared path which is accessible to both server and remote bot.

bot.log on remote bot:
run_bot - INFO - Retrying on google_cloud_utils.pubsub._execute_with_retry failed with [Errno 111] Connection refused. Retrying again.

StackTrace:
2020-12-23 16:40:55,697 - run_bot - ERROR - Error occurred while working on task.
Traceback (most recent call last):
File "src/python/bot/startup/run_bot.py", line 78, in task_loop
task = tasks.get_task()
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/python/base/tasks.py", line 202, in get_task
task = get_regular_task()
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/python/base/tasks.py", line 165, in get_regular_task
pubsub.subscription_name(application_id, queue), max_messages=1)
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/python/google_cloud_utils/pubsub.py", line 157, in pull_from_subscription
response = self._execute_with_retry(request)
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/python/base/retry.py", line 88, in _wrapper
result = func(*args, **kwargs)
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/python/google_cloud_utils/pubsub.py", line 107, in _execute_with_retry
return request.execute()
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/third_party/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/third_party/googleapiclient/http.py", line 901, in execute
headers=self.headers,
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/third_party/googleapiclient/http.py", line 204, in _retry_request
raise exception
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/third_party/googleapiclient/http.py", line 177, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/third_party/google_auth_httplib2.py", line 201, in request
uri, method, body=body, headers=request_headers, **kwargs)
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/third_party/httplib2/init.py", line 1994, in request
cachekey,
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/third_party/httplib2/init.py", line 1651, in _request
conn, request_uri, method, body, headers
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/third_party/httplib2/init.py", line 1557, in _conn_request
conn.connect()
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/third_party/httplib2/init.py", line 1236, in connect
raise socket_err
File "/local/mnt/workspace/clusterfuzz/remote_bot/clusterfuzz/src/third_party/httplib2/init.py", line 1210, in connect
self.sock.connect((self.host, self.port) + sa[2:])
ConnectionRefusedError: [Errno 111] Connection refused

Also i tried to change PUBSUB_EMULATOR_HOST from 'localhost:' + str(PUBSUB_EMULATOR_PORT) to 'vm-cfz1:' + str(PUBSUB_EMULATOR_PORT)

i get below error:

Traceback (most recent call last):
File "butler.py", line 316, in
main()
File "butler.py", line 302, in main
command.execute(args)
File "src/local/butler/run_server.py", line 155, in execute
test_utils.setup_pubsub(constants.TEST_APP_ID)
File "/local/mnt/workspace/clusterfuzz/src/python/tests/test_libs/test_utils.py", line 293, in setup_pubsub
create_pubsub_topic(client, project, queue['name'])
File "/local/mnt/workspace/clusterfuzz/src/python/tests/test_libs/test_utils.py", line 266, in create_pubsub_topic
if client.get_topic(full_name):
File "/local/mnt/workspace/clusterfuzz/src/python/google_cloud_utils/pubsub.py", line 197, in get_topic
response = self._execute_with_retry(request)
File "/local/mnt/workspace/clusterfuzz/src/python/base/retry.py", line 88, in _wrapper
result = func(*args, **kwargs)
File "/local/mnt/workspace/clusterfuzz/src/python/google_cloud_utils/pubsub.py", line 107, in _execute_with_retry
return request.execute()
File "/local/mnt/workspace/clusterfuzz/src/third_party/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/local/mnt/workspace/clusterfuzz/src/third_party/googleapiclient/http.py", line 907, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 503 when requesting http://vm-cfz1:9006/v1/projects/test-clusterfuzz/topics/jobs-linux?alt=json returned "io exception">

Thanks in advance.