Add crossbar.io configuration example

[ivanov17]

I'm trying to set up Buildbot multimaster and use crossbar.io as a WAMP router, as recommended. But I have some troubles with configuring crossbar.io to use with Buildbot.

I see that crossbar.io is a very customizable, so it has hundreds of options which is a bit confusing to me. I found the default node configuration, but it doesn't seem to be exactly what I need.

For example, it's not entirely clear if "raw sockets" can be used, or if buildbot only supports websockets for message queuing. Also, I don't know what serializers can I need to use. I think it should only be a json serializer, but I'm not entirely sure about that.

There doesn't seem to be a configuration example for Buildbot in their documentation. Also, https://crossbar.io website has been down for a couple of weeks now, and there is no guarantee that it will be up again. (See crossbario/crossbar#2085)

It would be great if the Buildbot documentation provided an example crossbar.io configuration for use with Buildbot.

Also, can you recommend crossbar.io alternatives? It currently uses blockchain and its container image size is over 2 Gb. I don't need to use decentralized network, I only need a WAMP router for my container network to connect between buildmaster and UI master. Maybe there is a simpler solution?

[ivanov17]

@tardyp @p12tic do you have any working example of a crossbar configuration for use with buildbot multimaster setup?

I would really appreciate any example.

My simple setup has a UI master, a buildmaster and a crossbar router, each in separate docker container. At the moment, my buildmaster does not start the scheduler after receiving changes when I use WAMP router. The same buildmaster configuration with a simple message queue works just fine.

All I get in the buildmaster logs after pushing to the git repository is

2023-08-07 13:37:06+0000 [Broker,0,10.89.3.1] perspective_addChange called
2023-08-07 13:37:06+0000 [-] added change with revision 390660d0cac3826d5bed0ae904ebc739bdcadd28 to database
2023-08-07 13:37:06+0000 [Broker,0,10.89.3.1] perspective_addChange called
2023-08-07 13:37:06+0000 [-] added change with revision c225113dcd3368e204472becbce24a0b3b10c72f to database
2023-08-07 13:37:06+0000 [Broker,0,10.89.3.1] perspective_addChange called
2023-08-07 13:37:06+0000 [-] added change with revision 23497bd842f2818cd54086dd2634aa9a63f694a8 to database

and nothing else.

[p12tic]

Indeed, we should look for alternatives to crossbar.

As for example config, the following should work (didn't test it right now):

.crossbar/config.json:

{
    "version": 2,
    "controller": {},
    "workers": [
        {
            "type": "router",
            "realms": [
                {
                    "name": "test_realm",
                    "roles": [
                        {
                            "name": "anonymous",
                            "permissions": [
                                {
                                    "uri": "",
                                    "match": "prefix",
                                    "allow": {
                                        "call": true,
                                        "register": true,
                                        "publish": true,
                                        "subscribe": true
                                    },
                                    "disclose": {
                                        "caller": false,
                                        "publisher": false
                                    },
                                    "cache": true
                                }
                            ]
                        }
                    ]
                }
            ],
            "transports": [
                {
                    "type": "web",
                    "endpoint": {
                        "type": "tcp",
                        "port": 1245
                    },
                    "paths": {
                        "ws": {
                            "type": "websocket"
                        }
                    }
                }
            ]
        }
    ]
}

which then can be configured on Buildbot side by the following:

c["mq"] = {
    "type" : "wamp",
    "router_url": "ws://localhost:1245/ws",
    "realm": "test_realm",
    "wamp_debug_level" : "warn"
}

One thing to note is that buildmaster must have the schedulers configured on that particular instance, so I would double check that if it doesn't start the builds.

[ivanov17]

@p12tic Thanks a lot. Yes, I've tried running a Crossbar container with a similar configuration that allows actions for an anonymous user, but to no avail. The only difference was that I used the websocket transport for the connection port. I don't think this is an important difference.

After some research I found a simple but functional WAMP router written in Go. It's called nexusd. It looks more lively than other lightweight routers.

It just works for me. You can build it with

go install github.com/gammazero/nexus/v3/nexusd@latest

and then run the binary, for example, with these options

nexusd --ws 0.0.0.0:8080 --realm buildbot

That's all, it works. Then you can to run the Buildbot multimaster and it should work. It can be more customizable with JSON based configuration, it allows caching, CORS policy etc. But it can work with Buildbot just from scratch.

I created a nexusd container image based on the Busybox image and it only needs 16Mb. I have not tested it in a high load environment, but for my working environment it works great.

Also, I created a PR for creating official nexusd container image, and it looks like mantainers will accept it. You can see the open issue here: gammazero/nexus#298

At the moment, the container image can be downloaded here: https://github.com/ivanov17/nexus/pkgs/container/nexusd

Unfortunately, there are no nexusd packages for Linux distributions. But I plan to create a request to the Fedora mantainers. Perhaps this will be a good start.

I have not looked at routers written in Java, but it seems to me that nexusd is the most promising for use with Buildbot.

[rvanlaar]

Here's the config that works for us:

{
    "$schema": "https://raw.githubusercontent.com/crossbario/crossbar/master/crossbar.json",
    "version": 2,
    "controller": {
    },
    "workers": [
        {
            "type": "router",
            "realms": [
                {
                    "name": "realm1",
                    "roles": [
                        {
                            "name": "anonymous",
                            "permissions": [
                                {
                                    "uri": "",
                                    "match": "prefix",
                                    "allow": {
                                        "call": true,
                                        "register": true,
                                        "publish": true,
                                        "subscribe": true
                                    },
                                    "disclose": {
                                        "caller": false,
                                        "publisher": false
                                    },
                                    "cache": true
                                }
                            ]
                        }
                    ]
                }
            ],
            "transports": [
                {
                    "type": "rawsocket",
                    "endpoint": {
                        "type": "tcp",
                        "port": 8081,
			"interface": "127.0.0.1",
                        "backlog": 1024
                    },
                    "options": {
                        "max_message_size": 1048576
                    },
                    "serializers": ["cbor", "msgpack", "json"]
                },
                {
                    "type": "web",
                    "endpoint": {
                        "type": "tcp",
                        "port": 8080,
			"interface": "127.0.0.1",
                        "backlog": 1024
                    },
                    "paths": {
                        "/": {
                            "type": "static",
                            "directory": "../web",
                            "options": {
                                "enable_directory_listing": true
                            }
                        },
                        "ws": {
                            "type": "websocket",
                            "serializers": [
                                "cbor", "msgpack", "json"
                            ],
                            "options": {
                                "allowed_origins": ["*"],
                                "allow_null_origin": true,
                                "enable_webstatus": false,
                                "max_frame_size": 1048576,
                                "max_message_size": 1048576,
                                "auto_fragment_size": 65536,
                                "fail_by_drop": true,
                                "open_handshake_timeout": 2500,
                                "close_handshake_timeout": 1000,
                                "auto_ping_interval": 0,
                                "compression": {
                                    "deflate": {
                                        "request_no_context_takeover": false,
                                        "request_max_window_bits": 13,
                                        "no_context_takeover": false,
                                        "max_window_bits": 13,
                                        "memory_level": 5
                                    }
                                }
                            }
                        },
                        "info": {
                            "type": "nodeinfo"
                        }
                    }
                }
            ]
        }
    ]
}

[ivanov17]

@rvanlaar Thank you, but I'm currently using nexusd. I'm using containerized applications in my setup, but the official crossbar container turned out to be unusable. Since they also added blockchain (see crossbario/crossbar#2078 for example), I decided I didn't want to dig into that to create my own container image.

Unfortunately, nexusd also has problems. Sometimes there are timeouts, so Buildbot Web UI process stops and the container needs to be restarted. This is a bit annoying because it only stops the Buildbot process, but Twisted still runs. I use healthchecks to control this.

But this does not affect the buildmaster container, builds are always executed. I hope that nexusd maintainers will fix the timeouts issue. At least the task has already been set: gammazero/nexus#294

[rvanlaar]

I also started using nexus. After not being able to upgrade packages due to crossbar version pins. So far, it looks good. No outage in the last day.

The timeouts you mentioned were also happening for us with crossbar.

We set the timeout in crossbar to 0.

For us the timeouts seemed to coincide with the buildbot process being overloaded, i.e. hitting more than 100% cpu for an extended period.

We solved that by splitting it up into two ui, two master processes and two workers.