proxy routes broken with multiple realm roles

[om26er]

If a realm route has only one role, everything works fine. However if there are multiple roles for a realm and the route is configured, the round robin logic is broken

crossbar/crossbar/worker/proxy.py

Lines 1386 to 1388 in fa0e418

	routes = self._routes[realm_name]
	self._roundrobin_idx = (self._roundrobin_idx + 1) % len(routes)
	route = list(routes.values())[self._roundrobin_idx]

In the current implementation, a new realm route is started for each role

crossbar/crossbar/node/node.py

Lines 962 to 978 in fa0e418

	for i, realm_name in enumerate(worker.get('routes', {})):
	roles = worker['routes'][realm_name]
	for role_id, connections in roles.items():
	if not isinstance(connections, list):
	connections = [connections] # used to be a single string, now a list of strings
	for connection_id in connections:
	self.log.debug(
	"Starting proxy realm route {realm}, {role} to {connection}",
	realm=realm_name,
	role=role_id,
	connection=connection_id,
	)
	yield self._controller.call(
	'crossbar.worker.{}.start_proxy_realm_route'.format(worker_id),
	realm_name,
	{role_id: connection_id},
	)

However, the round robin logic above doesn't check if a route has the specific role and tries to access it, resulting in a KeyError.

[om26er]

Here is a super basic crossbar config for a proxy worker, enough to reproduce the bug described above

{
   "version": 2,
   "workers": [
      {
         "type": "proxy",
         "connections": {
            "conn1": {
               "transport": {
                  "type": "rawsocket",
                  "endpoint": {
                     "type": "tcp",
                     "host": "127.0.0.1",
                     "port": 9001
                  },
                  "url": "ws://localhost",
                  "serializer": "cbor"
               }
            }
         },
         "routes": {
            "realm1": {
               "system": "conn1",
               "admin": "conn1"
            }
         },
         "transports": [
            {
               "type": "universal",
               "endpoint": {"type": "tcp", "port": 8080},
               "rawsocket": {
                  "serializers": ["json", "cbor"],
                  "auth": {
                     "anonymous": {
                        "type": "static",
                        "role": "admin"
                     }
                  }
               }
            }
         ]
      }
   ]
}

[om26er]

Since the code runs as a roundrobin, it "succeeds" on first attempt and fails on the other

om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed (Connection was refused by other side: 111: Connection refused.)
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed ('admin')
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed (Connection was refused by other side: 111: Connection refused.)
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed ('admin')
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed (Connection was refused by other side: 111: Connection refused.)
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed ('admin')
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed (Connection was refused by other side: 111: Connection refused.)
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed ('admin')
om26er@P1:~/Documents/backend_bug$ wick publish --url rs://localhost:8080 --realm realm1 hello
received unexpected ABORT message when expecting WELCOME: wamp.error.authentication_failed message=Frontend connection accept failed (Connection was refused by other side: 111: Connection refused.)

[om26er]

Here are the relevant Crossbar logs. NOTE: the connection refused is expected as I am not running a backend.

2022-03-26T02:27:16+0500 [Proxy      199315] <crossbar.worker.proxy.ProxyFrontendSession.onOpen> Proxy frontend session connected from peer tcp4:127.0.0.1:43348
2022-03-26T02:27:16+0500 [Proxy      199315] <crossbar.router.auth.anonymous.PendingAuthAnonymous.hello>(realm=realm1, details.realm=realm1, details.authid=None, details.authrole=None) [config={'type': 'static', 'role': 'admin'}]
2022-03-26T02:27:16+0500 [Proxy      199315] <crossbar.worker.proxy.ProxyFrontendSession._accept> Frontend session accepted (Accept(realm=<realm1>, authid=<N75Y-P4SH-XKC3-9R3V-MLMQ-W34P>, authrole=<admin>, authmethod=anonymous, authprovider=static, authextra=None)) - opening proxy backend session ...
2022-03-26T02:27:16+0500 [Proxy      199315] realm1 admin
2022-03-26T02:27:16+0500 [Proxy      199315] 
Traceback (most recent call last):
--- <exception caught here> ---
  File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 490, in _process_Hello
    session = yield self._accept(hello_result)
  File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 1326, in map_backend
    backend_proto = yield make_backend_connection(backend_config, frontend, self._cbdir)
twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.

2022-03-26T02:27:16+0500 [Proxy      199315] <crossbar.worker.proxy.ProxyFrontendSession.onClose>(wasClean=True)
2022-03-26T02:27:16+0500 [Proxy      199315] <crossbar.worker.proxy.ProxyFrontendSession.onOpen> Proxy frontend session connected from peer tcp4:127.0.0.1:43350
2022-03-26T02:27:16+0500 [Proxy      199315] <crossbar.router.auth.anonymous.PendingAuthAnonymous.hello>(realm=realm1, details.realm=realm1, details.authid=None, details.authrole=None) [config={'type': 'static', 'role': 'admin'}]
2022-03-26T02:27:16+0500 [Proxy      199315] <crossbar.worker.proxy.ProxyFrontendSession._accept> Frontend session accepted (Accept(realm=<realm1>, authid=<APYW-FME9-NHTN-5P6F-E6SE-ESPR>, authrole=<admin>, authmethod=anonymous, authprovider=static, authextra=None)) - opening proxy backend session ...
2022-03-26T02:27:16+0500 [Proxy      199315] realm1 admin
2022-03-26T02:27:16+0500 [Proxy      199315] 
Traceback (most recent call last):
  File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 490, in _process_Hello
    session = yield self._accept(hello_result)
  File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 350, in _accept
    backend_d = self._controller.map_backend(
  File "/home/om26er/scm/crossbario/crossbar/venv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1905, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "/home/om26er/scm/crossbario/crossbar/venv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1815, in _cancellableInlineCallbacks
    _inlineCallbacks(None, gen, status)
--- <exception caught here> ---
  File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 490, in _process_Hello
    session = yield self._accept(hello_result)
  File "/home/om26er/scm/crossbario/crossbar/venv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 1309, in map_backend
    backend_config = self.get_backend_config(realm, authrole)
  File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 1392, in get_backend_config
    connection_id = route.config[role_name]
builtins.KeyError: 'admin'

2022-03-26T02:27:16+0500 [Proxy      199315] <crossbar.worker.proxy.ProxyFrontendSession.onClose>(wasClean=True)
2022-03-26T02:27:17+0500 [Proxy      199315] <crossbar.worker.proxy.ProxyFrontendSession.onOpen> Proxy frontend session connected from peer tcp4:127.0.0.1:43352
2022-03-26T02:27:17+0500 [Proxy      199315] <crossbar.router.auth.anonymous.PendingAuthAnonymous.hello>(realm=realm1, details.realm=realm1, details.authid=None, details.authrole=None) [config={'type': 'static', 'role': 'admin'}]
2022-03-26T02:27:17+0500 [Proxy      199315] <crossbar.worker.proxy.ProxyFrontendSession._accept> Frontend session accepted (Accept(realm=<realm1>, authid=<FWXL-F9FV-PQRV-SJVH-FT65-LN6L>, authrole=<admin>, authmethod=anonymous, authprovider=static, authextra=None)) - opening proxy backend session ...
2022-03-26T02:27:17+0500 [Proxy      199315] realm1 admin
2022-03-26T02:27:17+0500 [Proxy      199315] 
Traceback (most recent call last):
--- <exception caught here> ---
  File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 490, in _process_Hello
    session = yield self._accept(hello_result)
  File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 1326, in map_backend
    backend_proto = yield make_backend_connection(backend_config, frontend, self._cbdir)
twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.

2022-03-26T02:27:17+0500 [Proxy      199315] <crossbar.worker.proxy.ProxyFrontendSession.onClose>(wasClean=True)
2022-03-26T02:27:18+0500 [Proxy      199315] <crossbar.worker.proxy.ProxyFrontendSession.onOpen> Proxy frontend session connected from peer tcp4:127.0.0.1:43354
2022-03-26T02:27:18+0500 [Proxy      199315] <crossbar.router.auth.anonymous.PendingAuthAnonymous.hello>(realm=realm1, details.realm=realm1, details.authid=None, details.authrole=None) [config={'type': 'static', 'role': 'admin'}]
2022-03-26T02:27:18+0500 [Proxy      199315] <crossbar.worker.proxy.ProxyFrontendSession._accept> Frontend session accepted (Accept(realm=<realm1>, authid=<7W7V-PLHN-Q99L-QCVT-U7EK-H6LF>, authrole=<admin>, authmethod=anonymous, authprovider=static, authextra=None)) - opening proxy backend session ...
2022-03-26T02:27:18+0500 [Proxy      199315] realm1 admin
2022-03-26T02:27:18+0500 [Proxy      199315] 
Traceback (most recent call last):
  File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 490, in _process_Hello
    session = yield self._accept(hello_result)
  File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 350, in _accept
    backend_d = self._controller.map_backend(
  File "/home/om26er/scm/crossbario/crossbar/venv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1905, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "/home/om26er/scm/crossbario/crossbar/venv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1815, in _cancellableInlineCallbacks
    _inlineCallbacks(None, gen, status)
--- <exception caught here> ---
  File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 490, in _process_Hello
    session = yield self._accept(hello_result)
  File "/home/om26er/scm/crossbario/crossbar/venv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 1309, in map_backend
    backend_config = self.get_backend_config(realm, authrole)
  File "/home/om26er/scm/crossbario/crossbar/crossbar/worker/proxy.py", line 1392, in get_backend_config
    connection_id = route.config[role_name]
builtins.KeyError: 'admin'

2022-03-26T02:27:18+0500 [Proxy      199315] <crossbar.worker.proxy.ProxyFrontendSession.onClose>(wasClean=True)

[om26er]

Luckily this issue can be fixed with a one-liner

om26er@P1:~/scm/crossbario/crossbar$ git diff
diff --git a/crossbar/worker/proxy.py b/crossbar/worker/proxy.py
index 74869f73..11b07cf7 100644
--- a/crossbar/worker/proxy.py
+++ b/crossbar/worker/proxy.py
@@ -1384,6 +1384,7 @@ class ProxyController(TransportController):
         assert self.has_role(realm_name, role_name)
 
         routes = self._routes[realm_name]
+        routes = {route_id: route for route_id, route in routes.items() if role_name in route.config}
         self._roundrobin_idx = (self._roundrobin_idx + 1) % len(routes)
         route = list(routes.values())[self._roundrobin_idx]