Crossbar connection between Windows agent and OCS

[henry-e-n]

Hi, I am trying to setup a local OCS system for our lab and am running into some trouble getting our different nodes to communicate properly.

First, some details: we have two computers, one Linux one Windows. Both are connected to our university network, and to a local network switch. Both can send and receive pings to each other over the local network. Both have working instances of OCS and Docker installed.

Ideally, we would run all of OCS on a single Linux machine which would house the containers for Grafana, InfluxDB and Publisher, Crossbar.io, and various other agents. Unfortunately, the Bluefors software we use is only supported on Windows machines at the moment. We have setup OCS to run on the Windows machine, and configured the Bluefors agent (see simonsobs/socs/) to read the Bluefors logs.

Our question is, how can we connect the Bluefors agent on the Windows machine to the instance of OCS running on the Linux machine such that it can publish data from the Bluefors logs to the instance of Grafana hosted on the Linux machine? In my understanding of the OCS architecture, it is the crossbar server that is primarily responsible for communicating the feed messages from clients to Grafana. As such, is there a way to simply manipulate the setup of crossbar on the two nodes (Linux and Windows) such that the Bluefors agent in network mode "host" can publish to the same server? If so, what specifically in the SCF and the docker compose files of both nodes would need to be altered from a simple single node setup?

Please see the attached file for a basic diagram of our wanted setup and let me know if there is any more information you need from me to help diagnose and fix this issue.

Thanks,
Henry

[BrianJKoopman]

Hi Henry,

Such a configuration is definitely possible. A few suggestions:

• Make sure the crossbar server container exposes the required port 8001 beyond just localhost. The ports section should look like:

    ports:
      - "8001:8001"

not like:

    ports:
      - "127.0.0.1:8001:8001"

• Check that the configured wamp_server and wamp_http addresses in the Windows machine's OCS site config file match the linux machine's address. That'll look like:

  wamp_server: ws://<ip of your linux machine>:8001/ws
  wamp_http: http://<ip of your linux machine>:8001/call
  wamp_realm: test_realm
  address_root: observatory

• Check Windows firewall isn't blocking the connection.

It would be helpful to see your configs if you could share, either here or offline.

Thanks,
-Brian

[henry-e-n]

Hi Brian,
Thanks for your reply. I have tried to make the changes you suggested but still seem to be having some trouble. For starters, I am not entirely sure I would even know if it is working. I was hoping it would be as simple as making the above changes, and then I would see the new measurements show up under the InfluxDB data source in Grafana. This does not seem to be the case, so either the issue is not yet fixed, or there is something else that needs to be changed.

I have included our config files below (with identifiers removed) such that our troubleshooting might be of use to others. We have established a Windows firewall rule that should allow all connections coming from the Linux machine, so I do not believe that is a problem.

LINUX MACHINE
site config file - default.yaml

# Site configuration for observatory.
hub:
  wamp_server: ws://<IP of LINUX on Local Network>:8001/ws
  wamp_http: http://<IP of LINUX on Local Network>:8001/call
  wamp_realm: test_realm
  address_root: observatory

hosts:

  <hostname>: {
    # Directory for logs.
    'log-dir': '/home/<user>/log/ocs/',

    # List of additional paths to Agent plugin modules.
    'agent-paths': [
        '/home/local/git/ocs/agents/',
        '/home/local/git/socs/agents/',
    ],

    # Agents running directly on the host machine
    'agent-instances': [
      {'agent-class': 'HostManager',
       'instance-id': 'hm-1'},
    ]
  }

  <hostname>-docker: {
    # Address of crossbar within Docker (based on service name)
    'wamp_server': 'ws://crossbar:8001/ws',
    'wamp_http': 'http://crossbar:8001/call',

    # Agents running within Docker containers
    'agent-instances': [
      {'agent-class': 'InfluxDBAgent',
       'instance-id': 'influxagent',
       'arguments': ['--initial-state', 'record']},
      {'agent-class': 'FakeDataAgent',
       'instance-id': 'fake-data1',
       'arguments': ['--mode', 'acq',
                     '--num-channels', '16',
                     '--sample-rate', '4']}
    ]
  }

docker-compose.yaml

version: '3.7'
volumes:
  grafana-storage:
  influxdb-storage:

services:
  # --------------------------------------------------------------------------
  # Grafana for the live monitor.
  # --------------------------------------------------------------------------
  grafana:
    image: grafana/grafana:latest
    ports:
#      - "127.0.0.1:3000:3000"
      - "<IP of LINUX on University Network>:3000:3000"
    volumes:
      - grafana-storage:/var/lib/grafana

  # InfluxDB Backend for Grafana
  influxdb:
    image: influxdb:1.7
    container_name: "influxdb"
    restart: always
    ports:
      - "8086:8086"
    environment:
      - INFLUXDB_HTTP_LOG_ENABLED=false
    volumes:
      - influxdb-storage:/var/lib/influxdb

  # --------------------------------------------------------------------------
  # Crossbar Server
  # --------------------------------------------------------------------------
  crossbar:
    image: simonsobs/ocs-crossbar:latest
    ports:
      - "8001:8001" # expose for OCS to ports beyond localhost

    environment:
         - PYTHONUNBUFFERED=1

  # --------------------------------------------------------------------------
  # OCS Components
  # --------------------------------------------------------------------------
  # Fake Data Agent for example housekeeping data
  ocs-fake-data1:
    image: simonsobs/ocs:latest
    hostname: <hostname>-docker
    environment:
      - INSTANCE_ID=fake-data1
      - LOGLEVEL=info
    volumes:
      - ${OCS_CONFIG_DIR}:/config:ro

  # InfluxDB Publisher
  ocs-influx-publisher:
    image: simonsobs/ocs:latest
    hostname: <hostname>-docker
    environment:
      - INSTANCE_ID=influxagent
    volumes:
      - ${OCS_CONFIG_DIR}:/config:ro

WINDOWS MACHINE
site config file

# Site configuration for a fake observatory.
hub:
# instead of local host
  wamp_server: ws://<IP of LINUX on Local Network>:8001/ws
  wamp_http: http://<IP of LINUX on Local Network>:8001/call
  wamp_realm: test_realm
  address_root: observatory

hosts:

  localhost: {
  # Directory for logs.
    'log-dir': '/home/<user-windows>/log/ocs/',
    # List of additional paths to Agent plugin modules.
    'agent-paths': [
        '/home/<user-windows>/ocs/ocs/agents/',
        '/home/<user-windows>/socs/socs/agents'
    ],

    # Agents running directly on the host machine
    'agent-instances': [
      {'agent-class': 'HostManager',
       'instance-id': 'hm-1'},
      ]
  }
  localhost-docker: {
    # Address of crossbar within Docker (based on service name)
    'wamp_server': 'ws://crossbar:8001/ws',
    'wamp_http': 'http://crossbar:8001/call',

    # Agents running within Docker containers
    'agent-instances': [
    {'agent-class': 'BlueforsAgent',
     'instance-id': 'bluefors',
     'arguments': [['--log-directory', '/logs']]
     }
    ]
  }

docker-compose.yaml

version: '3.7'
volumes:
  grafana-storage:
  influxdb-storage:

services:
  # I have also tried adding a crossbar server container like the one in the Linux docker compose in case 
  # that is necessary, however, it did not seem to make a difference. 
  ocs-bluefors-agent:
    image: ocs-bluefors-agent
    hostname: localhost-docker
    network_mode: "host"
    environment:
      - INSTANCE_ID=bluefors
      - LOGLEVEL=info
      - FRAME_LENGTH=600
      - STALE_TIME=2
      - MODE=follow
    volumes:
      - ${OCS_CONFIG_DIR}:/config:ro
      - /mnt/c/Users/<user-windows>/bluefors/logs/:/logs:ro

Lastly, in my troubleshooting attempts, I navigated to

wamp_http: http://<IP of LINUX on Local Network>:8001/call

and was met with the following error:

{"error":"Method not accepted by the REST bridge. (b'GET' not allowed, only POST, PUT)","args":[],"kwargs":{}}

Response Headers

  Access-Control-Allow-Credentials	true
  Access-Control-Allow-Origin	*
  Cache-Control	        no-store,no-cache,must-revalidate,max-age=0
  Content-Length	110
  Content-Type	        application/json; charset=UTF-8
  Date                      	Thu, 30 Nov 2023 15:10:26 GMT
  Server	                Crossbar/20.8.1

Request Headers

  Accept text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
  Accept-Encoding	gzip, deflate
  Accept-Language	en-US,en;q=0.5
  Connection	        keep-alive
  Host	                        <IP of LINUX on Local Network>:8001
  Upgrade-Insecure-Requests	1
  User-Agent	        Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0

Thanks again for you help!

• Henry

[BrianJKoopman]

Thanks for posting your configs. I think I see the problem. In the Windows machine's ocs config file the localhost-docker host block has the following:

    # Address of crossbar within Docker (based on service name)
    'wamp_server': 'ws://crossbar:8001/ws',
    'wamp_http': 'http://crossbar:8001/call',

This overrides the main config of:

  wamp_server: ws://<IP of LINUX on Local Network>:8001/ws
  wamp_http: http://<IP of LINUX on Local Network>:8001/call

However, the docker container on the Windows machine can't resolve crossbar to the correct address and so cannot connect. This config would probably work if there was an overlay network and the Windows docker container was connected to it. However, I'd recommend just dropping this override in the localhost-docker block.

[henry-e-n]

Hi Brian,

That explanation makes sense and I have now removed that section but still cannot seem to get the Bluefors feed to be recognized on the Linux Grafana.

My Windows SCF now looks like this:

# Site configuration for a fake observatory.
hub:
# instead of local host
  wamp_server: ws://<IP of LINUX on Local Network>:8001/ws
  wamp_http: http://<IP of LINUX on Local Network>:8001/call
  wamp_realm: test_realm
  address_root: observatory

hosts:

  localhost: {
  # Directory for logs.
    'log-dir': '/home/<user-windows>/logs-ocs',
    # List of additional paths to Agent plugin modules.
    'agent-paths': [
        '/home/<user-windows>/ocs/ocs/agents/',
        '/home/<user-windows>/socs/socs/agents'
    ],

    # Agents running directly on the host machine
    'agent-instances': [
      {'agent-class': 'HostManager',
       'instance-id': 'hm-1'},
      ]
  }
  localhost-docker: {
    # Agents running within Docker containers
    'agent-instances': [
    {'agent-class': 'BlueforsAgent',
     'instance-id': 'bluefors',
     'arguments': [['--log-directory', '/logs']]
     },
    {'agent-class': 'FakeDataAgent',
     'instance-id': 'fake-data1',
     'arguments': ['--mode', 'acq',
                   '--num-channels', '16',
                   '--sample-rate', '4']},
    ]
  }

In my own troubleshooting attempts I noticed that the container logs on Docker Desktop seem to be unable to connect to the "twisted.internet.endpoints" (see below). Also, when I try to navigate to the crossbar server at:

wamp_http: http://<IP of LINUX on Local Network>:8001/call

from the Windows machine, the site cannot be reached. This suggests to me a network issue, but as I said before, both machines can ping one another and I do not believe the firewalls are impeding any traffic.

Let me know if you have any suggestions.
Thanks,
Henry

[BrianJKoopman]

In my own troubleshooting attempts I noticed that the container logs on Docker Desktop seem to be unable to connect to the "twisted.internet.endpoints" (see below). Also, when I try to navigate to the crossbar server at:

wamp_http: http://<IP of LINUX on Local Network>:8001/call

from the Windows machine, the site cannot be reached. This suggests to me a network issue, but as I said before, both machines can ping one another and I do not believe the firewalls are impeding any traffic.

I do think you continue to have a networking problem. Do you control the router in your lab? It sort of sounded like everything was assigned an address by your University IT department. If that's the case there could be some firewall rules between the devices that you are unaware of. I would check with them, or figure out another networking configuration. It was common in SO for labs to use a small router and switch for their lab networks. You might want to check with your IT department about ideal setups.

Your idea for checking the http://<linux IP>:8001/call address from the Windows machine is a good one. That should return some json. The address http://<linux IP>:8001/ws will also return a page like this:

[henry-e-n]

Ok thanks, that is sort of what I figured! I will get in touch with our university's IT department and hope they can help me out. I will update/resolve this thread once it looks like things are working.

Thanks again,
Henry