HA - MongoDB and RocketChat Setup

[TheReal1604]

Your Rocket.Chat version: latest (0.39.0)

Hey guys,

i create a 2 Node setup with docker and a openstack HA proxy in front of these two.

The mongodb is a docker-container on both nodes in a replicaset. I created all users (admins, rocket (the user for rocket.chat, oplogger)

Now, i create rocket.chat docker containers with the following docker-compose config:

rocketchat:
  image: rocketchat/rocket.chat:latest
  environment:
    - PORT=3000
    - ROOT_URL=https://chat.domain.de
    - MONGO_URL=mongodb://rocket:password@rocket-1:27017,rocket-2:27017/rocketchat?replicaSet=rs0&readPreference=nearest&w=majority
    - MONGO_OPLOG_URL=mongodb://oplogger:password@rocket-1:27017,rocket-2:27017/local?authSource=admin&replicaSet=rs0
  ports:
    - 3000:3000
  extra_hosts:
    - "rocket-1:10.250.250.13"
    - "rocket-2:10.250.250.14"

My issue is now, the messages send are not displayed directly if you are connected to a different node, the user have to refresh the browser to see all other messages send to the other node. I checked https://github.com/RocketChat/Rocket.Chat.Docs/issues/68 already, but didnt get a hint to the right step to accomplish this.

I have tested the access without the load-balancer in front:

• Opening node1 rc instance in browser1
• Opening node2 rc instance in browser2
• Authenticate on both with the same user
• write a message on node1
  -> message not shown on node2 until i reload the site in browser2

The websocket are established correctly, too.

Anyone have a suggestion?

Edit: I checked also this documentation: https://rocket.chat/docs/installation/manual-installation/ubuntu/ -> no success

[TheReal1604]

I now tested to add a third node.. just to be sure the mongodb replicaset is correctly working because i read here: https://docs.mongodb.com/manual/core/replica-set-architecture-three-members/ that the minimum is three nodes.

Same problem. I am with my Smartphone from the internet on node1 and from internal on node3. RC needs a refresh to show all messages. (Via Source-IP LB)

[geekgonecrazy]

So from the look of this... you have two mongodb servers... but doesn't sound like they have been clustered together properly.

Start with a mongodb cluster... get it going then move to scaling Rocket.Chat.

Check out this for mongodb cluster with docker: https://medium.com/@gargar454/deploy-a-mongodb-cluster-in-steps-9-using-docker-49205e231319#.egh4k3t96 Also tons of resources out there on doing this.

[engelgabriel]

Please also see #4019 (comment)

[TheReal1604]

@geekgonecrazy @engelgabriel @rodrigok I tried to set the env INSTANCE_IP like rodrigok suggested.. But the issue persists.. there must be an other thing. Or should i delete the complete data in my rocketchat database?

[TheReal1604]

Ah, new error messages! I have now this error in my logs when the instance try to talk to each other:

streamBroadcast.coffee:83 StreamBroadcast ➔ Stream.error Stream broadcast from '10.250.250.17:3000' to 'localhost:3000' with name notify-all not authorized

[engelgabriel]

Did you try running the develop branch?

[TheReal1604]

@engelgabriel Nope. I run the latest docker container in production.

But.. SUCCESS!

After the addition from @rodrigok to set the env INSTANCE_IP on each RC docker container and unlock the network communication between each RC instance on port 3000 all is working like a charm now. Thank you @geekgonecrazy @rodrigok @engelgabriel you´re awesome. :-)

[drickerusa]

@geekgonecrazy @engelgabriel would this be the best place to document my testing and deployment or soon to be deployed to production. Or how would you like for me to do this? Is there a template to follow etc.? It might be helpful for those who are deploying HA RC deployments. I'm still doing some automated testing which I would love to get some feedback on. I'm not using docker in this implementation of RC.

[geekgonecrazy]

@drickerusa I've open a ticket on our documentation repo. This is where i've been trying to collect info and do a write up. RocketChat/docs-old#87

If docker specific you could document there or create another ticket on that repo. I just started there because Its easier to find. But its up to you 😄