Salt master behind ELB or proxy

[shortstack]

Description of Issue

Would like to have 1 or more Salt masters in private subnets, behind a proxy or load balancer, not necessarily for load balancing, but just an added layer of security in front of the systems.

Have looked at a dozen other issues looking to do this and all went cold or are outdated. Hoping someone has made this possible by now. I'd rather not have these systems sitting in public subnets.

Thought about setting up a Syndic but that kind of leaves us in the same place--with a master open to the public.

Setup

Salt master(s) running in AWS on Ubuntu 16.04 in private subnet(s)
Set up either behind Nginx or an ELB with TCP listeners in public subnet(s)
Set transport: tcp in master and minion configs.

Versions Report

Salt Version:
Salt: 2019.2.0

Dependency Versions:
cffi: Not Installed
cherrypy: 3.2.3
dateutil: 2.4.2
docker-py: Not Installed
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.12 (default, Nov 12 2018, 14:36:49)
python-gnupg: 0.3.8
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: Ubuntu 16.04 xenial
locale: UTF-8
machine: x86_64
release: 4.4.0-1087-aws
system: Linux
version: Ubuntu 16.04 xenial

[shortstack]

so far, just testing with 1 master behind either an ELB or an nginx TCP stream

keys get exchanged and accepted, salt-key shows minions

but beyond that, no commands can be run, no minions return

the debug logs just say that the connection is made and then immediately closed/dropped

[shortstack]

2019-08-23 20:11:51,306 [salt.crypt       :207 ][DEBUG   ][19307] salt.crypt.get_rsa_pub_key: Loading public key
2019-08-23 20:11:51,315 [salt.crypt       :868 ][DEBUG   ][19307] Decrypting the current master AES key
2019-08-23 20:11:51,315 [salt.crypt       :199 ][DEBUG   ][19307] salt.crypt.get_rsa_key: Loading private key
2019-08-23 20:11:51,316 [salt.crypt       :797 ][DEBUG   ][19307] Loaded minion key: /etc/salt/pki/minion/minion.pem
2019-08-23 20:11:51,319 [salt.crypt       :207 ][DEBUG   ][19307] salt.crypt.get_rsa_pub_key: Loading public key
2019-08-23 20:11:51,320 [salt.transport.tcp:308 ][DEBUG   ][19307] Closing AsyncTCPReqChannel instance
2019-08-23 20:11:51,321 [salt.transport.tcp:1058][DEBUG   ][19307] tcp stream to x.x.x.x:xxxx closed, unable to recv
2019-08-23 20:12:41,103 [salt.minion      :981 ][ERROR   ][19307] Minion unable to successfully connect to a Salt Master.

[garethgreenaway]

@shortstack Thanks for the report. I've had good luck in the past having the Salt master sitting behind the HAProxy load balancer using the standard zeromq setup. Make sure you're setting up port Salt master ports, 4505 and 4506, on the load balancer.

[shortstack]

thank you, @garethgreenaway!! looking through docs right now. if you have any sample configs that have worked for you, do you mind sharing?

struggle-bus-ing to find anything zeromq and haproxy related. and using just the tcp listener for haproxy results in a timeout for the minions.

[shortstack]

DISREGARD! it's alive! thank you!

[garethgreenaway]

@shortstack Awesome! Glad it's working. If everything is good please feel free to close this issue out. Thanks!

[shortstack]

@garethgreenaway did you ever have issues with runners / presence detection in this scenario? transport is the default / zeromq setup. i enabled presence detection on the master. everything else works fine, but manage.alived/present/joined/allowed all return zero minions, despite them all being up (with manage.up) and available.

[garethgreenaway]

@shortstack not that I recall. The one caveat i do remember from this approach is that the minions won't be connected to both masters at the same time so things like this likely won't work on multiple masters, just the one that the minion is connected to.

[shortstack]

current setup is only 1 master behind haproxy, so they are all going to the same one. what are the nuts and bolts behind presence detection? does that work via zeromq as well? could it maybe be getting blocked somewhere that i'm not aware of?

minions are distributed across multiple cloud environments.

i have a similar setup in another env, master and minions are the same versions as the one above. only 2 differences are 1) no haproxy in the middle, going straight to the master, and 2) all of the systems (master and minions) are on the same network.

[garethgreenaway]

If it's one master and presence isn't working that is weird, have you tried looking at the event runner to see if the presence events are ending up in the queue?

[shortstack]

yep =/

salt/run/20190909150519387060/new       {
  "_stamp": "2019-09-09T15:05:19.790032",
  "fun": "runner.manage.alived",
  "fun_args": [],
  "jid": "20190909150519387060",
  "user": "sudo_ubuntu"
}
salt/run/20190909150519387060/ret       {
  "_stamp": "2019-09-09T15:05:19.812438",
  "fun": "runner.manage.alived",
  "fun_args": [],
  "jid": "20190909150519387060",
  "return": [],
  "success": true,
  "user": "sudo_ubuntu"
}

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[max-arnold]

Check out the new master cluster feature that is part of 3007rc1: https://github.com/saltstack/salt/blob/master/doc/topics/tutorials/master-cluster.rst

For more details see https://github.com/saltstack/salt-enhancement-proposals/blob/1433501a1417f78c895345a675e21a8b6382bb61/0000-master-cluster.md