How to configure TCP correctly (proxy.addr, ...)

[kopax]

Hi everyone!

Thanks for sharing Fabio, I have heard good thing about it.
I am following the project since it has been integrated in PanteraS

I wanted to try Fabio because it now has a TCP proxying mode.
I am configuring my services by registering them in consul and adding env tag to my docker containers :

urlprefix-mysite.com/                              # host specific catch all route
urlprefix-:587 proto=tcp                          # route external port 587

I wasn't able to use the tcp proxying without playing with proxy.addr in fabio.properties

1. Do I have to configure proxy.addr in fabio.properties everytime I am adding a new tcp proxied port ?
2. If yes, how can I add multiple port ? I see my configuration with a very long line in it.
3. I have consul running, isn't it possible to get this value configured automatically ?
4. I use to proxy a mail-server on HAproxy using 4 TCP ports : 25, 143, 587, 993

• I have tried to send an email from thunderbird with proto=tcp,:587;proto=tcp and I have:

  Sending of the message failed.
  An error occurred while sending mail: Unable to establish a secure link with Outgoing server (SMTP) mail.kopaxgroup.com using STARTTLS since it doesn't advertise that feature. Switch off STARTTLS for that server or contact your service provider
  

• I have tried to send an email from thunderbird with proto=tcp,:587;proto=tcp+sni it connect but I wait a very long time and I have a time out error.

I didn't had to configure anything for proxying these services before. I can see I can configure ssl keystore and tls header value. I am not sure which is the right direction. Maybe you will have any idea why it is different ?

Thanks in advance

[magiconair]

For TCP proxying to work two things need to happen:

1. fabio needs to know that traffic on a given port is to be handled by the TCP proxy (i.e. copy bytes around instead of parsing HTTP requests). This is why you have to configure the TCP listener with -proxy.addr :3306;proto=tcp. Port 3306 in this example is the port on which the exposed service is available to the outside world.

2. services which can handle that traffic need to tell fabio that they want traffic that arrives on port 3306 on fabio. The service itself runs on a random port. This is why the service advertises urlprefix-:3306 proto=tcp.

So with this in mind lets go through your questions:

1. yes, you have to configure a TCP listener on fabio for every exposed port (not service) since I haven't implemented dynamic listeners at the moment. This will come but it isn't there yet.
2. if you want to expose multiple ports do this -proxy.addr = :25;proto=tcp,:143;proto=tcp,:587;proto=tcp,:993;proto=tcp it is a comma separated list of semicolon separated k=v pairs (imagine the first entry having a default key like port)
3. not right now since this isn't a consul problem but there is no code in fabio ATM to bring up and shutdown listeners automatically. It'll come.
4. try the listener config in 2. and make sure your services are registered with the urlprefix-:25 proto=tcp, urlprefix-:143 proto=tcp ,... tags in consul

Let me know if that answers your question.

[kopax]

Hi, thanks. It get more clear for 1,2,3.

Interesting that you use mysql port :3306 for your example, is this really working ?
I was trying on panteras and haproxy to proxy postgresql and mysql database and it never worked I never used the TCP proxy for them and I never knew why.

I have tried in fabio.properties:

 proxy.addr = :25;proto=tcp,:143;proto=tcp,:587;proto=tcp,:993;proto=tcp

This work fine I can do telnet on each port. I want to send an email using port 587 with STARTTLS.
I wasn't configuring anything in HAproxy.

I have this error when I try to send an email from thunderbird:

Sending of the message failed.
An error occurred while sending mail: Unable to establish a secure link with Outgoing server (SMTP) mail.domain.com using STARTTLS since it doesn't advertise that feature. Switch off STARTTLS for that server or contact your service provider.

I am reading this but not sure if it is the right direction. Maybe it is related to proxy.header.tls proxy.header.tls.value

I haven't configured anything regarding certificate. It's always a mess to understand what needs to be configured and how it should be. Do I really have to do that here in Fabio ? I use to do SSL terminaisons on nginx. I never had to configure anything with haproxy when doing TCP proxying.

[magiconair]

proxy.header.* only matters for HTTP(S) proxies. You can terminate TLS on fabio or not. In plain TCP mode fabio should just copy the bytes it receives to the service. you can use openssl s_client for testing.

[kopax]

Thanks for your indication, apparently I have verified and SSL is fine using plain TCP, the handshake is done after Fabio.

I might have misunderstood the basic usage scenario...

I use to have HAproxy installed, and I was registering my TCP services using haproxy_tcp=587

Now I am registering with urlprefix-:587 proto=tcp AND I have to configure proxy.addr in fabio.properties

I noticed that using proxy.addr without listen address (:587;proto=tcp) will open that port on all my interfaces.

This is fine, my mail server needs to be public, but how about if I need to keep it private ? I though, ok just set the ip like (172.16.0.3:587;proto=tcp), this work, but how about if I don't know on what slave will run my application ?

Should the fabio.properties be the same for all my masters ? I was only able to access the services if I configure Fabio on that same master host (which is also a slave).

I have this assumption, if I open :587 using proxy.addr, it appear it is just for the current host application and it does not load balance anything except the local application. If this is true with what configuration can I do load balancing from a single fabio instance ?

[magiconair]

Service discovery is done through consul. Fabio finds the service instances that are registered in consul and starts routing to them once they appear and stops routing when they are gone. This is different from other proxies where you have to configure the upstream servers.

As for the bind address: proxy.addr=:587 is the same as proxy.addr=0.0.0.0:587 which will listen on all interfaces. Currently there is no method to say proxy.addr=eth0:587 but I'm planning on doing that. Therefore, you either have fabio listen on all interfaces or you have to configure it per host either through the config file or startup parameters or env vars. All will work for every parameter in fabio.properties.

[kopax]

😢 I will revert to HAproxy, but only because it's impossible to proxy using a specific interface yet and I have services that require it. For when (approx) do you schedule your release ? I am waiting for it with impatience, from what I have seen and what I have played with, it is a great tool. Very easy and handy. Thanks for sharing it.

[magiconair]

Let me see what I can hack together. This isn't difficult. Didn't have the need for it yet since you would need to run fabio via something like supervisord, daemontools or systemd anyway which can do that little bit of shell magic.

[kopax]

Sure, just tell me how I can help.

It would be nice if you could create a de configuration variable. for example, instead of using the default :3600 for 0.0.0.0:3600, use a proxy.listen_ip='172.16.0.1' variable that would be used as a default interface.
This is standard variable and it could be used in panteras with the LISTEN_IP env settings.

[magiconair]

You can always use the shell to get the ip address, e.g.:

fabio -proxy_addr "$(ifconfig eth0 | grep 'inet addr:' | awk '{print $2}' | cut -d: -f2):3600;proto=tcp"

[magiconair]

listen_ip isn't going to happen since you can have more than one listener.

[kopax]

Sorry, I meant having the possibility to configure a default interface instead of 0.0.0.0
I have set some iptables rules that configure my public interface by first allowing a few public ports then dropping the rest. It was working fine enough with HAproxy. I only had to configure one ENV tag and consul_haproxy_template was updating haproxy proxy configuration. All HAproxy exposed TCP service are accessible through the private lan interface. If I needed to open a service I only had to add a new allow rules for that new port into the iptables chain. For some reason, Fabio is exposing on top of my firewall rules because of the default 0.0.0.0, is there a way to avoid that ?

[magiconair]

As I've said, use the shell to get the ip address(es) you want fabio to bind to and then run fabio -proxy.addr "1.2.3.4:3306;proto=tcp,5.6.7.8:9876,..."

[kopax]

have this assumption, if I open :587 using proxy.addr, it appear it is just for the current host application and it does not load balance anything except the local application. If this is true with what configuration can I do load balancing from a single fabio instance ?

This is the other part that I don't understand, how do I know the ip address from the host that run Fabio ? I though load balancing was for that. If I scale up on marathon I have no way to send the new application ip to Fabio.

This is also why I do need fabio to listen by default on a different interface than 0.0.0.0 otherwise, to stop Fabio placing rules on top of mine.

[magiconair]

Maybe lets take a step back and explain what it is that you're trying to do. Since you mention marathon the setup is a bit more complex. (This should have registered when you mentioned PanteraS).

The usual deployment scenarios are listed here: https://github.com/fabiolb/fabio/wiki/Deployment

Could you explain briefly what your setup looks like?

[kopax]

So far, I only have 3 hosts Master+Slave where PanteraS is installed.
They all have a private interface and together create a private cloud network on 172.16.x.x
I am using consul and HAproxy for HA and load balancing.

I have installed on one of them NGINX, which is used as the public https proxy and all the public accesses.

In PanteraS, HAproxy can also be used to proxy TCP. I sometimes open publicly on the public interface a new port proxied by HA proxy. (587,993) for example.

Note that all my Master+Slave hosts all have the same HAproxy configuration setup and could be used has an emergency proxy if needed.

[kopax]

Any update on this ? I hope I can migrate soon but the security requirement can't replace the HAproxy completely.

[magiconair]

I'm sorry but I still don't fully understand the problem you're trying to solve. Could you please try to explain it briefly again? Who is connecting to whom, with which protocol and what isn't working?

[kopax]

Who is connecting to whom

All my services are using consul service discovery, see panteras, they use environment tags in the container to register services into LB.

which protocol

TCP

what isn't working

I tried to replace HA proxy with Fabio and it doesn't fit all the previous requirements yet.

1. There is no dynamic listeners at the moment like in HA proxy.
2. The load balancer Fabio can't bind on my private interface eth3 and :587;proto=tcp is by default listening on all interface, I need to be able to specify the bind interface dynamically (by not specifying the address ip directly / by setting it in the configuration)
3. HAproxy is running on the 3 masters (A/B/C) and by default I dropped all ports in the iptables for my public interface. Re-enabling ports one one host (C) allow me to nominate a public proxy easely. With Fabio the load balancing TCP configuration is more complicated because of (2), if the application starts on (A) the first time or (B) the second time it will always use the host port. The two fabio's instance left are not able to do the load balancing to the service.

[magiconair]

1. yes, no solution right now other than generating the config file using consul-template for example.
2. use a shell script or consul-template to determine that
3. using iptables and private ip addresses is like using belts and suspenders. When you use iptables to block all ports you can have fabio bind to :port. Otherwise, use consul-template for now.

So what I understand is that fabio basically works but the ip and listener configuration isn't flexible enough for your use case. While I can add something like bind to interface quickly I can't provide dynamic listeners that easily.

I think for now consul-template is your best option to generate the config file you need and have fabio restarted on change.

[kopax]

We need to be able to bind to interface with at least a default interface. The possibility to do that using the ip address instead will block us in different use case. I will use it, please let me know if you decide do to it.

I will take your advice and implement the consul-template next, I though panteras was already taking care of that when using START_FABIO=true but after looking at the code I couldn't find anything like that except for HAproxy.
I understood from @sielaq that Fabio didn't need to be restarted but maybe it changed?

Thanks again for your answer.

[magiconair]

Why can't you use a shell script to determine the ip address in the meantime as outlined here?

#283 (comment)

This shouldn't be blocking.

[kopax]

For keeping flexibility of configuration. I don't always have the same interface name used. Also, on some host I have up to 3 network interfaces.

[magiconair]

But if you run ./fabio -proxy.addr 'eth0:9999' you have the same problem. I think what you need is https://github.com/hashicorp/go-sockaddr and the template language to find a private ip address. This is what we're using in consul.

[kopax]

That work for me. I think I got all the advice I needed to try one more migration. I haven't play with consul-template yet, the one I am using has been made by sielaq.

Closing for now. Thanks you again.