AWS Network Load Balancer uses PROXY protocol version 2 with TLVs

[scampbell123]

Hi, I've got 2 Proftpd servers running on AWS EC2 instances behind an ELB (Network Load Balancer). I need the end-user client IPs in my transfer logs for audit purposes. I've downloaded/configured/installed the mod_proxy_protocol.c file. It loads fine into my proftpd config. I loaded it last, after all my other LoadModule entries.

My Directives:

 ProxyProtocolEngine on
 AllowForeignAddress on
 ProxyProtocolVersion haproxyV2

Once I enabled the "Proxy Protocol v2" in AWS, I get these errors in my proftpd logs:

mod_proxy_protocol/0.1: error reading proxy info: Invalid argument
mod_proxy_protocol.c: error initializing session: Operation not permitted

My gut says that the ProxyProtocolVersion does not support AWS ELBs right now. Has anyone else gotten this to work? Does it really only work with the haproxy package?

Thanks for any guidance,
Scott

[Castaglia]

In your proftpd.conf, you might enable mod_proxy_protocol trace logging to get more information:

TraceLog /path/to/proftpd-trace.log
Trace proxy_protocol:20

Please provide that generated trace logging, when you can. Thanks!

[scampbell123]

Sure thing. Here it is below:

2019-05-08 14:49:42,660 [19914] <proxy_protocol:19>: waiting for max of 3 secs while polling socket 0 using select(2)
2019-05-08 14:49:42,660 [19914] <proxy_protocol:17>: received proxy protocol V2 LOCAL command, ignoring
2019-05-08 14:49:43,600 [19915] <proxy_protocol:19>: waiting for max of 3 secs while polling socket 0 using select(2)
2019-05-08 14:49:43,600 [19915] <proxy_protocol:17>: received proxy protocol V2 TCP/IPv4 transport family (84 bytes)
2019-05-08 14:49:43,600 [19915] <proxy_protocol:3>: proxy protocol V2 TCP/IPv4 transport family sent 84 bytes, expected 12 bytes
2019-05-08 14:49:43,995 [19916] <proxy_protocol:19>: waiting for max of 3 secs while polling socket 0 using select(2)
2019-05-08 14:49:43,996 [19916] <proxy_protocol:17>: received proxy protocol V2 LOCAL command, ignoring
2019-05-08 14:49:44,198 [19917] <proxy_protocol:19>: waiting for max of 3 secs while polling socket 0 using select(2)
2019-05-08 14:49:44,198 [19917] <proxy_protocol:17>: received proxy protocol V2 LOCAL command, ignoring

[Castaglia]

I think this might be our culprit:

proxy protocol V2 TCP/IPv4 transport family sent 84 bytes, expected 12 bytes

Now to see how/why this might be occurring. The preceding:

received proxy protocol V2 LOCAL command, ignoring

might also be involved, if the module is not properly reading and ignoring the rest of that LOCAL command.

[Castaglia]

I have a local instance of HAproxy, configured for FTP load balancing, using PROXY protocol v2. I can see the TCP checks, but in the mod_proxy_protocol logs, I don't see the "LOCAL" message yet. Still trying to reproduce this behavior locally.

[Castaglia]

Hmm. I think the "LOCAL" messages a red herring. Instead, reading closely over the PROXY protocol spec, I see:

If the length specified in the PROXY protocol header indicates that additional
bytes are part of the header beyond the address information, a receiver may
choose to skip over and ignore those bytes, or attempt to interpret those
bytes.

So perhaps the AWS NLB is sending these additional bytes as part of TLVs. I will try to reproduce these.

[Castaglia]

This should now be fixed in the master branch; I'll be making a release shortly. Thanks!