Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Contour TCP support for LDAP applications and traffic routing #4306

Closed
gnanirahulnutakki opened this issue Jan 27, 2022 · 10 comments
Closed
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/needs-triage Indicates that an issue needs to be triaged by a project contributor. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.

Comments

@gnanirahulnutakki
Copy link

Please describe the problem you have
[A clear, concise, description of the problem you are facing. What is the problem that feature X would solve for you?]
Hi Guys, I was recently assigned a task which requires me to use Contour as ingress. Ours is a LDAP application per say and I need to use ingress for TCP traffic. Basically support for L4.

I asked this question on slack and was told this by steve sloka

" Hey, Contour supports TCP over TLS in our HTTPProxy CRD (https://projectcontour.io/docs/v1.19.1/config/api/#projectcontour.io/v1.TCPProxy)."

"Yup that’s in there but doesn’t mean it can’t change in the future. The idea now was to focus on L7 and get that right. There can be a lot to L4 load balancers which others in the community have focused on, however, I agree that there are places for L4 in an edge proxy that we’ve also see from other users."

"At the moment, you can do L4 over TLS via the TCPProxy details I mentioned earlier and we can chat about the L4 additions. Could you open an issue on that to track it better?"

I am in the process to creating this manifest using TCPproxy, but I would really appreciate an official documentation or assistance on this

My questions are:

Does contour support TCP ingress?
How reliable and accurate is it to use contour for TCP
Are there are examples where contour was deployed for kubernetes ingress manifests for TCP
I would really appreciate some guidance here

@gnanirahulnutakki gnanirahulnutakki added kind/feature Categorizes issue or PR as related to a new feature. lifecycle/needs-triage Indicates that an issue needs to be triaged by a project contributor. labels Jan 27, 2022
@youngnick
Copy link
Member

youngnick commented Jan 28, 2022

Currently, Contour supports TCP ingress for TLS-wrapped protocols that support SNI only.

This is because Contour currently only supports having Envoy listen on two ports, insecure and secure. The insecure port is expected to receive HTTP traffic, and the secure port is expected to received TLS traffic. Contour configures Envoy to use the SNI hostname that's send as part of the TLS handshake to distinguish between different routing chains.

In order to serve plain TCP ports, Contour will need to be able to open extra ports on Envoy. This will require something like the design in #3263. Once that happens, Contour will support a way to specify what extra ports to open on Envoy's listen address. For them to pass TCP traffic, they'll need to be dedicated to a single protocol.

This does have the further complication that without higher-layer protocol support, all the TCP connections will appear to come from Envoy. Contour does support configuring Envoy to use the PROXY protocol which gets around this problem, but this then also requires that the backend service speak the PROXY protocol too.

To sum up this part of the answer, we should really have called TCPProxy TLSProxy instead, that's a more correct name for it.

We can (and probably will) add support for adding additional listeners in the future, which will mean that it will be possible to configure Contour to proxy TCP traffic. Because Contour usually requires at least one upstream load balancer (because its Envoys run in-cluster), when we do that, it won't be possible to have the original remote client address be visible at Layer 4 without the use of PROXY protocol.

PROXY protocol requires:

  • The upstream from Contour LB to have PROXY protocol configured.
  • The listener on Contour's Envoys to have PROXY protocol configured.
  • The backend service to support unwrapping PROXY protocol to retrieve the original client IP.

Of course, if you are using a Layer 7 protocol that has some other way to retrieve the client IP, or the client IP doesn't matter, then what we will be able to add will work.

To answer the other questions:

  • If you are using TLS and SNI to forward TCP traffic, as far as I know it's as reliable and accurate as forwarding TLS traffic that contains HTTP (which is to say, very).
  • I don't think we have examples for this at the moment, because of the complexities I explained above.

To answer your question in the shortest, and most likely least satisfying, way possible: Does Contour support TCP forwarding? It depends on what you're trying to forward.

Edit: After writing this whole comment, I realized that you said that you mean LDAP in the title. 🤦

As I implied above, Contour should be able to forward LDAP as long as it's LDAPS, terminated at the backend system. To be honest, I'm not sure if or how well terminating LDAPS at Contour and forwarding LDAP to the backend would work, even aside from losing the client IP.

@youngnick
Copy link
Member

Also, this issue is probably covered in #3086 a bit, we've also had #4222 recently which was a similar ask.

@gnanirahulnutakki
Copy link
Author

I was trying to create a httpproxy with tcpproxy in this way

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
name: vds-httpproxy
spec:
virtualhost:
fqdn: tcp-app.example.com
tcpproxy:
services:
- name: http2-fid1
port: 7070
- name: https-fid1
port: 7171
- name: http2-api1
port: 8089
- name: https-api1
port: 8090
- name: http2-admin1
port: 9100
- name: https-admin1
port: 9101
- name: ldap
port: 2389
- name: ldaps
port: 2636

So according to what you explained this probably might or might not work

@tsaarni
Copy link
Member

tsaarni commented Jan 28, 2022

It will not be very convenient but you could try something like this

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: example
  namespace: default
spec:
  virtualhost:
    fqdn: ldap-app.example.com
    tls:
      passthrough: true
  tcpproxy:
    services:
    - name: ldaps
      port: 2636

You would be able to use LDAPS only like @youngnick wrote and your LDAP client would need to use hostname ldap-app.example.com and HTTPS port as the LDAP connection URL. You would need to dedicate a hostname for LDAP traffic and you cannot have HTTP served on that same FQDN. These may be quite strict restrictions, but that is because Contour does not currently support port based TCP proxying.

@gnanirahulnutakki
Copy link
Author

gnanirahulnutakki commented Jan 28, 2022

I tried with the following manifest file but no luck, The TLS handshake does not seems to get through

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  namespace: fid-demo
  name: fid-contour
spec:
  virtualhost:
    fqdn: tcp-fid.example.com
    tls:
      passthrough: true
  tcpproxy:
    services:
    - name: fid
      port: 2636

LOGS FROM APACHE DIRECTORY STUDIO:

The authentication failed
 -  ERR_04120_TLS_HANDSHAKE_ERROR The TLS handshake failed, reason: Unspecified: Improper close state: Status = OK HandshakeStatus = NEED_WRAP
 - No Connection
  org.apache.directory.studio.connection.core.io.StudioLdapException:  ERR_04120_TLS_HANDSHAKE_ERROR The TLS handshake failed, reason: Unspecified: Improper close state: Status = OK HandshakeStatus = NEED_WRAP
bytesConsumed = 0 bytesProduced = 7 sequenceNumber = 1
	at org.apache.directory.studio.connection.core.io.api.DirectoryApiConnectionWrapper.toStudioLdapException(DirectoryApiConnectionWrapper.java:1350)
	at org.apache.directory.studio.connection.core.io.api.DirectoryApiConnectionWrapper.access$2(DirectoryApiConnectionWrapper.java:1342)
	at org.apache.directory.studio.connection.core.io.api.DirectoryApiConnectionWrapper$1.run(DirectoryApiConnectionWrapper.java:258)
	at org.apache.directory.studio.connection.core.io.api.DirectoryApiConnectionWrapper.runAndMonitor(DirectoryApiConnectionWrapper.java:1261)
	at org.apache.directory.studio.connection.core.io.api.DirectoryApiConnectionWrapper.doConnect(DirectoryApiConnectionWrapper.java:280)
	at org.apache.directory.studio.connection.core.io.api.DirectoryApiConnectionWrapper.connect(DirectoryApiConnectionWrapper.java:144)
	at org.apache.directory.studio.connection.core.jobs.CheckBindRunnable.run(CheckBindRunnable.java:78)
	at org.apache.directory.studio.connection.ui.RunnableContextRunner$1.run(RunnableContextRunner.java:140)
	at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:122)
Caused by: org.apache.directory.api.ldap.model.exception.LdapTlsHandshakeException: ERR_04120_TLS_HANDSHAKE_ERROR The TLS handshake failed, reason: Unspecified: Improper close state: Status = OK HandshakeStatus = NEED_WRAP
bytesConsumed = 0 bytesProduced = 7 sequenceNumber = 1
	at org.apache.directory.ldap.client.api.LdapNetworkConnection.checkSecured(LdapNetworkConnection.java:824)
	at org.apache.directory.ldap.client.api.LdapNetworkConnection.connect(LdapNetworkConnection.java:984)
	at org.apache.directory.studio.connection.core.io.api.DirectoryApiConnectionWrapper$1.run(DirectoryApiConnectionWrapper.java:227)
	... 6 more
Caused by: javax.net.ssl.SSLException: Improper close state: Status = OK HandshakeStatus = NEED_WRAP
bytesConsumed = 0 bytesProduced = 7 sequenceNumber = 1
	at org.apache.mina.filter.ssl.SslHandler.closeOutbound(SslHandler.java:497)
	at org.apache.mina.filter.ssl.SslFilter.initiateClosure(SslFilter.java:762)
	at org.apache.mina.filter.ssl.SslFilter.filterClose(SslFilter.java:693)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterClose(DefaultIoFilterChain.java:776)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1600(DefaultIoFilterChain.java:49)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterClose(DefaultIoFilterChain.java:1155)
	at org.apache.mina.core.filterchain.IoFilterAdapter.filterClose(IoFilterAdapter.java:146)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterClose(DefaultIoFilterChain.java:776)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1600(DefaultIoFilterChain.java:49)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.filterClose(DefaultIoFilterChain.java:1155)
	at org.apache.mina.core.filterchain.IoFilterAdapter.filterClose(IoFilterAdapter.java:146)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callPreviousFilterClose(DefaultIoFilterChain.java:776)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireFilterClose(DefaultIoFilterChain.java:769)
	at org.apache.mina.core.session.AbstractIoSession.closeNow(AbstractIoSession.java:353)
	at org.apache.directory.ldap.client.api.LdapNetworkConnection.inputClosed(LdapNetworkConnection.java:4763)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.inputClosed(DefaultIoFilterChain.java:997)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextInputClosed(DefaultIoFilterChain.java:735)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:49)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.inputClosed(DefaultIoFilterChain.java:1119)
	at org.apache.mina.core.filterchain.IoFilterAdapter.inputClosed(IoFilterAdapter.java:154)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextInputClosed(DefaultIoFilterChain.java:735)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:49)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.inputClosed(DefaultIoFilterChain.java:1119)
	at org.apache.mina.core.filterchain.IoFilterAdapter.inputClosed(IoFilterAdapter.java:154)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextInputClosed(DefaultIoFilterChain.java:735)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:49)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.inputClosed(DefaultIoFilterChain.java:1119)
	at org.apache.mina.core.filterchain.IoFilterAdapter.inputClosed(IoFilterAdapter.java:154)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextInputClosed(DefaultIoFilterChain.java:735)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireInputClosed(DefaultIoFilterChain.java:728)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:556)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$1200(AbstractPollingIoProcessor.java:68)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1222)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.process(AbstractPollingIoProcessor.java:1211)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:683)
	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

   ERR_04120_TLS_HANDSHAKE_ERROR The TLS handshake failed, reason: Unspecified: Improper close state: Status = OK HandshakeStatus = NEED_WRAP
bytesConsumed = 0 bytesProduced = 7 sequenceNumber = 1
  java.lang.Exception: No Connection
	at org.apache.directory.studio.connection.core.io.api.DirectoryApiConnectionWrapper.doBind(DirectoryApiConnectionWrapper.java:497)
	at org.apache.directory.studio.connection.core.io.api.DirectoryApiConnectionWrapper.bind(DirectoryApiConnectionWrapper.java:323)
	at org.apache.directory.studio.connection.core.jobs.CheckBindRunnable.run(CheckBindRunnable.java:79)
	at org.apache.directory.studio.connection.ui.RunnableContextRunner$1.run(RunnableContextRunner.java:140)
	at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:122)

  No Connection

@tsaarni
Copy link
Member

tsaarni commented Jan 29, 2022

Could it be that you used Contour's HTTP port in Apache Directory Studio's connection settings instead of HTTPS port?

I tried that config now myself and it works with some caveats:

  • Java based LDAP clients work (used this code snippet as client), but openldap based clients such as ldapsearch do not work, since it does not set SNI in the TLS handshake. You can observe that with Wireshark. In that case Envoy will refuse to passthrough the TLS connection. Like @youngnick explained above - Envoy does not know which backend service to forward it since TCP proxying happens on the basis of hostname in TLS SNI extension.
  • Client will by default verify that the server certificate has SubjectAltName field with value that matches with the hostname in the LDAP connection string. If there is a mismatch, the client will abruptly close the connection during handshake.

@gnanirahulnutakki
Copy link
Author

gnanirahulnutakki commented Feb 1, 2022

I changed the fqdn to match the certificate name in my LDAP application server so manifest looks like this now

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  namespace: fid-demo
  name: fid-contour
spec:
  virtualhost:
    fqdn: fid-0.fid.fid-demo.svc.cluster.local
    tls:
      passthrough: true
  tcpproxy:
    services:
    - name: fid
      port: 2636

And checked to see if it would work,

MacBook-Pro ~ % openssl s_client -debug -connect ldap.fid-0.fid.fid-demo.svc.cluster.local:443
CONNECTED(00000006)
write to 0x6000000c2a00 [0x7fdcc600ca03] (196 bytes => 196 (0xC4))
0000 - 16 03 01 00 bf 01 00 00-bb 03 03 67 56 fb da b5   ...........gV...
0010 - ff 59 1f 4f f6 d7 9f c8-d2 97 37 e8 ed 20 fe 84   .Y.O......7.. ..
0020 - 99 31 ae 80 48 dd e9 5d-60 70 15 00 00 5c c0 30   .1..H..]`p...\.0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 9f 00 6b 00 39   .,.(.$.......k.9
0040 - cc a9 cc a8 cc aa ff 85-00 c4 00 88 00 81 00 9d   ................
0050 - 00 3d 00 35 00 c0 00 84-c0 2f c0 2b c0 27 c0 23   .=.5...../.+.'.#
0060 - c0 13 c0 09 00 9e 00 67-00 33 00 be 00 45 00 9c   .......g.3...E..
0070 - 00 3c 00 2f 00 ba 00 41-c0 11 c0 07 00 05 00 04   .<./...A........
0080 - c0 12 c0 08 00 16 00 0a-00 ff 01 00 00 36 00 0b   .............6..
0090 - 00 02 01 00 00 0a 00 08-00 06 00 1d 00 17 00 18   ................
00a0 - 00 23 00 00 00 0d 00 1c-00 1a 06 01 06 03 ef ef   .#..............
00b0 - 05 01 05 03 04 01 04 03-ee ee ed ed 03 01 03 03   ................
00c0 - 02 01 02 03                                       ....
read from 0x6000000c2a00 [0x7fdcc6008803] (5 bytes => 0 (0x0))
4517766828:error:140040E5:SSL routines:CONNECT_CR_SRVR_HELLO:ssl handshake failure:/System/Volumes/Data/SWE/macOS/BuildRoots/5b2e67f8af/Library/Caches/com.apple.xbs/Sources/libressl/libressl-75.60.3/libressl-2.8/ssl/ssl_pkt.c:585:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Start Time: 1643647382
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

Even this does not seems to work. Any ideas?

@tsaarni
Copy link
Member

tsaarni commented Feb 1, 2022

Testing with openssl s_client is a good idea, but try adding also -servername <FQDN> to the command line since s_client does not add SNI by default.

@github-actions
Copy link

The Contour project currently lacks enough contributors to adequately respond to all Issues.

This bot triages Issues according to the following rules:

  • After 60d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, the Issue is closed

You can:

  • Mark this Issue as fresh by commenting
  • Close this Issue
  • Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

@github-actions github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 15, 2022
@github-actions
Copy link

The Contour project currently lacks enough contributors to adequately respond to all Issues.

This bot triages Issues according to the following rules:

  • After 60d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, the Issue is closed

You can:

  • Mark this Issue as fresh by commenting
  • Close this Issue
  • Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Dec 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/needs-triage Indicates that an issue needs to be triaged by a project contributor. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.
Projects
None yet
Development

No branches or pull requests

3 participants