New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

private-net: should forwarded ports work on localhost? #1256

Closed
steveeJ opened this Issue Aug 7, 2015 · 10 comments

Comments

Projects
None yet
6 participants
@steveeJ
Contributor

steveeJ commented Aug 7, 2015

Currently ports that are forwarded to containers are not accessible on the host via localhost:forwarded-port. It would be possible to do enable that and the technique is generally called NAT loopback.

It's a matter of choice if we want to change this or not so I want to bring it up for discussion.

Personally I'm unsure if we need it, but it could help in situations when service discovery is not there and the host expects a service on localhost while the container is not supposed to share the host netns.

@alban

This comment has been minimized.

Show comment
Hide comment
@alban

alban Aug 7, 2015

Member

Currently ports that are forwarded to containers are not accessible on the host via localhost:forwarded-port.

Do you have an example of commands to show what is not available? I tried the following:

sudo ./build-rkt-0.7.0+git/bin/rkt --local --insecure-skip-verify run --mds-register=false --volume volume-data,kind=host,source=/tmp/redis docker://redis

It exposes the port 6379:

        "ports": [
            {
                "name": "6379-tcp",
                "protocol": "tcp",
                "port": 6379,
                "count": 1,
                "socketActivated": false
            }
        ]

And then on the host: nc localhost 6379 and I could talk successfully to redis in the pod.

I also tried to setup a ssh forwarded port with ssh -L 6378:localhost:6379 localhost and to connect with nc localhost 6378 and it worked as well.

Member

alban commented Aug 7, 2015

Currently ports that are forwarded to containers are not accessible on the host via localhost:forwarded-port.

Do you have an example of commands to show what is not available? I tried the following:

sudo ./build-rkt-0.7.0+git/bin/rkt --local --insecure-skip-verify run --mds-register=false --volume volume-data,kind=host,source=/tmp/redis docker://redis

It exposes the port 6379:

        "ports": [
            {
                "name": "6379-tcp",
                "protocol": "tcp",
                "port": 6379,
                "count": 1,
                "socketActivated": false
            }
        ]

And then on the host: nc localhost 6379 and I could talk successfully to redis in the pod.

I also tried to setup a ssh forwarded port with ssh -L 6378:localhost:6379 localhost and to connect with nc localhost 6378 and it worked as well.

@alban

This comment has been minimized.

Show comment
Hide comment
@alban

alban Aug 7, 2015

Member

Oh sorry, I should not use the defaut network namespace for testing this. So the full command I tried is:

sudo ./build-rkt-0.7.0+git/bin/rkt --local --insecure-skip-verify run --mds-register=false --volume volume-data,kind=host,source=/tmp/redis --private-net=default --port=6379-tcp:8080 docker://redis

And then I have the behavior you described when trying with nc 192.168.0.13 8080 (it works) and nc localhost 8080 (it does not work).

Do you know if kube-proxy uses localhost to connect to a kubernetes endpoint?

Member

alban commented Aug 7, 2015

Oh sorry, I should not use the defaut network namespace for testing this. So the full command I tried is:

sudo ./build-rkt-0.7.0+git/bin/rkt --local --insecure-skip-verify run --mds-register=false --volume volume-data,kind=host,source=/tmp/redis --private-net=default --port=6379-tcp:8080 docker://redis

And then I have the behavior you described when trying with nc 192.168.0.13 8080 (it works) and nc localhost 8080 (it does not work).

Do you know if kube-proxy uses localhost to connect to a kubernetes endpoint?

@steveeJ steveeJ changed the title from should forwarded ports work on localhost? to private-net: should forwarded ports work on localhost? Aug 7, 2015

@jonboulle jonboulle added this to the v0.12.0 milestone Nov 6, 2015

@alban

This comment has been minimized.

Show comment
Hide comment
@alban
Member

alban commented Nov 25, 2015

@steveeJ

This comment has been minimized.

Show comment
Hide comment
@steveeJ

steveeJ Nov 25, 2015

Contributor

I'm okay with the current behavior as I would not expect it to behave differently. I brought up the question to see what the general assumption would be. We have to make the decision if we leave it as is (my vote) or if we enable NAT loopback.

Contributor

steveeJ commented Nov 25, 2015

I'm okay with the current behavior as I would not expect it to behave differently. I brought up the question to see what the general assumption would be. We have to make the decision if we leave it as is (my vote) or if we enable NAT loopback.

@jonboulle jonboulle modified the milestones: v0.13.0, v0.12.0 Nov 25, 2015

@jonboulle

This comment has been minimized.

Show comment
Hide comment
@jonboulle

jonboulle Nov 25, 2015

Contributor

It seems pretty unintuitive to me. Can you help explain your vote?

Contributor

jonboulle commented Nov 25, 2015

It seems pretty unintuitive to me. Can you help explain your vote?

@blalor

This comment has been minimized.

Show comment
Hide comment
@blalor

blalor Dec 11, 2015

I was bitten by this yesterday. I think the principle of least surprise would dictate that the forwarded port is also reachable via localhost. I'd be fine with it being a runtime option (to --port or something). But in the case of CoreOS where everything must run as a container, it really would simplify connecting to services if I didn't first have to look up the local host's IP, or depend on $HOSTNAME or $( hostname ).

blalor commented Dec 11, 2015

I was bitten by this yesterday. I think the principle of least surprise would dictate that the forwarded port is also reachable via localhost. I'd be fine with it being a runtime option (to --port or something). But in the case of CoreOS where everything must run as a container, it really would simplify connecting to services if I didn't first have to look up the local host's IP, or depend on $HOSTNAME or $( hostname ).

@yorickvP

This comment has been minimized.

Show comment
Hide comment
@yorickvP

yorickvP Jan 3, 2016

I was debugging this for a few hours yesterday, trying to reach a specific container from the host is really hard without doing this, I don't think there's any disadvantage to making it work on localhost.

yorickvP commented Jan 3, 2016

I was debugging this for a few hours yesterday, trying to reach a specific container from the host is really hard without doing this, I don't think there's any disadvantage to making it work on localhost.

@steveeJ

This comment has been minimized.

Show comment
Hide comment
@steveeJ

steveeJ Jan 6, 2016

Contributor

@blalor @yorickvP thanks for your feedback on this, supporting @jonboulle's intuition. This topic has come up often enough lately to convince me that we should have NAT loopback.

Contributor

steveeJ commented Jan 6, 2016

@blalor @yorickvP thanks for your feedback on this, supporting @jonboulle's intuition. This topic has come up often enough lately to convince me that we should have NAT loopback.

@steveeJ steveeJ modified the milestones: v0.16.0, v0.15.0 Jan 6, 2016

@iaguis

This comment has been minimized.

Show comment
Hide comment
@iaguis

iaguis Jan 19, 2016

Member

@steveeJ Any chance we can get this for the next release?

Member

iaguis commented Jan 19, 2016

@steveeJ Any chance we can get this for the next release?

@alban alban modified the milestones: v1.0.0, v0.16.0 Jan 20, 2016

@steveeJ

This comment has been minimized.

Show comment
Hide comment
@steveeJ

steveeJ Jan 22, 2016

Contributor

@iaguis it looks like we're having trouble getting this in at all.

Rewriting localhost:forwarded-port -> container-IP:forwarded-port doesn't work by design with iptables.

As an example, the following rule

-A OUTPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 1024 -j DNAT --to-destination 172.16.28.2:1024

Will be matched, but it will leave the package destination unchanged, as can be seen in the TRACE output:

Jan 22 21:47:05 steveej-laptop kernel: TRACE: raw:OUTPUT:policy:2 IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=55893 DF PROTO=TCP SPT=39992 DPT=1024 SEQ=1195156359 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A029579D20000000001030307) UID=1000 GID=100 
Jan 22 21:47:05 steveej-laptop kernel: TRACE: mangle:OUTPUT:policy:1 IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=55893 DF PROTO=TCP SPT=39992 DPT=1024 SEQ=1195156359 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A029579D20000000001030307) UID=1000 GID=100 
Jan 22 21:47:05 steveej-laptop kernel: TRACE: nat:OUTPUT:rule:1 IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=55893 DF PROTO=TCP SPT=39992 DPT=1024 SEQ=1195156359 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A029579D20000000001030307) UID=1000 GID=100

Any suggestions are welcome!


UPDATE: Fortunately I was wrong with my conclusion from last evening. I'll prepare a PR with a proposal.

Contributor

steveeJ commented Jan 22, 2016

@iaguis it looks like we're having trouble getting this in at all.

Rewriting localhost:forwarded-port -> container-IP:forwarded-port doesn't work by design with iptables.

As an example, the following rule

-A OUTPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 1024 -j DNAT --to-destination 172.16.28.2:1024

Will be matched, but it will leave the package destination unchanged, as can be seen in the TRACE output:

Jan 22 21:47:05 steveej-laptop kernel: TRACE: raw:OUTPUT:policy:2 IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=55893 DF PROTO=TCP SPT=39992 DPT=1024 SEQ=1195156359 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A029579D20000000001030307) UID=1000 GID=100 
Jan 22 21:47:05 steveej-laptop kernel: TRACE: mangle:OUTPUT:policy:1 IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=55893 DF PROTO=TCP SPT=39992 DPT=1024 SEQ=1195156359 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A029579D20000000001030307) UID=1000 GID=100 
Jan 22 21:47:05 steveej-laptop kernel: TRACE: nat:OUTPUT:rule:1 IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=55893 DF PROTO=TCP SPT=39992 DPT=1024 SEQ=1195156359 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A029579D20000000001030307) UID=1000 GID=100

Any suggestions are welcome!


UPDATE: Fortunately I was wrong with my conclusion from last evening. I'll prepare a PR with a proposal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment