Ubuntu + 127.0.1.1 IP address results in java.net.ConnectException: Connection refused

[minfrin]

Elasticsearch version (bin/elasticsearch --version): 6.8.5

Plugins installed: [analysis-icu]

JVM version (java -version): OpenJDK 64-Bit Server VM (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3, mixed mode, sharing)

OS version (uname -a if on a Unix-like system): Linux els02 4.15.0-1056-aws #58-Ubuntu SMP Tue Nov 26 15:14:34 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

On Ubuntu systems, the system is configured in such a way that the fully qualified DNS name of the Ubuntu machine is overridden in /etc/hosts to be "127.0.1.1", a hack to point box wide requests to localhost.

When attempting to form a cluster with an Ubuntu box, the remote node tells the local node that it's IP address is 127.0.1.1, the local node then tries to connect to the remote node on 127.0.1.1 instead of the proper DNS address, and the connection fails.

Steps to reproduce:

1. Configure a cluster as follows, try pass the proper DNS name of the cluster as the transport.publish_host.

cluster.name: x-jm-unstable-black
network.host: x-jm-unstable-black-els01.example.com
transport.bind_host: _site_,_local_
transport.publish_host: x-jm-unstable-black-els01.example.com

2. Observe that the IP address has become 127.0.1.1, resulting in connection refused.
3. The DNS name is important, as it is verified using "full" certificate verification. It appears that it is not possible to configure a secure "full" verified cluster on an Ubuntu machine, due to Ubuntu's misuse of the 127.0.1.1 address.

Provide logs (if relevant):

Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: x-jm-unstable-black-els02.example.com/127.0.1.1:9300
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
        ... 6 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
        ... 6 more

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Network)

[original-brownbear]

I think this one is worth some discussion before making any changes. The problem here is that out DiscoveryNode instances contain both an IP and a hostname and currently all implementations of org.elasticsearch.transport.TcpTransport#initiateChannel will then use the IP found in the DiscoveryNode to establish connections based on the cluster state. Now if you have inconsistent DNS resolution in your cluster this will break down and connecting by hostname would work.

I think you could make an argument that we should always be using the advertised host name for connections and not the IP since that is more in line with how the publish host setting is documented. That kind of change would have wide-spread consequences though (this could obviously break a number of other situations where there are problems with DNS resolution on some nodes and is a significant behavior change). -> added "team-discuss" for now.

[original-brownbear]

We discussed this during distributed area sync today and couldn't see a strong reason to change the behaviour here. Arguments regarding expected and consistent behaviour could be made in favour of changing this but the change would also break backwards compatibility and might have non-trivial side-effects. Leaving this open a while though to gauge public interest on the topic.

[minfrin]

The strong reason to change it is for security compliance.

No user of elasticsearch can use replication and tick the box marked "my data is encrypted in transit" at the same time due to this bug.

The box marked "my data is encrypted in transit" appears on many request for proposals related to cloud services, and saying "no" is obviously a competitive disadvantage.

[original-brownbear]

No user of elasticsearch can use replication and tick the box marked "my data is encrypted in transit" at the same time due to this bug.

But this is only true for systems that have inconsistent DNS resolution for the publish host across nodes is it? And it seems like this can relatively easily (obviously depends on the specific circumstances) be worked around by using a different publish host that resolves consistently across nodes?

[minfrin]

But this is only true for systems that have inconsistent DNS resolution for the publish host across nodes is it?

This is all of Ubuntu, and I'm assuming Debian and it's derivatives.

And it seems like this can relatively easily

If you're working with AWS instances no, you take what AWS gives you. Gone are the days when machines are set up manually as the norm, or you have options to hack around at the way systems work.

[tvernum]

No user of elasticsearch can use replication and tick the box marked "my data is encrypted in transit" at the same time due to this bug

Can you explain this in a little more detail?

This issue prevents using a FQDN for the publish_host in some circumstances but it doesn't prevent you from having encrypted traffic because

1. In many cases it would still possible to use certificates with IP based SubjAltNames.
2. Even if you need to turn off hostname verification (that is, switch ssl.verification_mode from full to certificate) the data is still encrypted in transit.

Am I missing something?

[original-brownbear]

Unrelated note to the above: It was just raised that this issue prevents using publish_host in an environment (Kubernetes) where nodes might see their IPs change during configuration changes. In those cases rejoining the cluster becomes impossible because the IP in the DisocveryNode is used for re-connect instead of the published host. This might be another + to eventually fixing this.

[minfrin]

No user of elasticsearch can use replication and tick the box marked "my data is encrypted in transit" at the same time due to this bug

Can you explain this in a little more detail?

Encryption is meaningless without signing.

To put this another way, installing a huge strongroom door is pointless if you wedge the door open and allow anyone through the door without checking.

Signing is this case means checking whether the name of the digital certificate matches the name of the machine you connected to. If the name does not match, you've connected to the wrong machine, and the connection must fail.

This issue prevents using a FQDN for the publish_host in some circumstances but it doesn't prevent you from having encrypted traffic because

1. In many cases it would still possible to use certificates with IP based SubjAltNames.

It is virtually never possible to use IP based subjectAltNames, because in most cases IP addresses are allocated via DHCP and could change at any time (IPv4) or are allocated randomly (IPv6), and these changes happen outside the control of the system that issues certificates.

Sure, in rare exceptions a system might be deployed by hand and have IPs assigned by hand. This is the exception, not the rule.

2. Even if you need to turn off hostname verification (that is, switch ssl.verification_mode from full to certificate) the data is still encrypted in transit.

The data is no longer encrypted in any meaningful way, no.

As soon as you turn hostname verification off, you're allowing anyone with a certificate to connect. This means that an ES containing client A's data will successfully connect to an ES containing client B's data, and this is the equivalent of leaving the strongroom door open without checking who goes through.

Am I missing something?

The way TLS works is long established. Networks, clouds, orchestration tools are not designed to accommodate systems like ES that don't follow the rules.

[tvernum]

Your words were "No user". What you really meant was "no user who has a setup similar to mine".

There are plenty of cases where IP addresses are an option for cluster formation, and there are plenty of cases where DNS resolution is consistent across nodes.

For any change in behaviour like this we need to assess the impact of the issue vs the impact of changing it. Blanket statements that assume that a particular configuration is universal muddy those waters. It is easier justify a particular change when the impact is clear and measured. That is why I asked for clarification of your statements - they overstated the extent of the issue but we need clarity and accuracy.

As soon as you turn hostname verification off, you're allowing anyone with a certificate to connect. This means that an ES containing client A's data will successfully connect to an ES containing client B's data, and this is the equivalent of leaving the strongroom door open without checking who goes through.

That isn't true if your trust domain for the cluster is a closed group.

Our recommended setup for TLS on the transport port is to have a CA per cluster. If you do that, then you never allow a node from a separate cluster to join your cluster because its certificate will not chain back to a trust anchor in the local cluster. In the case hostname verification offers very little added protection.

Hostname verification exists in HTTPS because a requirement there is for clients to be able to connect to servers that they have never communicated with before and know whether they have connected correctly. It's a solution to the decentralised nature of the web. You could solve the same problem with certificate pinning, but doing so requires that every client has a copy of the certificate for every server it ever wants to connect to. That's not a workable solution on the open internet.

There is no counterpart to hostname verification for TLS client authentication. In that model the client's certificate is trusted because it is signed by a specific party (not just "a CA", but a very specific CA which is trusted to establish client identity).

The issues that need to be solve with cluster formation are closer to the client authentication model than the server trust model. The issue you want to protect against are outside nodes trying to connect to your cluster. You cannot solve that with hostname verification because:

1. the initial connection will be a client connection, so it's not possible to do hostname verification
2. when we establish the inverse connection, the hostname is not likely to be the point of failure. It is quite likely that the new node has a certificate with a matching hostname. If they control the host that they are running the node on, then they can probably get a CA to issue them a certificate for it.

Hostname verification is only useful if the set of hostnames that your cluster accepts as members is a static group. In that case you can reasonably connect "which hosts do I want to connect to" and "are you the host you claim to be" and form a cluster with trusted membership.
But Elasticsearch is intended to be able to scale out and add new nodes without requiring a prerequisite step of explicitly adding that new hostname to the list of trusted members. For that to work you cannot rely on hostname verification because the existing cluster membership isn't in control of the hosts that attempt to join. Verifying that someone is who they claim to be doesn't help if you have no reason to trust that person.

To use your analogy, I can prove my identity to you by showing you my identity documents. That doesn't mean you should open the strongroom door to me.
What you need is some authorization from the owner of the strongroom that states that I should be granted access.
That can come in 3 different forms:

1. A preexisting list of who is allowed access to the room
2. A signed document that says "the holder of this document is permitted access"
3. A signed document that says "the person named .... is permitted access"

xpack.security.transport.ssl.verification_mode: certificate works like option 2. It requires that you have a CA that only issues certificates to nodes that are authorized to be members of the cluster, and that the private keys on those nodes are kept secure. If you do that, you have data that is encrypted in transit without issue.

If you use a general purpose CA, and rely on hostname verification, then you are trying to solve this via option 1, except ES doesn't provide that preexisting list, so the solution falls down.

[minfrin]

Your words were "No user". What you really meant was "no user who has a setup similar to mine".

DNS, the /etc/hosts file, and other similar naming systems are all decades old, well described and well documented. By sending the internal IP address of a node you break all these naming systems.

Obviously people can hack around the problem in various ugly ways, like we did, making our project unnecessarily late and burning up budget that could be better spent elsewhere.

Or people can just switch off security and ignore the problem.

Instead of arguing about it for ten months, just fix it. Put it behind a feature switch if you have to. All you need do is use the DNS lookup instead of the passed IP address. I estimate this kind of fix would take 4 hours max, including fudge factor.

[original-brownbear]

I'm removing >team-discuss here. We discussed this and decided that making the connects work based on the published host instead of a pre-resolved IP is the way to go here (taking reasonable BwC steps along the way obviously).

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)