HIVE-28781: Hive shouldn't disable IPv6 stack. - #5753
Conversation
| <property> | ||
| <name>templeton.controller.mr.child.opts</name> | ||
| <value> -Xmx64m -Djava.net.preferIPv4Stack=true</value> | ||
| <value> -Xmx64m -Djava.net.preferIPv4Stack=false</value> |
There was a problem hiding this comment.
Are you sure this is needs? I think the preferred IP version would be the 4. Ipv6 need to support, but it is optional. Majority of our population would use ipv4 as a default. 6 would only need on huge networks. I would not change the default behavior. Or does this change? Temporary ask a change request. But If you could explain why it is necessary I'm ok :)
There was a problem hiding this comment.
From what I saw in the web, if this flag is set to true then we disable IPv6. But does it enable it when set to false?
https://stackoverflow.com/questions/9882357/how-to-set-java-net-preferipv4stack-true-at-runtime
There was a problem hiding this comment.
The java.net.preferIPv4Stack variable name is misleading, in my opinion.
It controls if IPv6 stack is enabled or disabled.
When true, IPv6 stack is completely disabled and a java component that starts with this config won't work in IPv6-only environment.
When false, both IPv4 and IPv6 stacks are available.
There was a problem hiding this comment.
If there is no argument what is the default? I guess without argument it should be available for Ipv4 and Ipv6? Does it need for java explicitly say it is enabled or disabled?
There was a problem hiding this comment.
The default is false starting from Java 8.
Correct, there is no need to set it if we want the default behavior.
Fixed.
2b6363d to
b570e41
Compare
asf-ci-hive
added
tests pending
tests failed
and removed
tests passed
tests pending
labels
b570e41 to
f58672a
Compare
asf-ci-hive
added
tests pending
tests failed
and removed
tests failed
tests pending
labels
f58672a to
a89c19e
Compare
asf-ci-hive
added
tests pending
tests unstable
and removed
tests failed
tests pending
labels
a89c19e to
ba9585a
Compare
|
|
|
||
| # Disable ipv6 as it can cause issues | ||
| # TODO: change to -Djava.net.preferIPv4Stack=false after Hadoop adds support for IPv6 | ||
| HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true" |
There was a problem hiding this comment.
Can we assume libraries provided by Hadoop can handle IPv6?
There was a problem hiding this comment.
No, Hadoop presently doesn't support IPv6
There was a problem hiding this comment.
I guess this PR assumes the environment where Hive is deployed without YARN or HDFS. Just to clarify, I still wonder if shared or client libraries used in Hive, such as hadoop-common or File System API, correctly work with IPv6. Is the answer probably yes?
There was a problem hiding this comment.
Hadoop currently doesn't support IPv6, and I haven't found info about any hadoop libraries to support IPv6.
However, the default behavior on dual stack IPv4/IPv6 hosts is to prefer IPv4 addresses when available.
Currently, Hive can't work on IPv6-only hosts, but on dual-stack hosts it should work properly and prefer to bind to IPv4 addresses.
This is from Java Networking doc:
IPv4 / IPv6
java.net.preferIPv4Stack (default: false)
If IPv6 is available on the operating system the underlying native socket will be, by default, an IPv6 socket which lets applications connect to, and accept connections from, both IPv4 and IPv6 hosts. However, in the case an application would rather use IPv4 only sockets, then this property can be set to true. The implication is that it will not be possible for the application to communicate with IPv6 only hosts.java.net.preferIPv6Addresses (default: false)
When dealing with a host which has both IPv4 and IPv6 addresses, and if IPv6 is available on the operating system, the default behavior is to prefer using IPv4 addresses over IPv6 ones. This is to ensure backward compatibility, for example applications that depend on the representation of an IPv4 address (e.g. 192.168.1.1). This property can be set to true to change that preference and use IPv6 addresses over IPv4 ones where possible.
When java.net.preferIPv4Stack=false and java.net.preferIPv6Addresses=false, both IPv4 and IPv6 are enabled, but Java prefers to use IPv4 addresses unless it can find only IPv6 (if there is only IPv6 address for a DNS name, it will use IPv6, but on properly configured system it won't happen)
There was a problem hiding this comment.
I checked the documents of Java 8 and 17 just in case.
There was a problem hiding this comment.
In that sense, don't we have to overwrite HADOOP_OPTS here or should we configure java.net.preferIPv6Addresses=false here?
ayushtkn
left a comment
ayushtkn
left a comment
There was a problem hiding this comment.
@difin AFAIK Djava.net.preferIPv4Stack=true forces Java to use IPv4 over IPv6, even when IPv6 is available.
As of now Hadoop doesn't support IPV6 & moreover we didn't even test Hive fully to see if it works completely f9 with IPV6.
Moreover even if it works f9, don't you think this would be a behaviour change for Hive, like post upgrade suddenly the Hive machines will start preferring IPv6, I believe this would break compat, we should let users decide if they want to go with IPV6 or not. We can just claim from Hive, yes, IPV6 is now supported in Hive & to enable you can remove these flags from the launch scripts or so
|
Hi @ayushtkn ,
The only time Java would prefer to use IPv6 with these settings is when a DNS name has only IPv6 mapping. |
|
Sounds good then, if the CI is green on Dual stack machine, Then it is good with me |
Hi @ayushtkn, I executed all unit tests and q-tests on my local Linux VM, and there were no IPv6-related failures. |
6 participants
What changes were proposed in this pull request?
Hive used to disable IPv6 by setting this config in some services and in maven surefire plugin (for tests):
java.net.preferIPv4Stack=true. When this config istrue, IPv6 is completely disabled. This should be changed tofalseto support dual stack, IPv4 and IPv6.Why are the changes needed?
To add support for IPv6 in Hive.
Does this PR introduce any user-facing change?
No
Is the change a dependency upgrade?
No
How was this patch tested?
Hive pre-commit tests.