HttpDownloader fails to download on IPv6 network

[damienmg]

TL;DR: We failed to demo tensorflow build at fosdem on the recent infra because IPv6 only.

[abergmeier-dsfishlabs]

_{that is embarassing} SCNR

[philwo]

I'll try to reproduce and fix on Monday. Currently don't have an IPv6 network available for testing.

[philwo]

I've built a little HttpURLConnection playground to rule out that this is something Bazel (launcher) specific and tried downloading a file from a dual-stack host (bazel-mirror.storage.googleapis.com) with either IPv4/IPv6 enabled or IPv6-only.

I verified that I can download the file with "curl" in all network setups.

Even though this is supposed to "just work", it horribly fails with Java for all IPv6 related scenarios:

Trying to download from IPv4 host with dual-stack connectivity: Works.
Trying to download from IPv6 host with dual-stack connectivity: "No route to host" exception.
Trying to download from IPv4 host with IPv6-only: "Network unreachable" exception (but HttpURLConnection doesn't retry with IPv6)
Trying to download from IPv4 host with IPv6-only: "No route to host" exception.

I tried to build a manual fallback logic, but as one can't override the "Host" header in an HttpURLConnection for security reasons (AFAIK), this doesn't work either.

I'm not sure what's going on here and will investigate more.

[philwo]

This seems relevant: http://stackoverflow.com/questions/29103828/cant-connect-to-ipv6-only-host-from-java

tl;dr - a bug in the JRE lets it pick a wrong interface (awdl0) for IPv6 sockets on macOS. The workaround is to disable AirDrop via "sudo ifconfig awdl0 down". A fix would probably involve writing our own SocketFactory that skips over the bad interface and using that SocketFactory for all connections.

[philwo]

Unassigning, because I'm not working in this area.

[danieldanciu]

I think somebody should take a look at this. It's a bit embarrassing that Bazel basically doesn't work with ipv6 addresses, and for us, that means either enable ipv4 in our network or use another build system.

[philwo]

@danieldanciu Can you give me some more details about your environment? Have you seen this on macOS only, or also on other platforms? Are you running an IPv6-only network with DNS64 and NAT64?

@buchgr Any idea how we could fix / workaround this? Last time I investigated this, it looked like an OpenJDK bug only on macOS, related to OpenJDK picking the wrong interface (awdl0) for IPv6 communication.

[danieldanciu]

Sure. The platform is indeed a Mac, and the problem is that java.net.Socket is basically not working with ipv6 addresses on a Mac, as described here:
http://stackoverflow.com/questions/29103828/cant-connect-to-ipv6-only-host-from-java

There is no workaround on the new macs (with a touch bar).

The fix on the Bazel side would be to use java.nio.SocketChannel instead of java.net.Socket whenever making an http connection. I tried to fix this myself, but I couldn't find where Bazel is using the Socket, I suspect it may be using it indirectly through the grpc problem.

The only way to work around this was to actually edit the libnet.dylib library with a hex editor and alter the code such that the correct interface is picked up.

[philwo]

Thank you for the detailed description and pointers! I’ll make sure that this gets fixed.

[buchgr]

The HttpDownloader seems to use HttpURLConnection which most likely uses java.net.Socket under the hood. I don't think that one can just pass in his own socket.

I suspect it may be using it indirectly through the grpc problem.

what are you refering to?

[danieldanciu]

Typo, I meant "grpc library". But you're likely right that the socket is opened via HttpURLConnection. Unfortunately, the socket factory only allows creating java.net.Sockets, so you guys may need to write your own HttpUrlConnection that uses socketchannel. Although I am pretty sure someone at Google already wrote this. :)

[sergiocampama]

This just bit me as well, any chance it could be looked at as corporate infrastructure is moving into IPv6?

[danieldanciu]

What version of basel/java/mac are you using? The problem went away for me, and I think it was the upgrade to Bazel 0.28 that did it.

[or-shachar]

I just got the same issue on bazel 0.29.1

[DFrenkel]

We have a build machine that has IPv6 network capable of reaching out to external destinations and IPv4 address that is internal-only. We are also getting Network unreachable exceptions in our build, but I believe I have a workaround that is based on a documented way in Java to prefer IPv6 by setting java.net.preferIPv6Addresses system property.

Specifically you can do the following:

• Use --host_jvm_args=-Djava.net.preferIPv6Addresses=true startup option, for example by having the following line in your bazelrc: startup --host_jvm_args=-Djava.net.preferIPv6Addresses=true
• If you are using Java tests need to connect to the internet as well (integration tests sometimes needs that), also use --jvmopt tool flag, for example by having the following line in your bazelrc: build --jvmopt=-Djava.net.preferIPv6Addresses
• If you are using rules_jvm_external as well (for example, coursier_fetch for maven artifact version resolution), add -Djava.net.preferIPv6Addresses=true to an environment variable called COURSIER_OPTS to Provide JVM options for Coursier

[philwo]

I just tried to reproduce this and embarrassingly, this issue is still present - Bazel 3.4.1 on macOS 10.15.6 fails to download files via HTTP when the computer is in an IPv6-only (with NAT64 and DNS64) network. 😬

$ bazel build //src:bazel
Starting local Bazel server and connecting to it...
INFO: Repository bazel_toolchains instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /Users/philwo/src/bazel/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/bazel-toolchains/releases/download/3.1.0/bazel-toolchains-3.1.0.tar.gz failed: class java.net.ConnectException Network is unreachable (connect failed)
WARNING: Download from https://github.com/bazelbuild/bazel-toolchains/releases/download/3.1.0/bazel-toolchains-3.1.0.tar.gz failed: class java.net.ConnectException Network is unreachable (connect failed)
ERROR: An error occurred during the fetch of repository 'bazel_toolchains':
   java.io.IOException: Error downloading [https://mirror.bazel.build/github.com/bazelbuild/bazel-toolchains/releases/download/3.1.0/bazel-toolchains-3.1.0.tar.gz, https://github.com/bazelbuild/bazel-toolchains/releases/download/3.1.0/bazel-toolchains-3.1.0.tar.gz] to /private/var/tmp/_bazel_philwo/7a01905b4627ca044e5e3f5ad5b14d26/external/bazel_toolchains/bazel-toolchains-3.1.0.tar.gz: Network is unreachable (connect failed)
ERROR: no such package '@bazel_toolchains//rules': java.io.IOException: Error downloading [https://mirror.bazel.build/github.com/bazelbuild/bazel-toolchains/releases/download/3.1.0/bazel-toolchains-3.1.0.tar.gz, https://github.com/bazelbuild/bazel-toolchains/releases/download/3.1.0/bazel-toolchains-3.1.0.tar.gz] to /private/var/tmp/_bazel_philwo/7a01905b4627ca044e5e3f5ad5b14d26/external/bazel_toolchains/bazel-toolchains-3.1.0.tar.gz: Network is unreachable (connect failed)

$ wget https://mirror.bazel.build/github.com/bazelbuild/bazel-toolchains/releases/download/3.1.0/bazel-toolchains-3.1.0.tar.gz
--2020-07-23 15:10:58--  https://mirror.bazel.build/github.com/bazelbuild/bazel-toolchains/releases/download/3.1.0/bazel-toolchains-3.1.0.tar.gz
Resolving mirror.bazel.build (mirror.bazel.build)... 64:ff9b::82d3:16eb
Connecting to mirror.bazel.build (mirror.bazel.build)|64:ff9b::82d3:16eb|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 850795 (831K) [application/x-gzip]
Saving to: ‘bazel-toolchains-3.1.0.tar.gz’

bazel-toolchains-3.1.0.tar.gz    100%[=========================================================>] 830.85K  --.-KB/s    in 0.04s   

2020-07-23 15:10:58 (18.8 MB/s) - ‘bazel-toolchains-3.1.0.tar.gz’ saved [850795/850795]

"Luckily" our work environment apparently migrated the WiFi to be IPv6-only, so that we can now easily test and work on this issue soonish.

[sgowroji]

Hi there! We're doing a clean up of old issues and will be closing this one. Please reopen if you’d like to discuss anything further. We’ll respond as soon as we have the bandwidth/resources to do so.

[philwo]

I’m pretty sure that this isn’t fixed yet.

@meisterT Could you try whether Bazel works on an IPv6 only network these days?

[kaycebasques]

I think I just hit this snag while working on a gLinux laptop in a Google office. After switching to our legacy IPv4 network this same command succeeded.

kayce@kayce0:~/sandbox/echo$ bazel build //...
Starting local Bazel server and connecting to it...
INFO: Repository platforms instantiated at:
  /home/kayce/sandbox/echo/WORKSPACE:21:13: in <toplevel>
Repository rule http_archive defined at:
  /home/kayce/.cache/bazel/_bazel_kayce/5b77aa1b33d7b7c439479c603973101b/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/platforms/releases/download/0.0.8/platforms-0.0.8.tar.gz failed: class java.net.ConnectException Network is unreachable (connect failed)
WARNING: Download from https://github.com/bazelbuild/platforms/releases/download/0.0.8/platforms-0.0.8.tar.gz failed: class java.net.ConnectException Network is unreachable (connect failed)
ERROR: An error occurred during the fetch of repository 'platforms':
   Traceback (most recent call last):
	File "/home/kayce/.cache/bazel/_bazel_kayce/5b77aa1b33d7b7c439479c603973101b/external/bazel_tools/tools/build_defs/repo/http.bzl", line 132, column 45, in _http_archive_impl
		download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://mirror.bazel.build/github.com/bazelbuild/platforms/releases/download/0.0.8/platforms-0.0.8.tar.gz, https://github.com/bazelbuild/platforms/releases/download/0.0.8/platforms-0.0.8.tar.gz] to /home/kayce/.cache/bazel/_bazel_kayce/5b77aa1b33d7b7c439479c603973101b/external/platforms/temp17883718197947293404/platforms-0.0.8.tar.gz: Network is unreachable (connect failed)
ERROR: /home/kayce/sandbox/echo/WORKSPACE:21:13: fetching http_archive rule //external:platforms: Traceback (most recent call last):
	File "/home/kayce/.cache/bazel/_bazel_kayce/5b77aa1b33d7b7c439479c603973101b/external/bazel_tools/tools/build_defs/repo/http.bzl", line 132, column 45, in _http_archive_impl
		download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://mirror.bazel.build/github.com/bazelbuild/platforms/releases/download/0.0.8/platforms-0.0.8.tar.gz, https://github.com/bazelbuild/platforms/releases/download/0.0.8/platforms-0.0.8.tar.gz] to /home/kayce/.cache/bazel/_bazel_kayce/5b77aa1b33d7b7c439479c603973101b/external/platforms/temp17883718197947293404/platforms-0.0.8.tar.gz: Network is unreachable (connect failed)
ERROR: /home/kayce/.cache/bazel/_bazel_kayce/5b77aa1b33d7b7c439479c603973101b/external/local_config_platform/BUILD.bazel:4:9: @local_config_platform//:host depends on @platforms//cpu:x86_64 in repository @platforms which failed to fetch. no such package '@platforms//cpu': java.io.IOException: Error downloading [https://mirror.bazel.build/github.com/bazelbuild/platforms/releases/download/0.0.8/platforms-0.0.8.tar.gz, https://github.com/bazelbuild/platforms/releases/download/0.0.8/platforms-0.0.8.tar.gz] to /home/kayce/.cache/bazel/_bazel_kayce/5b77aa1b33d7b7c439479c603973101b/external/platforms/temp17883718197947293404/platforms-0.0.8.tar.gz: Network is unreachable (connect failed)
ERROR: /home/kayce/sandbox/echo/src/BUILD.bazel:38:16: While resolving toolchains for target //src:echo.elf: Target @local_config_platform//:host was referenced as a platform, but does not provide PlatformInfo
ERROR: Analysis of target '//src:echo.elf' failed; build aborted: 
INFO: Elapsed time: 27.832s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (7 packages loaded, 4 targets configured)

[seanptmaher]

Yeah, I think I'm facing it right now not even on gLinux, but random arch linux on GoogleGuest. Fixed on GoogleGuest-IPV4.

thanks for making siso work off-corp @philwo :)