Execution is very slow to fallback to ipv4 when using ipv6

[ericbf]

I'm trying the simplest of translations (single words) and it takes minutes at a time... What? Why is it taking so long?

I'm on a Mac, default terminal (bash), and installed translate-shell from homebrew

Example of slowness:

$ time trans -b en:pt test
teste

real	1m16.058s
user	0m0.445s
sys	0m0.168s

Example through curl showing quick execution:

$ time curl -A 'Mozilla' "http://translate.googleapis.com/translate_a/single?client=gtx&dt=bd&dt=ex&dt=ld&dt=md&dt=rw&dt=rm&dt=ss&dt=t&dt=at&dt=qc&sl=en&tl=pt&q=test" &> /dev/null 

real	0m0.159s
user	0m0.007s
sys	0m0.006s

Version:

$ trans --version
Translate Shell       0.9.6.9

platform              Darwin
gawk (GNU Awk)        4.2.1
fribidi (GNU FriBidi) 1.0.5
audio player          [NOT INSTALLED]
terminal pager        less
terminal type         xterm-256color
user locale           en_US.UTF-8 (English)
home language         en
source language       auto
target language       en
translation engine    google
proxy                 [NONE]
user-agent            Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/602.1 (KHTML, like Gecko) Version/8.0 Safari/602.1 Epiphany/3.18.2
theme                 default
init file             [NONE]

[ericbf]

Another example – timing version output:

$ time trans --version &> /dev/null 

real	0m0.293s
user	0m0.129s
sys	0m0.167s

[soimort]

It is a known issue that trans could be slow due to the slowness of the gawk and bash interpreters, but 1min is way too much. (The same command takes 1.5s to finish on my machine.)

1. Does this happen for other engines (e.g., bing, yandex), or just google?
2. Does this happen when installing from GitHub? (You may git clone this repo and simply call ./translate to try out.)
3. Please run it with -debug -b en:pt test and let me know at which step it gets stuck.

[ericbf]

1. Yes, it happens with other engines as well:

$ time trans -e bing -b en:pt test
Teste

real	4m5.786s
user	0m0.151s
sys	0m0.205s

$ time trans -e yandex -b en:pt test
teste

real	1m24.952s
user	0m0.211s
sys	0m0.219s

2. Yes, cloned directly from GitHub, it behaves the same way

$ time ./translate -b en:pt test
teste

real	1m21.834s
user	0m0.415s
sys	0m0.049s

3. The debug call seemed to be stuck on gawk, then everything else was very quick. See the video showing the behavior.

[soimort]

I have no idea how it could happen. The initialization of gawk seems very fast (as your trans --version is as fast as 0.293s), and the JSON parsing seems to be fast too, since it follows immediately after the HTTP response in the debug message, as shown in your video.

I suggest sniffing the traffic to find out where the bottleneck is:

$ [sudo] tcpdump -n -v host translate.googleapis.com

Run the trans command in a separate terminal then.

Normally a GET request and an HTTP/1.1 200 OK response will show up in no time. But if it gets stuck at some point, you can easily tell from the log.

[ericbf]

Here's a video showing the tcpdump and the actual output.

It seemed to get stuck the longest on this line:

12:45:12.839627 IP6 (flowlabel 0x15a11, hlim 64, next-header TCP (6) payload length: 44) 2002:1776:5c77:0:155a:a4e2:c258:3754.64920 > 2607:f8b0:4008:80d::200a.80: Flags [S], cksum 0xfd11 (correct), seq 935145119, win 65535, options [mss 1440,nop,wscale 6,nop,nop,TS val 1081263766 ecr 0,sackOK,eol], length 0

[soimort]

Simply put, your system default has been set to use IPv6, but your network environment doesn't really support that. It took 11 attempts via IPv6 but only to no avail. Once it started to fallback into our good old IPv4 (appeared at 12:45:52.948927 in your log), it proceeded very quickly.

Disabling IPv6 on your system will resolve the slowness, most likely. You can read more about the IPv6 slowness problem here: https://en.wikipedia.org/wiki/IPv6_brokenness_and_DNS_whitelisting

The reason why libcurl (or other well-developed HTTP libraries, and popular browsers like Chrome and Firefox) works fast even on a faulty IPv6 configuration is because of a fast fallback algorithm they implemented.

Since translate-shell implements networking through a virtual I/O, and gawk handles its TCP/IP directly through sockets, no such trick was ever implemented.

[soimort]

For translate-shell, there are two possible solutions I can think of:

1. Add options -4 and -6 to allow users to choose from IPv4 and IPv6.
2. Implement its own Happy Eyeballs algorithm. This isn't hard at all, it seems.

[ericbf]

I disabled ipv6 with networksetup -setv6off "Wi-Fi" and it worked!

$ time trans -b en:pt test
teste

real	0m1.072s
user	0m0.470s
sys	0m0.179s

Good catch! Thanks!!

[ericbf]

I'll leave this issue open for the long-term ipv6 versus ipv4 solution, as we can't rightly ask all users to disable ipv6 in their environments.

[soimort]

Options -4 and -6 are implemented, and now trans will retry connecting through IPv4 if the default timed out (300ms).

[fy2]

Same issue here. Search takes ages, but if I do
networksetup -setv6off "Wi-Fi"
If starts working fast again (Translate Shell 0.9.6.9-release).