Cache server performance is not so fast.

[junjihashimoto]

I think hydra.iohk.io is in the United States.
It is far from Asia and the download is not fast.
Would you please use a CDN?
You can use cloudflare for free.
If you need support, I'll do it.

[ocharles]

Maybe Cachix is a good option?

[junjihashimoto]

Yes. cachix uses CDN.

[angerman]

Where exactly are you having issues with IOGs hydra cache? We do have people in Asia and Oceania and this is the first time I hear about this; and I'm in SEA right now. Maybe we don't experience it much or just have a higher threshold of "acceptable performance"?

If you could provide more detail of the issue. You are experiencing, we might be able to help.

[junjihashimoto]

First, consider the download time of ghc as an example.
According to this(https://hydra.iohk.io/build/13318454#tabs-details), the size of ghc is 6GB.

Next, measure the RTT.
RTT of hydra.iohk.io averages 320ms.
RTT of cache.nixos.org averages 34ms.

The default TCP window size is 64KB.
TCP can only send one window size during RTT.

With hydra.iohk.io, it takes 8 hours to download 6GB.
(6000/(0.064/0.320))/(3600) = 8.3

With cache.nixos.org it takes an hour to download 6GB.
(6000/(0.064/0.034))/(3600) = 0.8

Since it is a cache, once it can be downloaded locally, it does not take a long time every time, but the first time it takes several hours.

I think it is acceptable to download GB data within an hour.

$ ping -c 5 hydra.iohk.io
PING packet-ipxe-hydra-1.ci.iohkdev.io (147.75.83.98) 56(84) bytes of data.
64 bytes from packet-ipxe-hydra-1 (147.75.83.98): icmp_seq=1 ttl=53 time=244 ms
64 bytes from packet-ipxe-hydra-1 (147.75.83.98): icmp_seq=2 ttl=53 time=306 ms
64 bytes from packet-ipxe-hydra-1 (147.75.83.98): icmp_seq=3 ttl=53 time=329 ms
64 bytes from packet-ipxe-hydra-1 (147.75.83.98): icmp_seq=4 ttl=53 time=349 ms
64 bytes from packet-ipxe-hydra-1 (147.75.83.98): icmp_seq=5 ttl=53 time=374 ms

--- packet-ipxe-hydra-1.ci.iohkdev.io ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 244.150/320.490/373.946/44.243 ms
$ ping -c 5 cache.nixos.org
PING dualstack.v2.shared.global.fastly.net (151.101.110.217) 56(84) bytes of data.
64 bytes from 151.101.110.217 (151.101.110.217): icmp_seq=1 ttl=57 time=19.8 ms
64 bytes from 151.101.110.217 (151.101.110.217): icmp_seq=2 ttl=57 time=51.0 ms
64 bytes from 151.101.110.217 (151.101.110.217): icmp_seq=3 ttl=57 time=40.1 ms
64 bytes from 151.101.110.217 (151.101.110.217): icmp_seq=4 ttl=57 time=40.3 ms
64 bytes from 151.101.110.217 (151.101.110.217): icmp_seq=5 ttl=57 time=20.3 ms

--- dualstack.v2.shared.global.fastly.net ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 19.766/34.303/51.013/12.292 ms

[junjihashimoto]

Since it is a limit for each connection, it may actually be shorter than this.

[junjihashimoto]

In summary, the current download bottleneck is RTT.
In distant places like Asia, using CDN will make it 10 times faster.

[jvanbruegge]

TCP can dynamically increase the window size. You can check with cat /proc/sys/net/ipv4/tcp_window_scaling if this enabled

[ocharles]

As I understand it, The Mathis equations states that the maximum throughput over TCP is inversely proportional to the RTT, so regardless of window scaling and all other things the time to download GHC can only go up as RTT increases.

[junjihashimoto]

It seems that the window size is about 500 in my linux environment. (The measurement time is too short.)

$ cat /proc/sys/net/ipv4/tcp_window_scaling
1
$ nix copy --from https://hydra.iohk.io /nix/store/f3nnq7y99r3gh3i4a50iggzdbhglm3l5-haskell.nix-unstable-ghc902-x86_64-linux-native >& /dev/null
$ sudo tcpdump host hydra.iohk.io and port 443 >& log
$ grep Flags log | sed -e 's/.*win \([0-9]\+\)[, ].*/\1/g' | sort|uniq -c
     70 501
     35 505

[junjihashimoto]

I'm using both iohk.cachix.org and hydra.iohk.io, but iohk.cachix.org doesn't have a ghc cache.
So the download speed of ghc is 1Mbyte/sec or less.

trace: No index state specified for haskell-project, using the latest index state that we know about (2022-04-11T00:00:00Z)!
[0/290 built, 1/2/96 copied (2149.5/2641.2 MiB), 163.2/219.3 MiB DL] fetching ghc-9.2.2 from https://hydra.iohk.io

[domenkozar]

I'm happy to sponsor cachix cache for haskell.nix

domen@cachix.org

[junjihashimoto]

Are there any concerns about setting this on the hydra server?
https://docs.cachix.org/continuous-integration-setup/hydra

[hamishmack]

Please try replacing https://hydra.iohk.io with https://cache.iog.io in your nix configuration.

Docs update is in #1481

[junjihashimoto]

It is really fast. I appreciate your help!