unexpected high cpu load and memory consumption

[RubenKelevra]

Version information:

I was running 0.5.0-rc4 from commit 116999a

Repo version: 9
System version: amd64/linux
Golang version: go1.14.2

Description:

While there was just single digit MBit/s traffic on the server, the CPU was under substantial load by IPFS and it used nearly half of the RAM.

ipfs       13567  6.1  6.2 4970396 1017004 ?     Ssl  Apr21 884:15 /usr/local/bin/ipfs-cluster-service daemon
ipfs     2403637  223 43.9 16769580 7202860 ?    Ssl  Apr26 16229:31 /usr/bin/ipfs daemon --enable-pubsub-experiment --enable-namesys-pubsub

The VM has 16 G memory and 4 cores of an Intel Xeon Gold 6140 CPU @ 2.30GHz.

The debug-files are attached.
ipfs_debug.tar.gz

[Stebalien]

It looks like you have ~243 QUIC connections. Is that correct? (ipfs swarm peers | grep -c quic).

cc @marten-seemann I'm seeing a lot of timers firing and a lot of timers being held by QUIC connections.

[marten-seemann]

@Stebalien Looking at ipfs.heap, it looks like the QUIC sessions are holding 41 MB of timers in total. That doesn't seem right for just 243 connections.

It seems like the session doesn't reset or stop the timer when it is closed (I just created quic-go/quic-go#2515), so maybe we're leaking timers there? Although the timer is set to a maximum of 30s (the idle timeout), so leaked timers should be garbage collected after that time frame.

[Stebalien]

The timer is getting created with a timeout of "max int".

[marten-seemann]

I'm not sure if that's the problem, since the timer would get reset as soon as the first packet is either sent or received. In any case, stopping the timer when the QUIC connection is closed seems like the right thing do.

[RubenKelevra]

It looks like you have ~243 QUIC connections. Is that correct? (ipfs swarm peers | grep -c quic).

This could be right, the server has enough power to handle a lot of connections, so the limiter is basically "open", to avoid that my server is cutting connections, but leave the other side handle the disconnects (if this makes any sense).

These are the settings I use:

loki_config.txt

[Stebalien]

This has since been fixed.