Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Accelerated DHT Client causes OOM kill upon start of IPFS, ResourceMgr.MaxMemory ignored #9990

Open
3 tasks done
dokterbob opened this issue Jun 25, 2023 · 12 comments
Open
3 tasks done

Accelerated DHT Client causes OOM kill upon start of IPFS, ResourceMgr.MaxMemory ignored #9990

dokterbob opened this issue Jun 25, 2023 · 12 comments
Labels
kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization

Comments

@dokterbob
Copy link
Contributor

Checklist

Installation method

third-party binary

Version

Kubo version: 0.20.0-b8c4725
Repo version: 13
System version: amd64/linux
Golang version: go1.19.1

Config

{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Methods": [
        "PUT",
        "POST"
      ],
      "Access-Control-Allow-Origin": [
        "http://<private>:10125",
        "http://localhost:3000",
        "http://127.0.0.1:5001",
        "https://webui.ipfs.io"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/0.0.0.0/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [
      "/dns4/<private>/tcp/4001",
      "/dns4/<private>/udp/4001/quic",
      "/dns4/<private>/udp/4001/quic-v1",
      "/dns4/<private>/udp/4001/quic-v1/webtransport",
      "/dns4/<private>/tcp/4001",
      "/dns4/<private>/udp/4001/quic",
      "/dns4/<private>/udp/4001/quic-v1",
      "/dns4/<private>/udp/4001/quic-v1/webtransport"
    ],
    "Gateway": "/ip4/0.0.0.0/tcp/8080",
    "NoAnnounce": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/quic",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
  ],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 1048576,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/3",
            "sync": false,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "100GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": false
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": true,
    "FilestoreEnabled": true,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": ""
  },
  "Identity": {
    "PeerID": "<private>",
    "PrivKey": "<definitely private>"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "Methods": null,
    "Routers": null
  },
  "Swarm": {
    "AddrFilters": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": true,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {
      "Limits": {},
      "MaxMemory": "1GB"
    },
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

Description

Context

Starting kubo

  • Within a container with memory limits of 2, 3 or 4 GB.
  • With ResourceMgr.MaxMemory set to 1 or 2 GB.
  • With ports forwarded on 100/100 fibre line.
  • With about 10 TB of 1 MB blocks added through filestore (so ~10M blocks).

Consistently results in a OOM kill within minutes of startup.

Observations

When the node is started;

  1. It establishes ~2k peer connections (as, arguably, it should).
  2. 6-9 MB/s of DHT traffic is flowing in, with 200-400 KB/s going out.
  3. Within minutes (less than 15m, more than 1m) it consistently swallows all available memory and gets killed.

With AcceleratedDHTClient set to false, this problem does not occur. Hence, it seems that the Accelerated DHT client does not adhere to the resource manager's memory limit. Perhaps the defaults just need to be made accelerated DHT aware.

Might want to fix this before taking it out of the experimental phase. ;)

Possibly related/relevant
@dokterbob dokterbob added kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization labels Jun 25, 2023
@Jorropo
Copy link
Contributor

Jorropo commented Jun 25, 2023

ResourceMgr.MaxMemory does not limit how much memory Kubo is allowed to use, it limit how much the network buffers are allowed to use.

@Jorropo
Copy link
Contributor

Jorropo commented Jun 25, 2023

Could you please post anipfs diag profile when lots of memory is used ?

@dokterbob
Copy link
Contributor Author

dokterbob commented Jun 26, 2023 •
edited

ResourceMgr.MaxMemory does not limit how much memory Kubo is allowed to use, it limit how much the network buffers are allowed to use.

Ah. That's a bit misleading from the name of the setting and AFAIK not documented anywhere.
So there is still no way to limit IPFS memory usage?

Could you please post anipfs diag profile when lots of memory is used ?

I can make another test run before it goes harakiri perhaps in about 3 days (once I'm done ingesting the 22 TB ipfs-search.com snapshot archive :p).

@Jorropo
Copy link
Contributor

Jorropo commented Jun 26, 2023 •
edited

So there is still no way to limit IPFS memory usage?

*Kubo 😉
Golang does not have the logistics for that kinda of controls, it's not worth my time implementing memory limits in a language that have implicit allocations everywhere, and also allocations randomly happen or not depending on complex heap escape rules that are not compatible with features as common as interfaces.

Need to rewrite everything into Zig before attempting that.

You can use GOMEMLIMIT (I need to get #9451 merged) but this is a weak cap, it will spin the CPU harder and harder as the limit is breached.

@dokterbob
Copy link
Contributor Author

I've turned it on again and set GOMEMLIMIT. It does seem that it's more well-behaved now.

Perhaps still notable though that without this, the accelerated DHT client is particularly memory hungry.

Config

{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Methods": [
        "PUT",
        "POST"
      ],
      "Access-Control-Allow-Origin": [
        "http://<private>:10125",
        "http://localhost:3000",
        "http://127.0.0.1:5001",
        "https://webui.ipfs.io"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/0.0.0.0/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [
      "/dns4/<private>/tcp/4001",
      "/dns4/<private>/udp/4001/quic",
      "/dns4/<private>/udp/4001/quic-v1",
      "/dns4/<private>/udp/4001/quic-v1/webtransport",
      "/dns4/<private>/tcp/4001",
      "/dns4/<private>/udp/4001/quic",
      "/dns4/<private>/udp/4001/quic-v1",
      "/dns4/<private>/udp/4001/quic-v1/webtransport"
    ],
    "Gateway": "/ip4/0.0.0.0/tcp/8080",
    "NoAnnounce": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/quic",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
  ],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 1048576,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/3",
            "sync": false,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "100GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": false
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": true,
    "FilestoreEnabled": true,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": ""
  },
  "Identity": {
    "PeerID": "<nop>",
    "PrivKey": "<no>"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "Methods": null,
    "Routers": null
  },
  "Swarm": {
    "AddrFilters": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": true,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {
      "Limits": {},
      "MaxMemory": "750MB"
    },
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

Bunch 'o profiles: https://dweb.link/ipfs/Qmays49dWQUXQzh7NgRFRDvrp5qT4nuCcSruAgGmUr539E

@Jorropo
Copy link
Contributor

Jorropo commented Jun 29, 2023 •
edited

I can't find the issues but I know about that indeed known.

The accelerated DHT client and batch provider load you 22TB of CIDs in memory (slice of ptr to cid + len) and pass it to the accelerated DHT client.
In theory you can limit it but I accidently deleted the code to do that recently (*it was never wired in Kubo anyway). We could expose this in v0.22. But the ideal fix and not workaround is to make that code streaming so you don't have to load everything.

Note: The Announcement cost per CID goes down as you have more CID since acceleratedDHTclient batch them up.

@Jorropo Jorropo mentioned this issue Jun 29, 2023
Open
@Jorropo
Copy link
Contributor

Jorropo commented Jun 29, 2023

After some back of the envelope computation it should be 1~2 GiB of memory just in CID counting pointer and GC overhead.

@dokterbob
Copy link
Contributor Author

Will this memory be freed after loading or does it always stay in memory?

@dokterbob
Copy link
Contributor Author

IPFS with GOMEMLIMIT still within 1GB by the way. Also not swapping (much).

@Jorropo
Copy link
Contributor

Jorropo commented Jun 29, 2023

Will this memory be freed after loading or does it always stay in memory?

it is freed after the providing process is finished.

@dokterbob
Copy link
Contributor Author

dokterbob commented Jul 23, 2023 •
edited

Just tried again with latest 0.21 release and accel. DHT. Even with GOMEMLIMIT (768 MiB) and ResourceMgr.MaxMemory (512 MiB), kubo gets OOM killed at 2 GiB.

My questions:

  1. Is this type of memory usage always the case? In which case I feel the min. required memory usage should most certainly be documented, e.g. in config).
  2. If not, is this sort of behaviour dependent on other factors (e.g. bandwith, amount of pinned resources ~290k in my case)? In which case it might be very valuable to investigate scaling behaviour as there might clearly be an issue for larger datasets if this is more than O(1).

@Jorropo
Copy link
Contributor

Jorropo commented Jul 24, 2023

GOMEMLIMIT isn't magic, it's just a target. Experimentally minimum is around ~3GiB, given you don't want CPU to spin constantly I think a minimum of 4GiB is required.

Golang is not a language that gives the required tools for programmers to correctly control memory usage, so trying to set a limit in the config wouldn't be very useful.
There is no simple answer to what memory usage scale with, it's both everything and nothing at the same time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dokterbob @Jorropo