IPFS Client OOMKill

[grostirolla1]

Version information:

go-ipfs version: 0.7.0-ea77213
Repo version: 10
System version: amd64/linux
Golang version: go1.14.4

Description:

When trying to add multiple files in parallel (or even adding lots of single files in sequence for a long period of time) the IPFS clients is being oomkilled. The memory keeps growing until it reaches the limit (in my case 5GiB), and the client is killed.
I've tried a couple of suggestions from this issue #3532 without much luck.

• ipfs config --bool Swarm.DisableBandwidthMetrics true
• --enable-gc
• --routing=dhtclient

How to Reproduce:

Generate files

• for i in {1..100}; do dd if=/dev/random of=file_$i count=1 bs=102400000; done
  Add files through API
• for i in {1..100}; do curl -XPOST -F file=@file_$i "http://localhost:5001/api/v0/add"& done
  Or directly on the CLI (faster)
• for i in {1..100}; do ipfs add file_$i & done

Depending on your machine configurations (disk, memory etc) the oomkill might happen even when sending 100 x 10MB files.
This seems like a problem on how the memory is managed by IPFS.

[welcome]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

[BigLep]

2021-05-24 next step: see is can reproduce this locally.

[tsoeiroecore]

Hey guys, any updates on this? I'm facing similar issues in a highly concurrent environment.

[BigLep]

@tsoeiroecore : are you able to provide a reproducible case?

[BigLep]

Closing due to no response with a reproducible case.