Excessive/Expected IPFS memory, threads, CPU? - private deployment

[paohallo]

Version information:

~$ipfs version --all
go-ipfs version: 0.7.0
Repo version: 10
System version: amd64/linux
Golang version: go1.14.4

System information:

~$ cat /proc/cpuinfo | grep "model name"
model name : Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
model name : Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz

~$ cat /proc/meminfo
MemTotal: 16398484 kB
MemFree: 522740 kB
MemAvailable: 11577716 kB
Buffers: 358036 kB
Cached: 5887524 kB
SwapCached: 12876 kB
Active: 6208340 kB
Inactive: 9162904 kB
Active(anon): 3178700 kB
Inactive(anon): 1027976 kB
Active(file): 3029640 kB
Inactive(file): 8134928 kB
Unevictable: 416 kB
Mlocked: 416 kB
SwapTotal: 1003516 kB
SwapFree: 0 kB
Dirty: 4 kB
Writeback: 0 kB
AnonPages: 9113824 kB
Mapped: 134616 kB
Shmem: 66848 kB
KReclaimable: 228508 kB
Slab: 339572 kB
SReclaimable: 228508 kB
SUnreclaim: 111064 kB
KernelStack: 13744 kB
PageTables: 39352 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 9202756 kB
Committed_AS: 16393256 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 25548 kB
VmallocChunk: 0 kB
Percpu: 13168 kB
HardwareCorrupted: 0 kB
AnonHugePages: 2048 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 1658752 kB
DirectMap2M: 15118336 kB
DirectMap1G: 2097152 kB

Description:

Running an experiment for potential PoC of a private IPFS deployment to determine memory and CPU consumption, and in general become more familiar with the IPFS solution. Not looking at a clustered deployment yet, as we don't need the replication to all nodes, only some nodes will seed/pin, others will pull on-demand.

Overview of the experiment:
Started with a minimal set of nodes (4).
Primary Node (Peer 1) (for the experiment with pinned content)
Consumer Node (Peer 2) (for the experiment, just continuously pulls from Primary, doing ipfs repo gc after 10 files
Pinned 60K+ objects (variety of different files, large (1GB x 10 /dev/urandom generated files), medium (10MB small x 100 /dev/urandom files), (1K x 5K /dev/urandom files), and other assorted.
Peer 3 - Present but not participating in any get or other operations
Peer 4 - Present but not participating in any get or other operations

The operation involves getting 10 1GB files from the primary node, then issuing the GC operation, and repeat. Monitoring the memory, CPU and threads of the ipfs daemon on the consumer Node.

Ran the same setup a couple of times over the holiday season to capture some data, and first iteration, ipfs get operation just stalled, and was none responsive when the ipfs instance got to 5.8G resident memory usage.

In the second iteration the system kept running until I interrupted it this morning (started on morning of 2nd Jan 2021), at which point it had reached the following consumption level:

So although this is not an expected deployment model or typical usecase, what are the expected scale numbers to expect in terms of memory usage, threads and garbage collection for the ipfs daemon (without the --enable-gc), and ipfs repo gc being triggered between operations?

[welcome]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

[aschmahmann]

@paohallo as the bot mentions:

Finally, remember to use https://discuss.ipfs.io if you just need general support.

This way answers to questions like yours can be utilized by other people as closed Github issues tend not to be very discoverable.