-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The ARC size (current) is ten times larger than the Max size (zfs_arc_max), it will result in "Out of memory: killed process". #16325
Comments
Dupe of #16322, if I had to guess. |
@robn you wrote at #16322 Could you capture cat /proc/spl/kmem/slab before the import, and at the time of the OOM, and post them? I did it. Full log /proc/slabinfo and /proc/meminfo memory_info_2024-07-08_17-03-53.txt Important note! If I import the pool as read-only, everything works fine, and the RAM is not consumed. Before import pool zp-erp1-hdd
import pool zp-erp1-hdd
3.3 GiB - ARC size (current)
20.6 GiB - ARC size (current)
51.1 GiB - ARC size (current)
|
Today I reinstalled proxmox with previous version. It didn' help. The problem is still exist. Now:
was before reinstall
|
I installed FreeBSD 14 and imported this pool. It works fine! |
System information
Describe the problem you're observing
Immediately after importing the pool, the ARC size (current) starts growing and quickly exceeds the Max size (zfs_arc_max).
It continues to grow until there is no free RAM left on the server. This causes an Out of memory: killed process. The server has a total of 64GB of RAM.
In arc_summary, I see that the Anonymous metadata size starts growing, and along with it, the ARC size (current) starts growing. When the Anonymous metadata size reaches 50.1 GiB and the Target size (adaptive) becomes 52.2 GiB, there is no free RAM left in the system.
arc_summary | grep -E 'ARC size (current)|Min size (hard limit)|Max size (high water)|Anonymous metadata size'
ARC size (current): 2616.7 % 52.3 GiB
Min size (hard limit): 50.0 % 1024.0 MiB
Max size (high water): 2:1 2.0 GiB
Anonymous metadata size: 99.5 % 50.9 GiB
Why is the ARC Max size limit (/sys/module/zfs/parameters/zfs_arc_max) set to 2GB being ignored? How is this possible?
What is Anonymous metadata size? I didn't find any information.
Describe how to reproduce the problem
A complete video demonstrating the problem.
https://youtu.be/ygukivJmaKw?si=jF5vZMDQm7FyfUmw&t=436
I import the pool.
root@erpband-pve1:~# zpool import -d /dev/disk/by-partlabel/ zp-erp1-hdd
After a 3-5 minutes, the ARC size starts to grow.
The maximum ARC size is set to 2GB, but it is being ignored.
I wrote a script
arc_monitor.txt
that outputs the parameters ARC size (current), Anonymous metadata size, Max size (high water), and Min size (hard limit) in a table. It clearly shows how the Anonymous metadata size grows, and along with it, the ARC size (current) grows.
Additional information
Include any warning/errors/backtraces from the system logs
There are no errors or warnings in the logs. The pool imports successfully, but it consumes all the RAM. Here is the log file for the period from the pool import to the system freeze:
logs_2024-07-07_18-40-36.txt
.
The text was updated successfully, but these errors were encountered: