Loki (in Docker) reports "no space left on device" but there's plenty of space/inodes #1502
Comments
|
sorry, nothing obvious sticks out to me here. The 5 million chunks is certainly a lot... Loki is just using Go's You could maybe try asking in the Go slack or github? Edited typos |
|
Other things I can think of: chunk names are pretty long, what happens if you try to create a file in that directory with a really long name (instead of write_test), it's hard for me to pin down details on this but there is a size associated with file names and this I think has a limit as well, so too many long file names might be causing this. I have no idea how inodes work in relation to the host volume and from within docker, it seems like your dd test would indicated there are enough inodes but may be worth checking in and outside the container? What filesystem are you using? |
|
Looks like you can do an On one of my raspberry pi test loki's it has a couple hundred thousand chunks in the directory and that corresponds to a directory size of 20M |
|
Thanks for taking a look. Filesystem is ext4 mounted as a Docker volume.
|
|
It looks like all of the "failed to flush user" errors refer to three files, none of which actually exist: These are different chunk names than I originally reported, so these might change over time. I also restarted the container since the original report. I'll continue to monitor and report back if the invalid chunks change. Can also provide an strace of the error. |
|
Files with those filenames in the above errors can't be created manually, but filenames with very similar names in the same directory can be created. |
|
Ok, found it. I was also unable to create that file inside the directory mounted to I'll investigate if using a filesystem other than ext4 would allow for more files in the |
|
Disabled That caused I/O errors and a read-only file system, and after rebooting dropped into an initramfs prompt with a corrupt volume. After running |
|
This is really great work @shane-axiom ! Sorry you had a scare there on getting your data. I found this blog which you may have seen, that talks about the problem being a hash collision, this would make sense why only some names fail to write but not others. The disappointing information seems to be there isn't much that you can do aside from just disabling the It wasn't clear to me if the It wasn't obvious from me in any docs if this would have any affect on the b-tree file hashing, or if it's just related to the max files which can be stored. I'm also curious if disabling There are plans in the works to overhaul how local filesystem storage works, we want to find a way to combine chunks into larger files to reduce the file count and help improve performance. I'm afraid this work is quite a few months from being done though. In the meantime, you could try the new flag #1406 which lets you cut bigger chunks (although this will use more memory) FYI this is not released yet you would need to use a master-xxxx image or Or increase the Or reduce the number of labels or cardinality on labels to reduce number of streams. |
|
@slim-bean Yep, looks like |
|
This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
|
Thank you stale bot. @slim-bean Disabling
We don't have any performance metrics running for loki other than loki-canary, but it seems fine after this change, and subjectively I haven't noticed any significant difference. Just upgraded to loki 1.3.0 and haven't tried Before this gets closed, do you think it's worth distilling this |
stale
bot
removed
the
stale
|
Thanks for the update @shane-axiom yes this should definitely make it into the docs somewhere I haven't looked to see where. If you have a chance to add something that would be awesome, else I will try to add something too, for now we'll just kick stale bot and see where we end up in another 30 days :) |
|
This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
The chunks created by Loki were stored in a persistent volume. This does not scale well, since volumes cannot easily be resized in Kubernetes. Also, at least the ext4-filesystem had issues, when large numbers of logs were saved. These issues are due to the dir_index as discussed in [1]. An object store provides a more scalable and cheaper solution. Loki supports S3 as an object storage and also other object stores that understand the S3 API like Ceph or OpenStack Swift. [1] grafana/loki#1502 Change-Id: Id55095c3b6659f40708712c1a494753dbcab7686
|
hey everybody: This is still happening, we just ran into this. Having it closed by the stale bot it a bit disappointing, can this be reopened? As a quick fix, we'll probably just migrate to a more battle-tested chunk store backend (S3), but having issues like this in the fs backend is still annoying, as that's what most people probably start with. |
Even closed, this issue is still relevant. I had several issues with |
|
Same here; still very much an issue with latest Loki; it's a very fresh installation with pulling logs from maybe 20-30 hosts and already hitting >3M chunks in a single directory. It's just not feasible with a local filesystem; is it not typical to hash files in subfolder structure? |
|
@weakcamel it seems that the current posture on this issue is: I think there is also a lack of documentation about this, lets wait and see if it's fixed. |
|
Thanks @theonlydoo! Interesting to see that the discussion thread points to Cortex which does not seem to support filesystem storage at all: https://cortexmetrics.io/docs/chunks-storage/ Seems that at this point it's better to look at alternative storage options. |
You can enable the |
|
You need to be running kernel 4.13 or newer. |
|
Thanks for the tip. I will try it out when I got a chance!
On Apr 19, 2021, at 2:59 PM, Kristian Klausen ***@***.***> wrote:
# tune2fs -O large_dir /dev/nvme0n1
tune2fs 1.42.9 (28-Dec-2013)
Setting filesystem feature ‘large_dir’ not supported.
You need to be running kernel 4.13 or newer<torvalds/linux@e08ac99>.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<#1502 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABETAJ7KJK42A22GGQJAUS3TJSRUNANCNFSM4KFTTEOQ>.
|
|
Had the same issue with 3.7 today, enabling |
|
If anyone's using NetApp ONTAP volumes, the variable that controls the size of the directory index is The article also mentions some of the tradeoffs of raising the size of the directory index, and some of that advice probably applies to ext4 as well. |
|
How can we enable large_dir on Digital ocean mounted kubernetes persistance volumes? I think this issue should be reopened? |
|
+1 for enabling |
|
I was able to connect to our kubernetes node and apply there the But now loki still can't start and is writing a lot of following errors about Has anyone an idea how to fix this? |
|
Came here for: Solution from above: Environment: |
|
Enabling Running on |
|
Enabled |
|
We had the same problem in a Rook/Ceph cluster with RBD images (ceph-block storage). What did work was to execute the same command from the node where the volume was mounted. |
|
Another ❤️ for the |
It worked for me too but it took some time to fix |
|
I have also been troubled by this issue. It can only query data for up to 1 hour, and it is particularly slow. Later on, the storage was switched to minio s3, and the performance is no longer an issue |
|
Setting |
|
As @wschoot points out EXERCISE CAUTION before you run Anything using < grub-2.12 will fail to boot and report an unknown filesystem error similar to: And any grub rescue operations or ISO live / recovery using any grub2 utils, e.g. grub2-install, grub2-mkconfig will report the same error. AND .... IT GETS WORSE once |
|
Oof that's harsh. Hopefully they'll get in that newer grub version! Just to elaborate on what this all means (I have a cache where sometimes I get like 200,000 files in a single directory and it's rather slow, other times it's like 1500 larger files instead and it's a fair bit faster, and I can place a limit on number of files in the cache settings. So I decided to research what limits would likely result in differences in directory lookup time.) This is all in the documentation technically (since the ext4 disk structure is documented) but I wanted to figure out reasonably concrete numbers. I found the classic linear directory has an overhead of 8 bytes per entry plus the file name itself, 4KB block so for short like 10-12 character names you can hold several hundred entries. For long names like this program is using, the names are around 72 characters, 80 bytes per directory so you'll end up with about 50 files per 4KB block. If the directory gets bigger than a single 4KB block, it's converted to using dir_index, it uses an htree. dir_index, the htree index holds block holds about 500 entries, each pointing to a block holding conventional directory entries, so 500*50 would be about 25000 entries. Now "birthday paradox" suggests you could hit the limits of a single layer htree at like 10,000-15,000 entries instead as some block fills before the rest. If a "leaf" gets over 4KB again a second layer is added to the tree. With a 2-level htree, 50050050 is 12,500,000 directory entries. A 3rd layer was not allowed, so the hashes didn't hash out quite evenly, some directory block got it's 50 entries and filled up a little ahead of schedule and the directory failed with 5.5 million entries instead. The large_dir option allows a 3-level htree (125,000,000 potential leafs with about 50 files apiece), 6.25 billion files (or really since you might hit the limit a little below that due to uneven hashing, like 3, 4 billion.) I'll note the hash value used is truncated to 32 bits (around 4 billion entries) anyway so having a 4-level h-tree wouldn't really do much unless you were using extraordinarily long file names. Obviously I would seriously consider a redesign if you were going to hit billions of items in a single directory. (For my purposes, my file names are more like 10-12 characters so I'd hit about 200 files before going from a classic directory to an htree and 100,000 limit on a 1-level h-tree, so proabaly keep file limit to 50,000 or so to avoid significant amounts of 2-level tree action.) |
Yes. The issue is common among services like email servers or others storing large number of files. and the store the files in there. This reduces the files per directory by orders of magnitude. See here how this is done for Dovecot: https://doc.dovecot.org/configuration_manual/mail_location/#directory-hashing |
|
^^ @slim-bean could you reopen this one maybe? I strongly believe Loki should "learn" to store its chunks / files in a way that does not require filesystems to scale to millions of files in a single directory. |
|
sorry, I haven't checked on this issue in quite some time.... I will say it wasn't ever really our intention for folks to take the filesystem this far, Loki was really meant to be used with object storage... But I know there are lots and lots of folks out there who are running with filesystems and I'd love for y'all to be happy and not having issues. I wonder what we should rename this issue to. |
slim-bean
added
the
keepalive
|
Maybe something like "Support large numbers of loki chunks on filesystems using prefix directory hierarchy"? |
I believe the word "hashed" or "hashing" is used for this kind of directory hierarchy / structure:
If I may quote what our dear friend ChatGPT has to say:
|
|
The chunk filenames in |
|
Hello, just to provide input for the discussion. A proper solution for this would be appreciated. I intend to roll out loki to customers soon. Using an object store is not feasable for any of those customers due to the very sensitive nature of the data that these systems will be carrying they are usually air gapped. Just to give you an idea, nobody will ever analize the logs on prem. I intend to just have the customer put the entire docker mount of loki onto (ideally) a tape drive for example and send it to me for analysis. I will then have my (once again air gapped) computer where I will have my grafana dashboards to analize everything. While I dont doubt that tar can handle infinite amount of files in a directory I can already see a customer just trying to copy the directory onto an external storage disk with god forbid ntfs or any other random file systems before sending them to me. So yeah a solution that loki does not create all chunks in a single directory would indeed be appreciated. Any of the methodologies described above would solve those problems. Literally just creating 4 levels of directories for example taking 2 letters of a hash of the chunk file name would probably do the job. The main reason why I personally think that loki is awesome is precisely because of the file system storage and the fact that its a single binary that just works. (Plus the push rest api is easy to use, thats not relevant here) I dont think you should lazor focus on just beeing yet another cloud logging tool. It can so easily be perfect for large on prem solutions too. I would not have minded personally to use apache casandra instead of the fs but for some reason you guys deprecated that. All the other storage options are non options for air gapped systems especially if you have somewhat (justifiably) paranoid customers. I hope you guys manage to find an acceptable solution to this problem. Sincerely |
Fair point.
|
|
Is it known whether this issue applies when using the |
and others
Describe the bug
When running the Loki 1.2.0 Docker image, Loki is reporting that it can't write chunks to disk because there is "no space left on device", although there appears to be plenty of space.
Plenty of space and inodes available on disk where
/tmp/lokivolume lives:/tmp/lokinamed volume mount fromdocker inspectExecing into the loki container and doing manual write tests to verify that files can be written
I haven't been able to find any disk limitation in the Docker container, and the fact that I can still manually write files to the volume inside the container makes me suspect the bug is in the loki code, but I could definitely be wrong!
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Loki continues to successfully write chunks to
/tmp/lokiwhile disk space and inodes are available.Environment:
/etc/loki/local-config.yaml)The text was updated successfully, but these errors were encountered: