Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

posixacl and full dataset #16199

Open
miekg opened this issue May 15, 2024 · 0 comments
Open

posixacl and full dataset #16199

miekg opened this issue May 15, 2024 · 0 comments
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@miekg
Copy link

miekg commented May 15, 2024

System information

Type Version/Name
Distribution Name Ubuntu 22.04.4 LTS
Distribution Version 22.04.4 LTS
Kernel Version 5.15.0-106-generic
Architecture x86_64
OpenZFS Version zfs-2.1.5-1ubuntu6~22.04.4, zfs-kmod-2.1.5-1ubuntu6~22.04.3

Describe the problem you're observing

Hello,

We seen a weird issue with a zfs dataset. The dataset ran out of quota while being written via an rsync and this broke a setfacl that ran during that time. This data set was created as:

zfs create ... -o quota=...G -o snapdir=hidden    
zfs set acltype=posixacl ...

At some point we are applying the acl's agian and this led to an ACL being written (with a \0) that breaks getfacl with an input/output error. Setting set acltype=noacl makes that error go away (of course) makes the data accessible again. Our best guess is that because the dataset was hitting it's quota setfacl wrote something weird (null byte) and that broke ACLs.

Currently our only fix is to disable ACLs.

Happy to provide more info if needed/required.

Describe how to reproduce the problem

unsure

Include any warning/errors/backtraces from the system logs

A zfs list -o all of the dataset:

NAME                    TYPE        CREATION                USED  AVAIL     REFER  RATIO  MOUNTED  ORIGIN  QUOTA  RESERV  VOLSIZE  VOLBLOCK  RECSIZE  MOUNTPOINT            SHARENFS  CHECKSUM   COMPRESS        ATIME  DEVICES  EXEC  SETUID  RDONLY  ZONED  SNAPDIR  ACLMODE      ACLINHERIT     CREATETXG  CANMOUNT  XATTR  COPIES  VERSION  UTF8ONLY  NORMALIZATION  CASE         VSCAN  NBMAND  SHARESMB  REFQUOTA  REFRESERV   GUID  PRIMARYCACHE  SECONDARYCACHE  USEDSNAP  USEDDS  USEDCHILD  USEDREFRESERV  DEFER_DESTROY  USERREFS  LOGBIAS     OBJSETID  DEDUP          MLSLABEL  SYNC      DNSIZE  REFRATIO  WRITTEN  CLONES  LUSED  LREFER  VOLMODE  FSLIMIT  SSLIMIT  FSCOUNT  SSCOUNT  SNAPDEV  ACLTYPE   CONTEXT  FSCONTEXT  DEFCONTEXT  ROOTCONTEXT  RELATIME  REDUND_MD  OVERLAY  RESUMETOK  ENCRYPTION   KEYLOCATION  KEYFORMAT   PBKDF2ITERS  ENCROOT  KEYSTATUS    SPECIAL_SMALL_BLOCKS  RSNAPS
tank/live/battaglialab  filesystem  Wed May  8 10:39 2024  37.1T  1000G     37.1T  1.00x  yes      -        none    none        -         -     128K  /export/battaglialab  off       on         off             on     on       on    on      off     off    hidden   discard      restricted        509983  on        on     1       5        off       none           sensitive    off    off     off          38.1T       none  3252391526688700669  all           all                65.8M   37.1T         0B             0B  -                     -  latency         4846  off            none      standard  legacy     1.00x     329K  -       36.6T   36.6T  default     none     none     none     none  hidden   off       none     none       none        none         on        all        on       -          off          none         none                  0  -        -                               0  -

zfs history during the time this happened:
(on the our stuff is our provisioning to (re)set certain properties)

15-05 10:43:47 [root@zwik8 log]# zpool history tank |grep batta | grep 2024-05-13
2024-05-13.00:00:11 zfs set readonly=off tank/live/battaglialab
2024-05-13.00:02:59 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.00:03:15 zfs set quota=none tank/live/battaglialab
2024-05-13.01:00:06 zfs set readonly=off tank/live/battaglialab
2024-05-13.01:01:41 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.01:01:53 zfs set quota=none tank/live/battaglialab
2024-05-13.01:59:41 zfs snapshot tank/live/battaglialab@cncz_20240512_235924_000
2024-05-13.02:00:21 zfs set readonly=off tank/live/battaglialab
2024-05-13.02:00:31 zfs destroy tank/live/battaglialab@cncz_20240510_235834_000
2024-05-13.02:01:35 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.02:01:46 zfs set quota=none tank/live/battaglialab
2024-05-13.03:00:24 zfs set readonly=off tank/live/battaglialab
2024-05-13.03:06:14 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.03:06:46 zfs set quota=none tank/live/battaglialab
2024-05-13.04:00:14 zfs set readonly=off tank/live/battaglialab
2024-05-13.04:04:15 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.04:04:43 zfs set quota=none tank/live/battaglialab
2024-05-13.05:00:04 zfs set readonly=off tank/live/battaglialab
2024-05-13.05:00:33 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.05:00:38 zfs set quota=none tank/live/battaglialab
2024-05-13.05:58:35 zfs snapshot tank/live/battaglialab@cncz_20240513_035821_000
2024-05-13.06:00:10 zfs set readonly=off tank/live/battaglialab
2024-05-13.06:01:03 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.06:01:06 zfs set quota=none tank/live/battaglialab
2024-05-13.06:31:45 zfs set readonly=off tank/live/battaglialab
2024-05-13.06:33:04 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.06:33:16 zfs set quota=none tank/live/battaglialab
2024-05-13.07:00:07 zfs set readonly=off tank/live/battaglialab
2024-05-13.07:00:20 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.07:00:22 zfs set quota=none tank/live/battaglialab
2024-05-13.08:00:06 zfs set readonly=off tank/live/battaglialab
2024-05-13.08:00:19 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.08:00:21 zfs set quota=none tank/live/battaglialab
2024-05-13.09:00:03 zfs set readonly=off tank/live/battaglialab
2024-05-13.09:00:16 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.09:00:18 zfs set quota=none tank/live/battaglialab
2024-05-13.09:58:32 zfs snapshot tank/live/battaglialab@cncz_20240513_075830_000
2024-05-13.09:58:45 zfs destroy tank/live/battaglialab@cncz_20240509_035833_000
2024-05-13.10:00:07 zfs set readonly=off tank/live/battaglialab
2024-05-13.10:00:19 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.10:00:22 zfs set quota=none tank/live/battaglialab
2024-05-13.11:00:03 zfs set readonly=off tank/live/battaglialab
2024-05-13.11:00:16 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.11:00:18 zfs set quota=none tank/live/battaglialab
2024-05-13.12:00:03 zfs set readonly=off tank/live/battaglialab
2024-05-13.12:00:16 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.12:00:18 zfs set quota=none tank/live/battaglialab
2024-05-13.13:00:07 zfs set readonly=off tank/live/battaglialab
2024-05-13.13:00:21 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.13:00:23 zfs set quota=none tank/live/battaglialab
2024-05-13.13:31:00 zfs set refquota=38000G tank/live/battaglialab
2024-05-13.13:31:03 zfs set quota=none tank/live/battaglialab
2024-05-13.13:32:26 zfs set refquota=39000G tank/live/battaglialab
2024-05-13.13:32:29 zfs set quota=none tank/live/battaglialab
2024-05-13.13:59:09 zfs snapshot tank/live/battaglialab@cncz_20240513_115843_000
2024-05-13.14:00:27 zfs set readonly=off tank/live/battaglialab
2024-05-13.14:01:37 zfs destroy tank/live/battaglialab@cncz_20240511_075854_000
2024-05-13.14:02:25 zfs set refquota=39000G tank/live/battaglialab
2024-05-13.14:02:49 zfs set quota=none tank/live/battaglialab
2024-05-13.15:00:08 zfs set readonly=off tank/live/battaglialab
2024-05-13.15:01:02 zfs set refquota=39000G tank/live/battaglialab                                                                                              
2024-05-13.15:01:13 zfs set quota=none tank/live/battaglialab
2024-05-13.16:00:08 zfs set readonly=off tank/live/battaglialab
2024-05-13.16:00:27 zfs set refquota=39000G tank/live/battaglialab
2024-05-13.16:00:32 zfs set quota=none tank/live/battaglialab
2024-05-13.17:00:12 zfs set readonly=off tank/live/battaglialab
2024-05-13.17:00:49 zfs set refquota=39000G tank/live/battaglialab
2024-05-13.17:00:56 zfs set quota=none tank/live/battaglialab
2024-05-13.17:39:51 zfs set readonly=off tank/live/battaglialab
2024-05-13.17:41:05 zfs set refquota=39000G tank/live/battaglialab
2024-05-13.17:41:18 zfs set quota=none tank/live/battaglialab
2024-05-13.17:58:39 zfs snapshot tank/live/battaglialab@cncz_20240513_155837_000
2024-05-13.18:00:03 zfs set readonly=off tank/live/battaglialab
2024-05-13.18:00:15 zfs set refquota=39000G tank/live/battaglialab
2024-05-13.18:00:18 zfs set quota=none tank/live/battaglialab
2024-05-13.19:00:03 zfs set readonly=off tank/live/battaglialab
2024-05-13.19:00:15 zfs set refquota=39000G tank/live/battaglialab
2024-05-13.19:00:17 zfs set quota=none tank/live/battaglialab
2024-05-13.20:00:03 zfs set readonly=off tank/live/battaglialab
2024-05-13.20:00:14 zfs set refquota=39000G tank/live/battaglialab
2024-05-13.20:00:16 zfs set quota=none tank/live/battaglialab
2024-05-13.21:00:12 zfs set readonly=off tank/live/battaglialab
2024-05-13.21:01:00 zfs set refquota=39000G tank/live/battaglialab
2024-05-13.21:01:10 zfs set quota=none tank/live/battaglialab
2024-05-13.21:58:46 zfs snapshot tank/live/battaglialab@cncz_20240513_195844_000
2024-05-13.21:58:59 zfs destroy tank/live/battaglialab@cncz_20240511_155829_000
2024-05-13.22:00:03 zfs set readonly=off tank/live/battaglialab
2024-05-13.22:00:13 zfs set refquota=39000G tank/live/battaglialab
2024-05-13.22:00:15 zfs set quota=none tank/live/battaglialab
2024-05-13.23:00:03 zfs set readonly=off tank/live/battaglialab
2024-05-13.23:00:14 zfs set refquota=39000G tank/live/battaglialab
2024-05-13.23:00:16 zfs set quota=none tank/live/battaglialab
@miekg miekg added the Type: Defect Incorrect behavior (e.g. crash, hang) label May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@miekg