zfs volume full end_request: I/O error, dev zdXXX, sector #####

[byteharmony]

When a zpool gets full i get a dmesg full of this:

end_request: I/O error, dev zd800, sector 137022848
end_request: I/O error, dev zd944, sector 6471792
end_request: I/O error, dev zd736, sector 848976
end_request: I/O error, dev zd656, sector 19465072
end_request: I/O error, dev zd752, sector 2041424
end_request: I/O error, dev zd800, sector 137022944
end_request: I/O error, dev zd800, sector 137022912
end_request: I/O error, dev zd656, sector 19467384
end_request: I/O error, dev zd800, sector 137023872
__ratelimit: 22 callbacks suppressed
end_request: I/O error, dev zd656, sector 3151856
end_request: I/O error, dev zd576, sector 14902210
end_request: I/O error, dev zd800, sector 137024224
end_request: I/O error, dev zd736, sector 7178472
end_request: I/O error, dev zd752, sector 2041416
end_request: I/O error, dev zd784, sector 146703232
end_request: I/O error, dev zd576, sector 14902211
end_request: I/O error, dev zd704, sector 59863536
end_request: I/O error, dev zd736, sector 1823472
end_request: I/O error, dev zd656, sector 19467384
end_request: I/O error, dev zd752, sector 72875976
end_request: I/O error, dev zd800, sector 137024192
end_request: I/O error, dev zd864, sector 29960792
end_request: I/O error, dev zd944, sector 6398608
end_request: I/O error, dev zd736, sector 7007448

Of course I know I never want a full zpool, but is this the expected way to fail? Any idea what will happen on this pool on reboot?

Thanks,
BK

[behlendorf]

You mean when a thin zvol backed is backed by a full zpool you will get these errors. Yes, this is pretty much expected behavior. If you want to ensure this never happens you can set a reservation on the zvol to ensure there is always enough space.

[byteharmony]

Yes, that is the situation, thin zvol, zpool fills. It appears to heal by deleting snapshots which frees space.

I was planning on writing some simple monitoring to address zpool space but before I did that I wanted to throw this out on the list. The errors in the log look like hardware failure when it's really just full zpool. That error message might help someone looking for the issue?

BK

[behlendorf]

@byteharmony The error is generated by the generic block device code in the kernel when a device can't be written to for some reason. That's why it looks like a hardware failure, it's originating from the same code although in this case the device in a zvol instead of a real device. The name is logged in the error message so you can tell them apart, e.g. zdX.