EXT4-fs Errors in /var/log/messages

[mrwister]

Hi,

We are running Flashcache on a server with the following disk configuration:

2 x SATA 2TB Drives -> 1 x Linux RAID0 -> Flashcache SSD (120GB)

So we have 2x SATA drives, made into a Linux software RAID 1 and then Flashcache sitting on top of that.
The server is running a virtualisation kernel (OpenVZ) and was compiled against that source.

In the /var/log/messages log we are often getting messages like:

EXT4-fs error (device dm-3): ext4_mb_generate_buddy: EXT4-fs: group 13888: 23881 blocks in bitmap, 23882 in gd

dm-3 is the SSD caching device. It is set as a caching device without any partitions.

Any ideas?

Many Thanks,
Mark

[mrwister]

Apologies, the RAID0 is actually a RAID1 mirror, so it is:

2 x SATA 2TB Drives -> 1 x Linux RAID1 -> Flashcache SSD (120GB)

[mohans]

Any flashcache errors in /var/log/messages at all ?

We don't use flashcache with ext4, so I don't have direct experience with that configuration, the only thing I can think of is to try mounting ext4 "nobarrier".

[mrwister]

Hi mohans,

Nothing flashcache specific in /var/log/messages, butI've got the following in the boot log:

[ 23.628665] flashcache: flashcache-1.0 initialized
[ 23.633676] device-mapper: flashcache: Slow (clean) Shutdown Detected
[ 23.633681] Only CLEAN blocks exist in cache
[ 23.633684] device-mapper: flashcache: flashcache_writeback_load: md_blocks = 114028, md_sectors = 912224, md_block_size = 8
[ 23.633687]
[ 23.633691] device-mapper: flashcache: Allocate 456104KB (16B per) mem for 29190656-entry cache(capacity:114471MB, associativity:512, block size:8 sectors(4KB))
[ 26.083579] device-mapper: flashcache: flashcache_writeback_load: Cache metadata loaded from disk with 1259735 valid 0 DIRTY blocks
[ 26.530224] EXT4-fs (dm-3): warning: mounting fs with errors, running e2fsck is recommended
[ 26.617328] EXT4-fs (dm-3): mounted filesystem with ordered data mode
[ 37.467823] 802.1Q VLAN Support v1.8 Ben Greear greearb@candelatech.com
[ 37.467830] All bugs added by David S. Miller davem@redhat.com
[ 37.467845] 8021q: adding VLAN 0 to HW filter on device eth0
[ 37.467921] 8021q: adding VLAN 0 to HW filter on device eth1
[ 40.226720] Slow work thread pool: Starting up
[ 40.226816] Slow work thread pool: Ready
[ 40.226874] FS-Cache: Loaded
[ 40.267522] FS-Cache: Netfs 'nfs' registered for caching
[ 40.479774] warning: `vzctl' uses 32-bit capabilities (legacy support in use)
[ 40.493144] CT: 1: started
[ 80.488562] CT: 49: started
[ 81.628496] EXT4-fs error (device dm-3): ext4_mb_generate_buddy: EXT4-fs: group 1952: 23987 blocks in bitmap, 23988 in gd
[ 81.642487] JBD: Spotted dirty metadata buffer (dev = dm-3, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
[ 81.744393] CT: 51: started
[ 95.090157] readahead-collector: starting delayed service auditd
[ 95.614164] readahead-collector: sorting
[ 95.617661] EXT4-fs error (device dm-3): ext4_mb_generate_buddy: EXT4-fs: group 1: 0 blocks in bitmap, 1 in gd
[ 95.619149] JBD: Spotted dirty metadata buffer (dev = dm-3, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
[ 96.037651] readahead-collector: finished
[ 9283.653717] EXT4-fs error (device dm-3): ext4_mb_generate_buddy: EXT4-fs: group 13888: 23881 blocks in bitmap, 23882 in gd
[ 9283.695134] JBD: Spotted dirty metadata buffer (dev = dm-3, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
[ 9283.755832] JBD: Spotted dirty metadata buffer (dev = dm-3, blocknr = 0). There's a risk of filesystem corruption in case of system crash.

dm-3 is the FlashCache SSD.

/proc/flashcache//flashcache_errors is not showing any errors.

Thanks,
Mark

[mohans]

You did a clean shutdown which cleaned every dirty block to disk before removing
the flashcache device. So there should be nothing uncommitted on the ssd.

I don't however see any ext4 mount messages that indicate that IO barriers are
not used. Can you try to unmount, fsck the volume (the flashcache device),
remount explicitly with nobarrier and see what happens ?

---

From: mrwister
reply@reply.github.com

To: Mohan Srinivasan mohan_srinivasan@yahoo.com
Sent: Wed, November 23, 2011 11:49:41 AM
Subject: Re: [flashcache] EXT4-fs Errors in /var/log/messages (#40)

Hi mohans,

Nothing flashcache specific in /var/log/messages, butI've got the following in
the boot log:

[ 23.628665] flashcache: flashcache-1.0 initialized
[ 23.633676] device-mapper: flashcache: Slow (clean) Shutdown Detected
[ 23.633681] Only CLEAN blocks exist in cache
[ 23.633684] device-mapper: flashcache: flashcache_writeback_load: md_blocks =
114028, md_sectors = 912224, md_block_size = 8
[ 23.633687]
[ 23.633691] device-mapper: flashcache: Allocate 456104KB (16B per) mem for
29190656-entry cache(capacity:114471MB, associativity:512, block size:8
sectors(4KB))
[ 26.083579] device-mapper: flashcache: flashcache_writeback_load: Cache
metadata loaded from disk with 1259735 valid 0 DIRTY blocks
[ 26.530224] EXT4-fs (dm-3): warning: mounting fs with errors, running e2fsck
is recommended
[ 26.617328] EXT4-fs (dm-3): mounted filesystem with ordered data mode
[ 37.467823] 802.1Q VLAN Support v1.8 Ben Greear greearb@candelatech.com
[ 37.467830] All bugs added by David S. Miller davem@redhat.com
[ 37.467845] 8021q: adding VLAN 0 to HW filter on device eth0
[ 37.467921] 8021q: adding VLAN 0 to HW filter on device eth1
[ 40.226720] Slow work thread pool: Starting up
[ 40.226816] Slow work thread pool: Ready
[ 40.226874] FS-Cache: Loaded
[ 40.267522] FS-Cache: Netfs 'nfs' registered for caching
[ 40.479774] warning: `vzctl' uses 32-bit capabilities (legacy support in use)
[ 40.493144] CT: 1: started
[ 80.488562] CT: 49: started
[ 81.628496] EXT4-fs error (device dm-3): ext4_mb_generate_buddy: EXT4-fs:
group 1952: 23987 blocks in bitmap, 23988 in gd
[ 81.642487] JBD: Spotted dirty metadata buffer (dev = dm-3, blocknr = 0).
There's a risk of filesystem corruption in case of system crash.
[ 81.744393] CT: 51: started
[ 95.090157] readahead-collector: starting delayed service auditd
[ 95.614164] readahead-collector: sorting
[ 95.617661] EXT4-fs error (device dm-3): ext4_mb_generate_buddy: EXT4-fs:
group 1: 0 blocks in bitmap, 1 in gd
[ 95.619149] JBD: Spotted dirty metadata buffer (dev = dm-3, blocknr = 0).
There's a risk of filesystem corruption in case of system crash.
[ 96.037651] readahead-collector: finished
[ 9283.653717] EXT4-fs error (device dm-3): ext4_mb_generate_buddy: EXT4-fs:
group 13888: 23881 blocks in bitmap, 23882 in gd
[ 9283.695134] JBD: Spotted dirty metadata buffer (dev = dm-3, blocknr = 0).
There's a risk of filesystem corruption in case of system crash.
[ 9283.755832] JBD: Spotted dirty metadata buffer (dev = dm-3, blocknr = 0).
There's a risk of filesystem corruption in case of system crash.

dm-3 is the FlashCache SSD.

/proc/flashcache//flashcache_errors is not showing any errors.

Thanks,
Mark

---

Reply to this email directly or view it on GitHub:
#40 (comment)

[mrwister]

I'll try to implement your suggestions, but the sever is running live services and I will have to schedule downtime.
In the meantime, I'll setup a test server and attempt to replicate the error, then try turning off IO barriers, etc.

Thanks,
Mark

[mrwister]

Hi mohans,

Is there a way to get the Flashcache to resync with the underlying data on an existing partition?
e.g.

a) Unmount/destroy an existing Flashcache device
b) Mount the underlying disk device as a normal partition
c) Write new data to the underlying partition
d) Create a new flashcahe device for the underlying disk device

At present when I create a flashcache device against an existing partition, any existing data on that partition is not visible.

Thanks,
Mark

[mohans]

Flashcache only faults in blocks into the cache as they are read or written so
the only way to get existing blocks in a partition into the cache after a
flashcache create would be to actually read them in.

---

From: mrwister
reply@reply.github.com

To: Mohan Srinivasan mohan_srinivasan@yahoo.com
Sent: Thu, November 24, 2011 8:29:54 AM
Subject: Re: [flashcache] EXT4-fs Errors in /var/log/messages (#40)

Hi mohans,

Is there a way to get the Flashcache to resync with the underlying data on an
existing partition?
e.g.

a) Unmount/destroy an existing Flashcache device
b) Mount the underlying disk device as a normal partition
c) Write new data to the underlying partition
d) Create a new flashcahe device for the underlying disk device

At present when I create a flashcache device against an existing partition, any
existing data on that partition is not visible.

Thanks,
Mark

---

Reply to this email directly or view it on GitHub:
#40 (comment)

[tony3]

Correct me if I am wrong, Mark, but I think you are asking if you can create a FC device that caches an existing partition. The answer is definitely yes. My setup is almost identical to yours (2x 2TB drives with a 120GB SSD.) That is how I created my cached drives (on-the-fly from the running OS using the "lazy" create patch.) I was going to document it in a post here, but it seemed like a better idea to create a wiki entry for the entire creation process:

https://github.com/facebook/flashcache/wiki/QuickStart-Recipe-for-Ubuntu-11.10

Look out for the bug that I encountered while MD was checking the drives (issue 39.) All disk I/O froze on the machine and I had to reboot to regain use of it. Mohans, any idea why that might have happened or troubleshooting tips for isolating the problem? I had full access to /sys and /proc, but I was limited to binaries/libraries that had already been read from disk. If I tried to access anything not in memory, bash blocked waiting for disk I/O.

[mrwister]

Thanks for the info.
Is it safe to mount the SSD with nobarrier?

[mohans]

Tony - We don't use md here at all, and I have not had time to reproduce the
problem with md (IO hang while md does data checking/reconstruct) yet :(

---

From: tony3
reply@reply.github.com

To: Mohan Srinivasan mohan_srinivasan@yahoo.com
Sent: Tue, November 29, 2011 12:26:29 AM
Subject: Re: [flashcache] EXT4-fs Errors in /var/log/messages (#40)

Correct me if I am wrong, Mark, but I think you are asking if you can create a
FC device that caches an existing partition. The answer is definitely yes. My
setup is almost identical to yours (2x 2TB drives with a 120GB SSD.) That is how
I created my cached drives (on-the-fly from the running OS using the "lazy"
create patch.) I was going to document it in a post here, but it seemed like a
better idea to create a wiki entry for the entire creation process:

https://github.com/facebook/flashcache/wiki/QuickStart-Recipe-for-Ubuntu-11.10

Look out for the bug that I encountered while MD was checking the drives (issue
39.) All disk I/O froze on the machine and I had to reboot to regain use of it.
Mohans, any idea why that might have happened or troubleshooting tips for
isolating the problem? I had full access to /sys and /proc, but I was limited
to binaries/libraries that had already been read from disk. If I tried to
access anything not in memory, it blocked waiting for disk I/O.

---

Reply to this email directly or view it on GitHub:
#40 (comment)

[mohans]

It should be safe to mount ext4 with nobarrier. This might result in some loss
of performance.

---

From: mrwister
reply@reply.github.com

To: Mohan Srinivasan mohan_srinivasan@yahoo.com
Sent: Tue, November 29, 2011 5:13:50 AM
Subject: Re: [flashcache] EXT4-fs Errors in /var/log/messages (#40)

Thanks for the info.
Is it safe to mount the SSD with nobarrier?

---

Reply to this email directly or view it on GitHub:
#40 (comment)

[tony3]

Thanks, Mohans. I assumed by the delay in response that you didn't use md devices. If there is any info that you think would be useful to have after an I/O freeze, please let me know and I'll try to reproduce the freeze and get the info.

On a side note I thought that nobarrier would increase performance:
http://www.kernel.org/doc/Documentation/filesystems/ext4.txt
"If your disks are battery-backed in one way or another, [which a SSD is basically] disabling barriers may safely improve performance."

[mohans]

Tony - Quick question, have you tried the latest releases (3.1/3.2) to see if
those fix the IO freezes ?

---

From: tony3
reply@reply.github.com

To: Mohan Srinivasan mohan_srinivasan@yahoo.com
Sent: Tue, November 29, 2011 10:56:10 AM
Subject: Re: [flashcache] EXT4-fs Errors in /var/log/messages (#40)

Thanks, Mohans. I assumed by the delay in response that you didn't use md
devices. If there is any info that you think would be useful to have after an
I/O freeze, please let me know and I'll try to reproduce the freeze and get the
info.

On a side note I thought that nobarrier would increase performance:
http://www.kernel.org/doc/Documentation/filesystems/ext4.txt
"If your disks are battery-backed in one way or another, [which a SSD is
basically] disabling barriers may safely improve performance."

---

Reply to this email directly or view it on GitHub:
#40 (comment)

[tony3]

If you're talking about the leading/bleeding-edge kernels, no. I haven't tried them yet. I ran into the bug on two redundant near-production machines and disabled FC immediately after discovery so I could put them into production. FC was the last piece to be stress tested. I might have another pair of machines in a couple of weeks to test with, but right now I am limited to minor changes in the middle of the night to one of the pair. If I knew what to look for after the freeze I would try to hang it again, but right now I don't know enough about the FC code or kernel disk I/O to provide any more data than I have already. Suggestions for what data to collect are of course welcome.

[fimbulvetr]

I have this issue as well.

[20394.040523] EXT4-fs error (device dm-0): ext4_mb_generate_buddy: EXT4-fs: group 197: 2048 blocks in bitmap, 0 in gd

HDD behaves fine when mounted without dm/flashcache. I will have only a limited to time to test this if you want me to try anything before these go production. I will run with barriers disabled tomorrow and report back if the error persists.