All IO uncached when caching a partition

[cachemonger]

Hi,

I am trying to use flashcache to cache a partition (/dev/sdb1) using an SSD (/dev/sdd). Creating the cache works just fine, and I mount the new /dev/mapper/flashcache device in place of the original partition (instead of "mount /dev/sdb1 /mnt/target" I use "mount /dev/mapper/flashcache /mnt/target").

I see IO going through the flashcache driver when I watch the stats in /proc/flashcache/../flashcache_stats, but everything is listed as "uncached". I've double checked that I have cache_all enabled, and it is not set to filter sequential IO.

Everything works fine if I use the underlying /dev/sdb device instead (no partitions; create ext4 filesystem on /dev/sdb). If I do this, I see that IO is being cached.

The only thing I can think of is that sdb1 starts on the 63rd sector of sdb, and this is not 4K-aligned. Is this an unsupported configuration, or am I doing something else wrong?

Thanks!

[mohans]

I think the alignment is the issue. Because the start is misaligned, off a 4K
boundary, all IO is misaligned. The DM therefore breaks incoming IOs into
smaller than 4K, which end up being uncached by flashcache.

If you look at the IO histogram reported by dmsetup table, you can see the sizes
of the incoming IOs. Flashcache will only cache IOs that are exactly 4KB.

Can you try and force the alignment of sdb1 at a 4K boundary and retest ?

---

From: cachemonger
reply@reply.github.com

To: Mohan Srinivasan mohan_srinivasan@yahoo.com
Sent: Mon, December 19, 2011 3:16:11 PM
Subject: [flashcache] All IO uncached when caching a partition (#43)

Hi,

I am trying to use flashcache to cache a partition (/dev/sdb1) using an SSD
(/dev/sdd). Creating the cache works just fine, and I mount the new
/dev/mapper/flashcache device in place of the original partition (instead of
"mount /dev/sdb1 /mnt/target" I use "mount /dev/mapper/flashcache /mnt/target").

I see IO going through the flashcache driver when I watch the stats in
/proc/flashcache/../flashcache_stats, but everything is listed as "uncached".
I've double checked that I have cache_all enabled, and it is not set to filter
sequential IO.

Everything works fine if I use the underlying /dev/sdb device instead (no
partitions; create ext4 filesystem on /dev/sdb). If I do this, I see that IO is
being cached.

The only thing I can think of is that sdb1 starts on the 63rd sector of sdb, and
this is not 4K-aligned. Is this an unsupported configuration, or am I doing
something else wrong?

Thanks!

---

Reply to this email directly or view it on GitHub:
#43

[cachemonger]

Hi Mohans,

I'm not sure what you're asking. Do you want me to move the partition?

[mohans]

Can you paste the output of 'dmsetup table' ? If you see sub-4K IOs coming in,
that's an indicator that the IOs coming into DM are misaligned (off 4K
boundaries), causing the DM to break up the IOs into sub-4K IOs which will all
be uncached by flashcache. Before you do anything, it would be good to see the
histogram of IO sizes as seen by flashcache first.

---

From: cachemonger
reply@reply.github.com

To: Mohan Srinivasan mohan_srinivasan@yahoo.com
Sent: Wed, December 21, 2011 9:35:46 AM
Subject: Re: [flashcache] All IO uncached when caching a partition (#43)

Hi Mohans,

I'm not sure what you're asking. Do you want me to move the partition?

---

Reply to this email directly or view it on GitHub:
#43 (comment)

[ghost]

Hi all,
I have the same problem, everything seems pretty much aligned. Here are my stats, would appreciate any help.

# dmsetup table
fc-md: 0 2147481328 flashcache conf:
        ssd dev (/dev/sda), disk dev (/dev/disk/by-uuid/a6e4f3d2-12d1-41ae-b2f4-2f6782dab2c8) cache mode(WRITE_BACK)
        capacity(112640M), associativity(512), data block size(2048K) metadata block size(4096b)
        skip sequential thresh(0K)
        total blocks(56320), cached blocks(0), cache percent(0)
        dirty blocks(0), dirty percent(0)
        nr_queued(0)
Size Hist: 1024:1 4096:210445 8192:2165 12288:518 16384:2102 

# cat /proc/flashcache/flashcache_version 
Flashcache Version : flashcache-1.0
git commit: 1.0-139-g46eb0a330bcd

# dmsetup status
fc-md: 0 2147481328 flashcache stats: 
        reads(34015), writes(169867)
        read hits(0), read hit percent(0)
        write hits(0) write hit percent(0)
        dirty write hits(0) dirty write hit percent(0)
        replacement(0), write replacement(0)
        write invalidates(0), read invalidates(0)
        pending enqueues(0), pending inval(0)
        metadata dirties(0), metadata cleans(0)
        metadata batch(0) metadata ssd writes(0)
        cleanings(0) fallow cleanings(0)
        no room(0) front merge(0) back merge(0)
        disk reads(34022), disk writes(169893) ssd reads(0) ssd writes(0)
        uncached reads(34022), uncached writes(169893), uncached IO requeue(0)
        uncached sequential reads(0), uncached sequential writes(0)
        pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)

[mohans]

Why have you configured flashcache with a 2048K blocksize ?

    capacity(112640M), associativity(512), data block size(2048K) 

A 2MB blocksize, according to dmsetup status. This will certainly lead to all
IOs being non-cacheable.

Can you configure flashcache with the default 4KB block size ?

---

From: Mikhail Sayapin
reply@reply.github.com

To: Mohan Srinivasan mohan_srinivasan@yahoo.com
Sent: Mon, March 12, 2012 7:00:11 AM
Subject: Re: [flashcache] All IO uncached when caching a partition (#43)

Hi all,
I have the same problem, everything seems pretty much aligned. Here are my
stats, would appreciate any help.

# dmsetup table
fc-md: 0 2147481328 flashcache conf:
        ssd dev (/dev/sda), disk dev 
(/dev/disk/by-uuid/a6e4f3d2-12d1-41ae-b2f4-2f6782dab2c8) cache mode(WRITE_BACK)
        capacity(112640M), associativity(512), data block size(2048K) metadata 
block size(4096b)
        skip sequential thresh(0K)
        total blocks(56320), cached blocks(0), cache percent(0)
        dirty blocks(0), dirty percent(0)
        nr_queued(0)
Size Hist: 1024:1 4096:210445 8192:2165 12288:518 16384:2102 

# cat /proc/flashcache/flashcache_version 
Flashcache Version : flashcache-1.0
git commit: 1.0-139-g46eb0a330bcd

# dmsetup status
fc-md: 0 2147481328 flashcache stats: 
        reads(34015), writes(169867)
        read hits(0), read hit percent(0)
        write hits(0) write hit percent(0)
        dirty write hits(0) dirty write hit percent(0)
        replacement(0), write replacement(0)
        write invalidates(0), read invalidates(0)
        pending enqueues(0), pending inval(0)
        metadata dirties(0), metadata cleans(0)
        metadata batch(0) metadata ssd writes(0)
        cleanings(0) fallow cleanings(0)
        no room(0) front merge(0) back merge(0)
        disk reads(34022), disk writes(169893) ssd reads(0) ssd writes(0)
        uncached reads(34022), uncached writes(169893), uncached IO requeue(0)
        uncached sequential reads(0), uncached sequential writes(0)
        pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)

---

Reply to this email directly or view it on GitHub:
#43 (comment)

[ghost]

Hello,
I did almost everything according to https://github.com/facebook/flashcache/wiki/QuickStart-Recipe-for-Ubuntu-11.10, but it seems there is an error in "-b 4096" param as "b" is in sectors, and 4096 sectors is 2048K. I will try to reconfig, thanks for pointing that out.

[mohans]

Hmm may be a parsing bug in the utility. Need to fix it.

Sent from my iPhone

On Mar 12, 2012, at 9:20 PM, Mikhail Sayapin reply@reply.github.com wrote:

Hello,
I did almost everything according to https://github.com/facebook/flashcache/wiki/QuickStart-Recipe-for-Ubuntu-11.10, but it seems there is an error in "-b 4096" param as "b" is in sectors, and 4096 sectors is 2048K. I will try to reconfig, thanks to pointing that out.

---

Reply to this email directly or view it on GitHub:
#43 (comment)

[ghost]

It worked. So, when creating device with flashcache_create and -b 4096, it makes block size equal to 4096 sectors, or 2/16 MB depending on your device sector size. When using -b 4k (which seems by default), block size is 4096 bytes. I corrected the wiki command. Thanks!