os/bluestore: add discard method for ssd's performance #14727

voidbag · 2017-04-22T06:47:54Z

discard method is added to BlockDevice.h for ssd's performance
For ssd's performance, discard should be used, when BlueStore or BlueFS releases block device's area.

Signed-off-by: Taeksang Kim voidbag@gmail.com

liewegas · 2017-04-25T15:52:27Z

We should add a bdev_discard bool config option to enable/disable this, as discard is bad on some devices.

Also, teh bluestore piece is more complicated. The existing discard op is synchronous.. we don't want to call it where you have it. We either need a thread doing these, or to use aio.

And we probably need to add some ordering infrastructure to make sure that a deallocate+discard doesn't run concurrently with a new allocation and write into that space. I'm not sure exactly what the rules are, but at a minimum we need to make sure the discard aio completes before allowing the space to be reallocated (i.e., discard before release to allocator).

Note that the last time I had a conversation with @allensamuels (sandisk) about this we decided it wasn't that important that we worry about discards. I forget what the reason was.

varadakari · 2017-05-19T10:22:13Z

This needs a rebase.

varadakari · 2017-05-19T10:26:26Z

Not all the vendors implement discard/trim mechanism, they can take the request and drop/not implement in the firmware level. But it is generally good idea to pass the discards to help the gc process. As Sage mentioned we need to measure the cost of sync calls vs async, some test results if possible on a aged device with gc running.

voidbag · 2017-05-19T10:33:43Z

@varadakari
I'll rebase it
I had a lot of things to do...
I will update it based on Sage 's comment until next week.
Sorry for the delay.

Thanks

wido · 2017-06-27T11:46:41Z

@voidbag: I asked on the ML and was pointed to this PR.

Is there anything else you need to add/fix before this can be reviewed and maybe merged?

voidbag · 2017-06-29T00:14:33Z

@wido
I think i have done the implementation.
The remaining thing is to fix the failure of the test, which u and other can point out.
I'm also trying to find the cause of the failure of test.

Thanks

wido · 2017-09-11T12:41:27Z

So this one is still pending?

liewegas · 2017-09-11T17:09:52Z

Yeah, I'm still nervous about it. Two things: 1. The special async interface for discard, separate from teh aio one, rubs me the wrong way, although it's probaby justified since you can't submit discard requests via libaio 2. I'm not sure this will actually be a good thing to do on a real SSD. For XFS most seem to recommend using fstrim periodically instead of online discard because so many firmwares suck and there are often performance anomalies (or bugs) from lots of small discards. Even if that isn't the case, if we are discarding small bits as we free them, they may be dropped by teh SSD such that we don't trim the larger region they eventuallly form. I suppose we should probably rebase this, and do some testing, but leave it off by default (maybe with the option marked DEV or something so that there is some warning for users?)?

voidbag · 2017-09-11T21:28:35Z

I'll rebase this pr and check the issues you all pointed out, until this weekend

@liewegas I think fstrim is for just local filesystem unlike BlueFS using raw block device.
So i think BlueFS has to issue discard command to raw block device like other filesystem.

liewegas · 2017-09-12T00:50:58Z

Right about fstrim. I mean that it is a periodic operation that reexamines the freelist and trims it, versus an online approach that trims as we free things. The offline one will submit fewer trims and they should (as well as the device allows) regardless of the size of the individual released extents.

liewegas · 2017-09-12T00:53:03Z

FWIW I think we will want to have an fstrim-like function either way in order to discard upgraded bluestores. One it's there we can compare results of a periodic approach vs the online one. I think the StupidAllocot in-memory structure should make it reasonably easy to implement since it already orders/batches free extents by size; we can focus on discarding the big extents first (or only).

wido · 2017-09-12T06:39:59Z

@liewegas The problem with mounting EXT4 or XFS with the discard option isn't that discarding is the problem, but it's SATA 3.0

The SATA 3.1 spec brings queued trimming which overcomes these problems. Many controllers out there however only implement SATA 3.0.

A few resources:

So yes, off by default and let the user enable it if the need to. Might need to write a log line telling them that in certain scenarios having discard on can degrade performance.

voidbag · 2017-09-18T21:15:35Z

src/os/bluestore/BlueFS.cc

    for (auto p = to_release[i].begin(); p != to_release[i].end(); ++p) {
+      if (cct->_conf->bdev_enable_discard && r == 0)
+        bdev[i]->discard(p.get_start(), p.get_len());


I think this code is also wrong, because r is undefined in that scope...
The intention of the code is that it discards given range which is failed to discard async, synchronously.

And i don't understand your(@liewegas) comment "This doesn't seem right.. first, if you discard async, you do'nt want ot later discard() sync, or call release(), right?"
Does it mean 'Don't discard synchronously and don't call release(), if discard has been done async'
Am i right??

That's a comment I deleted after I looked harder and understood the code :)

voidbag · 2017-09-18T21:22:38Z

@liewegas
I read all your comments.
I'll briefly describe what I have to do according to your comments.
Step1: Rebase this, fix bugs and add options whose postfix is _dev (This PR)
Step2: Implement periodic discard comment using stupid alloc with only big extents. (new PR)
Step3: Implement fstrim function with every extents like other filesystem (xfs..) (new PR)

Am I right? Plz correct me if i made a mistake.

liewegas · 2017-09-18T21:31:59Z

I think 3 isn't needed; just 2!

voidbag · 2017-09-18T22:17:56Z

Okay, then,
First, i tried to add _dev options in this PR

wido · 2017-09-19T06:33:09Z

src/common/config_opts.h

@@ -1024,6 +1024,8 @@ OPTION(bdev_debug_aio_suicide_timeout, OPT_FLOAT, 60.0)
 // NVMe driver is loaded while osd is running.
 OPTION(bdev_nvme_unbind_from_kernel, OPT_BOOL, false)
 OPTION(bdev_nvme_retry_count, OPT_INT, -1) // -1 means by default which is 4
+OPTION(bdev_enable_discard, OPT_BOOL, true)


Shouldn't these be set to false?

wido · 2017-11-03T12:18:38Z

Do we have an update on this PR?

There are more SATA 3.1 and SAS 3.0 SSDs (Samsung PM1633a) out there which async queue TRIM so it can be 'sync' in the code.

Would like to see this in BlueStore

k0ste · 2018-01-24T05:44:36Z

I'm properly understand that if bdev_enable_discard is enabled TRIM command issued for: wal, db and db_slow (backend) device. What if backend device is HDD?

wido · 2018-01-24T08:40:27Z

@k0ste Yes, it does. I think we can skip (or make it configurable) TRIM for db_slow as that's usually on a HDD or non-TRIM device.

k0ste · 2018-01-24T12:41:33Z

@wido yes we should. I think is_rotational function can be used for it.
@voidbag please take a look. TRIM should not be issued if backend device is HDD.

wido · 2018-01-24T14:21:32Z

@k0ste Good suggestion! @voidbag can you look into that?

voidbag · 2018-01-24T14:36:13Z

src/os/bluestore/KernelDevice.cc

+  if (!rotational)
+    r = block_device_discard(fd_direct, (int64_t)offset, (int64_t)len);
+  return r;
+}


@wido @k0ste
I had already implemented that part.
You all don't have to worry about issuing discard command to rotational device.

wido · 2018-01-25T10:20:29Z

I see @voidbag ! You are correct

wido · 2018-02-16T16:00:26Z

@voidbag Can you resolve the conflicts?

After that I hope it's ready to be merged

voidbag · 2018-02-19T23:28:11Z

@wido Rebase has been done.

Could anyone tell me how the bluestore*.yaml file has to be modified?
Do I have to modify existing file or create new one.
If I have to create new one, where do I have to do?

Periodic discard
I have analyzed allocator code(Stupid, Bitmap), roughly.
And I realized that bitmap allocator has fixed range of disk unlike stupid allocator that has only free area.
There isn't any good idea with bitmap allocator in my mind.
Current bad idea is a flag for undiscarded block, which requires full search among discarded and undiscarded blocks.
So,
I'll start to write the periodic discard separating undiscarded and discarded blocks with 'stupid allocator', as soon as this pr is merged.

Thanks

wido · 2018-02-20T08:55:08Z

Thanks! @liewegas might know better, but I think that ./qa/objectstore/bluestore.yaml is sufficient.

Although the testing HW should also support discard in order to make this work.

Would be very nice if this one gets merged. It has been open for a very long time.

liewegas · 2018-02-20T12:25:20Z

Just enable the setting in bluestore.yaml.. that should be sufficient. Thanks!

liewegas · 2018-02-20T12:25:44Z

The smithi testing hardware that we use primarily for rados is all nvme based so this should be exercised.

Discard method is added for ssd's performance. Signed-off-by: Taeksang Kim <voidbag@gmail.com>

discard is added to BlueFS.cc and BlueStore.cc Signed-off-by: Taeksang Kim <voidbag@gmail.com>

Signed-off-by: Taeksang Kim <voidbag@gmail.com>

set bdev_enable_discard and bdev_async_discard true. Signed-off-by: Taeksang Kim <voidbag@gmail.com>

wido · 2018-02-21T09:35:56Z

Great, hopefully the tests are OK so that this one can be merged.

tchaikov · 2018-02-24T04:12:26Z

the failures in the first run were caused by sepia issues.

liewegas added bluestore feature performance labels Apr 25, 2017

voidbag force-pushed the wip-voidbag-bdev-discard branch 5 times, most recently from 997a923 to af4c6e4 Compare May 31, 2017 14:42

voidbag commented Sep 18, 2017

View reviewed changes

wido reviewed Sep 19, 2017

View reviewed changes

voidbag force-pushed the wip-voidbag-bdev-discard branch 4 times, most recently from 1566d64 to e070bd4 Compare November 4, 2017 02:55

voidbag commented Jan 24, 2018

View reviewed changes

voidbag force-pushed the wip-voidbag-bdev-discard branch from 121f00f to fbe2a58 Compare January 24, 2018 14:51

wido approved these changes Jan 25, 2018

View reviewed changes

voidbag force-pushed the wip-voidbag-bdev-discard branch from fbe2a58 to b5e5501 Compare February 8, 2018 12:00

voidbag force-pushed the wip-voidbag-bdev-discard branch from b5e5501 to eadc188 Compare February 19, 2018 18:24

voidbag force-pushed the wip-voidbag-bdev-discard branch 2 times, most recently from 74a007a to b63ea00 Compare February 20, 2018 12:47

liewegas added needs-qa and removed needs-review labels Feb 20, 2018

liewegas approved these changes Feb 20, 2018

View reviewed changes

voidbag added 4 commits February 21, 2018 00:00

os/bluestore/KernelDevice: add discard command

465c21d

Discard method is added for ssd's performance. Signed-off-by: Taeksang Kim <voidbag@gmail.com>

os/bluestore: add discard in BlueFS and BlueStore

2893d09

discard is added to BlueFS.cc and BlueStore.cc Signed-off-by: Taeksang Kim <voidbag@gmail.com>

os/bluestore/KernelDevice: async discard has been implemented.

8a2e1db

Signed-off-by: Taeksang Kim <voidbag@gmail.com>

objectstore/bluestore.yaml: enable bdev discard

b6bd7c9

set bdev_enable_discard and bdev_async_discard true. Signed-off-by: Taeksang Kim <voidbag@gmail.com>

voidbag force-pushed the wip-voidbag-bdev-discard branch from b63ea00 to b6bd7c9 Compare February 20, 2018 15:01

tchaikov added the wip-kefu-testing label Feb 22, 2018

tchaikov merged commit 327c81d into ceph:master Feb 24, 2018

travisn mentioned this pull request Mar 14, 2018

bluestore-block File size automatically grows, the actual only 20G, but a single node bluestore-block size of more than 80G rook/rook#1463

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

os/bluestore: add discard method for ssd's performance #14727

os/bluestore: add discard method for ssd's performance #14727

voidbag commented Apr 22, 2017 •

edited

liewegas commented Apr 25, 2017

varadakari commented May 19, 2017

varadakari commented May 19, 2017

voidbag commented May 19, 2017

wido commented Jun 27, 2017

voidbag commented Jun 29, 2017

wido commented Sep 11, 2017

liewegas commented Sep 11, 2017 via email

voidbag commented Sep 11, 2017

liewegas commented Sep 12, 2017

liewegas commented Sep 12, 2017

wido commented Sep 12, 2017

voidbag Sep 18, 2017

liewegas Sep 18, 2017

voidbag commented Sep 18, 2017

liewegas commented Sep 18, 2017

voidbag commented Sep 18, 2017

wido Sep 19, 2017

wido commented Nov 3, 2017

k0ste commented Jan 24, 2018

wido commented Jan 24, 2018

k0ste commented Jan 24, 2018

wido commented Jan 24, 2018

voidbag Jan 24, 2018

wido commented Jan 25, 2018

wido commented Feb 16, 2018

voidbag commented Feb 19, 2018 •

edited

wido commented Feb 20, 2018

liewegas commented Feb 20, 2018

liewegas commented Feb 20, 2018

wido commented Feb 21, 2018

tchaikov commented Feb 24, 2018

os/bluestore: add discard method for ssd's performance #14727

os/bluestore: add discard method for ssd's performance #14727

Conversation

voidbag commented Apr 22, 2017 • edited

liewegas commented Apr 25, 2017

varadakari commented May 19, 2017

varadakari commented May 19, 2017

voidbag commented May 19, 2017

wido commented Jun 27, 2017

voidbag commented Jun 29, 2017

wido commented Sep 11, 2017

liewegas commented Sep 11, 2017 via email

voidbag commented Sep 11, 2017

liewegas commented Sep 12, 2017

liewegas commented Sep 12, 2017

wido commented Sep 12, 2017

voidbag Sep 18, 2017

Choose a reason for hiding this comment

liewegas Sep 18, 2017

Choose a reason for hiding this comment

voidbag commented Sep 18, 2017

liewegas commented Sep 18, 2017

voidbag commented Sep 18, 2017

wido Sep 19, 2017

Choose a reason for hiding this comment

wido commented Nov 3, 2017

k0ste commented Jan 24, 2018

wido commented Jan 24, 2018

k0ste commented Jan 24, 2018

wido commented Jan 24, 2018

voidbag Jan 24, 2018

Choose a reason for hiding this comment

wido commented Jan 25, 2018

wido commented Feb 16, 2018

voidbag commented Feb 19, 2018 • edited

wido commented Feb 20, 2018

liewegas commented Feb 20, 2018

liewegas commented Feb 20, 2018

wido commented Feb 21, 2018

tchaikov commented Feb 24, 2018

voidbag commented Apr 22, 2017 •

edited

voidbag commented Feb 19, 2018 •

edited