bluestore-block File size automatically grows, the actual only 20G, but a single node bluestore-block size of more than 80G

[limiao2008]

limisaodeMacBook-Pro:~ limisao$ kubectl get pvc
NAME           STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
redis-master   Bound     pvc-f7122b7b-e180-11e7-a028-f23c9197deda   5Gi        RWO            rook-block     46d
redis-slave    Bound     pvc-67f51b96-e183-11e7-a028-f23c9197deda   5Gi        RWO            rook-block     46d

# du -sh *
4.0K	block
4.0K	block.db
4.0K	block.wal
8.0K	bluefs
79G	bluestore-block
27M	bluestore-db
576M	bluestore-wal
8.0K	ceph_fsid
8.0K	fsid
8.0K	keyring
8.0K	kv_backend
8.0K	magic
8.0K	mkfs_done
8.0K	ready
4.0K	rook-osd.3.asok
8.0K	rook.config
16K	tmp
8.0K	type
8.0K	whoami

[kokhang]

Can you give us more information about your environment? How many nodes in the cluster? Also what is your cluster manifest? kubectl -n rook get cluster rook -o yaml?

[jbw976]

Just as a note, the bluestore-block file, when the user has specified a directory for the OSD, is "thinly provisioned" up front. i.e. the file descriptor is set to the full size of the disk, but blocks for the file are NOT allocated up front. You'll see the file size be much bigger than will actually be reported by du -sh bluestore-block.

What does the following report when run from the OSD's filesystem?

du -sh bluestore-block; ls -hal bluestore-block

Also, what does the following look like when run from inside the rook toolbox?

rookctl status; ceph status

[lorenz]

I have one cluster that runs on the same thinly provisioned files and heavy write loads (70% write) cause the bluestore-rook file to blow up to 5 times the used data volume according to ceph status. I don't think that this can be solved easily since it would basically rely on TRIM/UNMAP being passed through to the filesystem layer and then freeing up these blocks.

[oliviabarrick]

Why is this still completely undocumented on the website? Bluestore should not be the default.

[travisn]

Agreed, will make it priority to get this documented clearly with a warning. Bluestore in a directory was intended to be a test scenario, but it seems users are finding it useful beyond that.

Upstream, bluestore in a directory was also intended to be for testing scenarios rather than production. A change was just merged upstream that will improve this situation of the space never being reclaimed: ceph/ceph#14727. However, it still won't entirely solve the issue and Rook won't get the change until we pick up Mimic.

Using filestore for directories in the examples also seems like a good thing for now unless there is some other issue with filestore i'm forgetting. @jbw976 @liewegas

[lorenz]

@travisn Are you sure that is the case? It looks like this will only work on block devices, not necessarily on filesystems.

[travisn]

yeah, i'm not sure if this would help with the filestore

[hameno]

How do I change from bluestore to filestore if a cluster is already created?

[travisn]

@hameno it will require bringing up new OSDs and then removing the old ones after the data has been replicated. The Ceph documentation describes the process here. Unfortunately there is not an in-place migration for individual OSDs.

[hameno]

Thanks, that's basically what I ended up doing.

[Paxa]

How to check what is current backend, bluestore or filestore?