RGW-NFS: Use rados cluster_stat to report filesystem usage

[supriti]

Partially fixes: http://tracker.ceph.com/issues/22202

Signed-off-by: Supriti Singh supriti.singh@suse.com

[supriti]

@mattbenjamin please review

[mattbenjamin]

when I ran this change under gdb, I naively checked the value of stats after executing RGWGetClusterStatReq, and found it to be all-0. I have objects in this cluster, it's over a week old. Can you think of a reason for that?

Thread 1 "ganesha.nfsd" hit Breakpoint 2, rgw_statfs (rgw_fs=, parent_fh=, vfs_st=0x7fffffffd490, flags=)
at /home/mbenjamin/ceph-noob/src/rgw/rgw_file.cc:1630
1630 if (rc < 0) {
(gdb) p rc
$1 =
(gdb) list
1625 RGWLibFS fs = static_cast<RGWLibFS>(rgw_fs->fs_private);
1626 struct rados_cluster_stat_t stats;
1627
1628 RGWGetClusterStatReq req(fs->get_context(), fs->get_user(),stats);
1629 int rc = rgwlib.get_fe()->execute_req(&req);
1630 if (rc < 0) {
1631 lderr(fs->get_context()) << "ERROR: getting total cluster usage"
1632 << cpp_strerror(-rc) << dendl;
1633 return rc;
1634 }

(gdb) n
1644 vfs_st->f_bsize = 1 << CEPH_BLOCK_SHIFT;
(gdb) p stats
$3 = {kb = 0, kb_used = 0, kb_avail = 0, num_objects = 0}

[mattbenjamin]

It doesn't seem like we should set a blocksize of 4M as of now; 1M is the most common client default for rwsize; separately, you have reported problems using this value. in upcoming nfs writeback changeset, I dimension into 4M extents == chunks, but subdivide each extent into 1M pages

[mattbenjamin]

I have a better source for this constant later (in extent package), good to keep it at local scope; can we make it constexpr? can we make it uint32_t?

[mattbenjamin]

space before stats

[supriti]

@mattbenjamin
I was testing using vstart and running a ganesha instance pointing to vstart ceph.conf. Running "df -h" on mount point shows the usage.

I also ran with gdb. I started ganesha process, and attached gdb to it. I set a breakpoint at rgw_statfs.
I can see the right stats in stack trace.
gdb) l 1619 get filesystem attributes 1620 */ 1621 int rgw_statfs(struct rgw_fs *rgw_fs, 1622 struct rgw_file_handle *parent_fh, 1623 struct rgw_statvfs *vfs_st, uint32_t flags) 1624 { 1625 RGWLibFS *fs = static_cast<RGWLibFS*>(rgw_fs->fs_private); 1626 struct rados_cluster_stat_t stats; 1627 1628 RGWGetClusterStatReq req(fs->get_context(), fs->get_user(), stats); (gdb) n 1625 RGWLibFS *fs = static_cast<RGWLibFS*>(rgw_fs->fs_private); (gdb) n 1628 RGWGetClusterStatReq req(fs->get_context(), fs->get_user(), stats); (gdb) n 1625 RGWLibFS *fs = static_cast<RGWLibFS*>(rgw_fs->fs_private); (gdb) n 1628 RGWGetClusterStatReq req(fs->get_context(), fs->get_user(), stats); (gdb) n 1629 int rc = rgwlib.get_fe()->execute_req(&req); (gdb) n 1628 RGWGetClusterStatReq req(fs->get_context(), fs->get_user(), stats); (gdb) p stats $1 = {kb = 140405592657808, kb_used = 4416878, kb_avail = 0, num_objects = 29362600} (gdb)

[supriti]

@mattbenjamin addressed your comments and submitted new patch. Please check

[mattbenjamin]

lgtm--I'll re-test later today, should be fine

[supriti]

@mattbenjamin ping. Were you able to test this patch.

[mattbenjamin]

@supriti I've retested. In my environment, I continue to see 0 values reported from rados::cluster_stat()--having said that, I see nothing wrong with the logic being executed, and I guess I have to assume my test cluster is reporting 0-stats.

[mattbenjamin]

@supriti this works beautifully; note to self: don't forget to run ceph-mgr ;)