Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance handleops readdirplus operation to return handles along with dirents #174

Closed
soumyakoduri opened this issue Apr 18, 2017 · 6 comments
Closed

Enhance handleops readdirplus operation to return handles along with dirents #174

soumyakoduri opened this issue Apr 18, 2017 · 6 comments
Labels
CB: gfapi FA: Performance Improvements
Milestone
Release 3.11 (STM)

Comments

@soumyakoduri
Copy link
Contributor

Any application using glfs handles (like NFS-Ganesha) may need to do lookup on each dirent returned by readdir/readdirp operation to create handles. But since the lookup in glusterfs is very costly operation (as need to be sent to all replica bricks), as the directory gets larger or the replica count increases, the time taken for single readdir operation to complete can take hours of time. To avoid that we need an extended readdirp API which can return handles along with dirent stat as part of its reply.
@ShyamsundarR ShyamsundarR added this to the Release 3.11 (STM) milestone Apr 26, 2017
@ShyamsundarR ShyamsundarR added this to Release 3.11 in Releases Apr 26, 2017
@ShyamsundarR
Copy link
Contributor

This is being tracked for a week later backport to 3.11 post branching! (May-05)

mscherer pushed a commit that referenced this issue May 2, 2017
@soumyakoduri @jdarcy
gfapi/handleops: Introducing glfs_xreaddirplus_r() fop for handleops
41000cd 
Its known that readdirplus operation fetches stat as well for each of the
dirents. But often applications may need extra information, like for eg.,
NFS-Ganesha which operates on handles needs handles for each of those
dirents returned. So this would require extra calls to the backend, in this
case LOOKUP (which is very expensive operation) resulting in very low
readdir performance.

To address that introducing this new API using which applications can
make request for any extra information to be returned as part of
readdirplus response.

Currently this new api returns stat and handles as demanded by application.
The synopsis of the API is noted in glfs.h.

@todo:
* Enhance test script using this new API

Below were the perf results on single brick volume with and without
these changes -

Dataset used -
10*100 directories and each directory containing 100 empty files.

I used NFS-Ganesha application to test these changes -
>for i in {1..5}; do systemctl restart nfs-ganesha; sleep 10; mount -t nfs -o vers=4 localhost:/brick_vol /mnt; cd /mnt; echo "ITERATION$i"; date; find . > tmp-nfs.log; date; cd /; umount /mnt; sleep 2; done;

Without these changes -
ITERATION1
Mon Mar 20 17:22:26 IST 2017
Mon Mar 20 17:23:18 IST 2017
ITERATION2
Mon Mar 20 17:23:39 IST 2017
Mon Mar 20 17:24:28 IST 2017
ITERATION3
Mon Mar 20 17:24:49 IST 2017
Mon Mar 20 17:25:36 IST 2017
ITERATION4
Mon Mar 20 17:30:57 IST 2017
Mon Mar 20 17:31:37 IST 2017
ITERATION5
Mon Mar 20 17:31:57 IST 2017
Mon Mar 20 17:32:40 IST 2017
[root@dhcp35-197 /]#

On an average ~46.2 sec

With these changes applied -
ITERATION1
Mon Mar 20 17:35:03 IST 2017
Mon Mar 20 17:35:15 IST 2017
ITERATION2
Mon Mar 20 17:35:36 IST 2017
Mon Mar 20 17:35:46 IST 2017
ITERATION3
Mon Mar 20 17:36:06 IST 2017
Mon Mar 20 17:36:17 IST 2017
ITERATION4
Mon Mar 20 17:41:38 IST 2017
Mon Mar 20 17:41:49 IST 2017
ITERATION5
Mon Mar 20 17:42:10 IST 2017
Mon Mar 20 17:42:20 IST 2017

On an average ~10.8 sec

Updates #174
BUG: 1442950
Change-Id: I0f74f74dc62085ca4c4a23c38e3edc84bd850876
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-on: https://review.gluster.org/15663
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
mscherer pushed a commit that referenced this issue May 5, 2017
@soumyakoduri @kshlm
gfapi/handleops: Introducing glfs_xreaddirplus_r() fop for handleops
e7927c9 
Its known that readdirplus operation fetches stat as well for each of the
dirents. But often applications may need extra information, like for eg.,
NFS-Ganesha which operates on handles needs handles for each of those
dirents returned. So this would require extra calls to the backend, in this
case LOOKUP (which is very expensive operation) resulting in very low
readdir performance.

To address that introducing this new API using which applications can
make request for any extra information to be returned as part of
readdirplus response.

Currently this new api returns stat and handles as demanded by application.
The synopsis of the API is noted in glfs.h.

@todo:
* Enhance test script using this new API

Below were the perf results on single brick volume with and without
these changes -

Dataset used -
10*100 directories and each directory containing 100 empty files.

I used NFS-Ganesha application to test these changes -
>for i in {1..5}; do systemctl restart nfs-ganesha; sleep 10; mount -t nfs -o vers=4 localhost:/brick_vol /mnt; cd /mnt; echo "ITERATION$i"; date; find . > tmp-nfs.log; date; cd /; umount /mnt; sleep 2; done;

Without these changes -
ITERATION1
Mon Mar 20 17:22:26 IST 2017
Mon Mar 20 17:23:18 IST 2017
ITERATION2
Mon Mar 20 17:23:39 IST 2017
Mon Mar 20 17:24:28 IST 2017
ITERATION3
Mon Mar 20 17:24:49 IST 2017
Mon Mar 20 17:25:36 IST 2017
ITERATION4
Mon Mar 20 17:30:57 IST 2017
Mon Mar 20 17:31:37 IST 2017
ITERATION5
Mon Mar 20 17:31:57 IST 2017
Mon Mar 20 17:32:40 IST 2017
[root@dhcp35-197 /]#

On an average ~46.2 sec

With these changes applied -
ITERATION1
Mon Mar 20 17:35:03 IST 2017
Mon Mar 20 17:35:15 IST 2017
ITERATION2
Mon Mar 20 17:35:36 IST 2017
Mon Mar 20 17:35:46 IST 2017
ITERATION3
Mon Mar 20 17:36:06 IST 2017
Mon Mar 20 17:36:17 IST 2017
ITERATION4
Mon Mar 20 17:41:38 IST 2017
Mon Mar 20 17:41:49 IST 2017
ITERATION5
Mon Mar 20 17:42:10 IST 2017
Mon Mar 20 17:42:20 IST 2017

On an average ~10.8 sec

This is backport of below upstream patch -
        https://review.gluster.org/15663

>Updates #174
>BUG: 1442950
>Change-Id: I0f74f74dc62085ca4c4a23c38e3edc84bd850876
>Signed-off-by: Soumya Koduri <skoduri@redhat.com>
>Reviewed-on: https://review.gluster.org/15663
>Smoke: Gluster Build System <jenkins@build.gluster.org>
>NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
>Reviewed-by: Niels de Vos <ndevos@redhat.com>
>CentOS-regression: Gluster Build System <jenkins@build.gluster.org>

BUG: 1447571
Change-Id: I0f74f74dc62085ca4c4a23c38e3edc84bd850876
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-on: https://review.gluster.org/17164
Smoke: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
@gluster-ant
Copy link
Collaborator

A patch https://review.gluster.org/17372 has been posted that references this issue.
Commit message: Update release notes of 3.11 for readdirplus enhancements

soumyakoduri added a commit to soumyakoduri/glusterfs that referenced this issue May 24, 2017
@soumyakoduri
Update release notes of 3.11 for readdirplus enhancements
233ee7e 
Updates: gluster#174
Change-Id: I006cafc622c9ee2d776556c287e3ed1743ffa84b
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
@gluster-ant
Copy link
Collaborator

A patch https://review.gluster.org/17372 has been posted that references this issue.
Commit message: Update release notes of 3.11 for readdirplus enhancements

2 similar comments
@gluster-ant
Copy link
Collaborator

A patch https://review.gluster.org/17372 has been posted that references this issue.
Commit message: Update release notes of 3.11 for readdirplus enhancements

@gluster-ant
Copy link
Collaborator

A patch https://review.gluster.org/17372 has been posted that references this issue.
Commit message: Update release notes of 3.11 for readdirplus enhancements

mscherer pushed a commit that referenced this issue May 25, 2017
@soumyakoduri @ShyamsundarR
Update release notes of 3.11 for readdirplus enhancements
078083a 
Updates: #174
Change-Id: I006cafc622c9ee2d776556c287e3ed1743ffa84b
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-on: https://review.gluster.org/17372
CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
Smoke: Gluster Build System <jenkins@build.gluster.org>
Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
@ShyamsundarR
Copy link
Contributor

All artifacts for this feature are in, code + documentation. Closing the issue.

As a pointer for the future, when submitting the last artifact for the issue, use fixes #n in the commit message to auto close the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
CB: gfapi FA: Performance Improvements
Projects
None yet
Milestone
Release 3.11 (STM)
Development

No branches or pull requests
3 participants
@ShyamsundarR @soumyakoduri @gluster-ant