Smallfile improvement in glusterfs

[mohit84]

As we know, we always tried to improve Smallfile performance in glusterfs.After doing a long duration testing we have found memory allocation is a big area that affects smallfile performance significantly in glusterfs. As we know before merged the patch #2675, gluster uses own thread based pool to allocate/deallocate memory blocks.In testing we observed it performs well as compare to glibc thread based pool but it does not perform well as compare to tcmalloc pool so we have decided to move to tcmalloc as a default option instead of using glibc pool.

I have executed a smallfile perf test case for 4.8M files 64K (more than 20 times on usual day operation) on latest devel branch.I have setup 12x3(nvme) volume after configure 4 event threads on 12 physical machines, there is no other configurable option i have enabled.

Hardware details:
12 physical machines (6 client, 6 servers)
Every machine is having 64 CPU 32G RAM, 10g NIC
Run smallfile tool to run operations and total number of files are 4.8M

Below is the data 

   

FOPS	Before #2675(mempool is enabled)	After #2675(disable mempool)	Enable tcmalloc(on top #2675)
create	14164.8	12064.8(-14.83)	16265.5(14.83)
ls-l	235044	211641(-9.96)	266195(13.25)
chmod	25391.9	23191(-8.67)	26516(4.43)
stat	29976.7	30762.5(2.62)	30334.9(1.19)
read	26140.7	23930.2(-8.46)	30394.4(16.27)
append	12853.7	12654(-1.55)	13384.9(4.13)
mkdir	2628.81	2488.85(-5.32)	2934.45(11.63)
rmdir	2436.01	2250.12(-7.63)	2654.48(8.97)
cleanup	6556.44	5439.39(-17.04)	8992.76(37.16)

During running a test case i have observed iobuf_pool lock has significant contention so i have tried to replace iobuf_pool with thread based pool(mempool). Iobuf_pool lock is process based lock so it is highly contended while we used thread based mempool we were getting performance improvement.After running multiple test cases for long duration with multiple options(iobuf_with malloc, iobuf_with_thread based pool) we have decided for the time being it is a good idea to move on standard malloc for iobuf also while a requested buffer size is less than 128k otherwise use current Iobuf pool allocation to allocate the memory.

Below is the data after using tcmalloc pool for buffer less than 128k

FOPS	Before #2675(mempool is enabled)	After #2675(disable mempool)	Enable tcmalloc(on top of #2675)	Enable tcmalloc with iobuf_128Kmalloc(on top of #2675)
create	14164.8	12064.8(-14.83)	16265.5(14.83))	16703.8(17.92)
ls-l	235044	211641(-9.96)	266195(13.25)	276855(17.79)
chmod	25391.9	23191(-8.67)	26516(4.43)	26812.1(5.59)
stat	29976.7	30762.5(2.62)	30334.9(1.19)	30391.2(1.38)
read	26140.7	23930.2(-8.46)	30394.4(16.27)	33075.2(26.53)
append	12853.7	12654(-1.55)	13384.9(4.13)	13474.5(4.83)
mkdir	2628.81	2488.85(-5.32)	2934.45(11.63)	3568.03(35.73)
rmdir	2436.01	2250.12(-7.63)	2654.48(8.97)	3396.19(39.42)
cleanup	6556.44	5439.39(-17.04)	8992.76(37.16)	9286.83(41.64)

Along with iobuf_patch there is one minor optimization we did in memory allocation, after call gf_calloc/gf_malloc/gf_free we do take a lock for memory accounting in xlator, we did optimize it further to delete some of the variable and convert some variable to atomic to avoid a lock

Below is the result

FOPS	Before #2675(mempool is enabled)	After #2675(disable mempool)	Enable tcmalloc(on top of #2675)	Enable tcmalloc with iobuf_128Kmalloc(on top of #2675)	Enable tcmalloc with 128KMalloc with lockfree mem_accounting(on top of #2675)
create	14164.8	12064.8(-14.83)	16265.5(14.83)	16703.8(17.92)	16884(19.20)
ls-l	235044	211641(-9.96)	266195(13.25)	276855(17.79)	285055(21.28)
chmod	25391.9	23191(-8.67)	26516(4.43)	26812.1(5.59)	26900.6(5.94)
stat	29976.7	30762.5(2.62)	30334.9(1.19)	30391.2(1.38)	30394.1(1.39)
read	26140.7	23930.2(-8.46)	30394.4(16.27)	33075.2(26.53)	33275.3(27.29)
append	12853.7	12654(-1.55)	13384.9(4.13)	13474.5(4.83)	13627.5(6.02)
mkdir	2628.81	2488.85(-5.32)	2934.45(11.63)	3568.03(35.73)	3685.53(40.20)
rmdir	2436.01	2250.12(-7.63)	2654.48(8.97)	3396.19(39.42)	3472.65(42.55)
cleanup	6556.44	5439.39(-17.04)	8992.76(37.16)	9286.83(41.64)	9834.65(50.00)

During tcmalloc testing i have found one minor challenge with tcmalloc is it consumes more than 20% memory as compare to glibc (ptmalloc) while IO is running for long duration because it happened due to the the design of tcmalloc. After read the tcmalloc documentation i have found it provides some api’s(ReleaseMemroy) to return the excess memory to the system. In my case the client(fuse) consume at max 850M(tcmalloc) vs 700M(glibc malloc) and brick consumes 150M(tcmalloc) vs 125M(glibc malloc) so i it not much high.Later we can call tcmalloc functions to cleanup the excess memory.

A big thanks to Xavi/Yaniv for giving a guidance to make it successful.

[mohit84]

All the required patches are merged so i am closing the issue.