Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Upload speed much slower than download speed in glftpd #35
Tested with v2.09 , v2.10
glftpd has some limit on upload speed (when glftpd is receiving), at around 4-6Gbit per thread, this depends on cpu clock speed where this limit sits.
Download speed does not seem to be limited.
These tests are without ssl enabled.
Upload 1,024.0 Mbytes/1.75(s)/613,917.57Kbps
Download 1,024.0 Mbytes/0.69(s)/1,551,650.03Kbps
Running multiple upload transfers at once, gives num_of_xfers x limit (around those 5Gbit)
Any reason upload speed is limited this way ?
Could be many reasons. Possibly because of the internall 8kb read buffer which isnt configurable. I dont have such hardware to profile on so cant say.
Feel free to play around with turning off real time crc calculation or using ul_buffered to write larged chunks to disk. Maybe that will have some effect.
Also see if other ftp servers have any different results...
Thank you for the reply. I have tested the options suggested.
All tests are done with either ramdisks or nvme drives, to remove storage bottleneck.
To disable calc_crc gives another around 3Gbit, moving from 4-6Gbit to 7-9Gbit.
Without calc_crc, I get these results below.
The ul_buffered_force option does not look to have any impact on speeds on upload.
Without ul_buffered_force option
Tested with other ftp server as suggested (ProFTPD)
It looks like its a specific glFTPd upload issue.
Thanks for the reply.
Tested both with and without post_check (the zipscript-c), does not change anything.
as described by Vunacle. I can confirm all tests are exactly as reproducible.
I did a few tests for comparison.
The following test are performed using a ftp client on the same box as glftpd.
Test file is a 4GB sparse file created with fallocate.
thanks for the feedback with the tests.
zero: dd if=/dev/zero of=speedfile_zero bs=1G count=4
could you do a test with the two created files?
maybe it is really the internal buffer of 8kb that is too low
Thank you for the reply.
Did some tests with changing the buffer size on ProFTPD from auto and down to 8kb
Used this option in ProFTPD -> SocketOptions sndbuf 8192 rcvbuf 8192
Varied the buffer size from 8kb to 512kb to see the changes.
This is the speeds I got when xfering from glFTPd to ProFTPD.
Upload ProFTPD (8kb buffer) 1,024.0 Mbytes/1.10(s)/974,357.37Kbps
Looks to be capped around 950M-1GB/s with 8kb option in place.
Did some testing with a version with larger buffers on upload in glFTPd (fresh install)
Upload buffer scaling on upload
UPLOAD glFTPd (8kb buffer) 1,024.0 Mbytes/1.00(s)/1,072,669.15Kbps
UPLOAD glFTPd (256kb buffer) 1,024.0 Mbytes/1.14(s)/939,406.67Kbps
With these adjustments, performance on upload now looks to be the same in glFTPd as with ProFTPD. with calc_crc off.
I suspect the calc_crc lower performance is a cpu limitation, as 1 core on the cpu sits at 100% when calc_crc is on, during a single thread transfer. I guess its internal cpu/mem latency and IPC+clock speed that decide how fast calc_crc on can run.
Is there any way to speed up calc_crc on maybe ?, as calc_crc on does speedup zipscript a lot.
This looks to be what was the issue, buffers was just too small for high speed networking links (10G+)
This is great work, really some massive improvements.
Posting some tests done with the improved crc alg.
UPLOAD glFTPd (256kb buffer) 1,024.0 Mbytes/0.84(s)/1,281,314.83Kbps
So now calc_crc on, is able to hit 10Gbit.
Here is some tests in case someone wanted to know the upload numbers with SSL Datachannel enabled.
Upload glFTPd (256kb buffer) 1,024.0 Mbytes/1.72(s)/626,088.53Kbps
Upload glFTPd (8kb buffer) 1,024.0 Mbytes/1.36(s)/790,097.00Kbps
Its all cpu limited it looks. even the cpu should be capable of doing around 1GB/s of AES in hw. per. core.
The fix for the slower upload issue is what was suggested earlier in this thread, that we need a glFTPd version with a larger internal buffer size, which is 8kb as standard, it needs to be able to scale up to use high bandwidth network speeds, beyond 5-6Gbit.
I expect it will be included in the next version, as it now has been proven to improve performance. But someone from the actual glFTPd team will have to confirm this.
Another improvements that was shown in the tests is a faster crc algorithm, that gave a nice boost in calc_crc speed too. Which again I expect will be kept and be in the next version as well.