Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Reduce number of copies in ssl transport layer #14058
Please answer these questions before submitting your issue.
Should this be an issue in the gRPC issue tracker?
Create new issues for bugs and feature requests. An issue needs to be actionable. General gRPC discussions and usage questions belong to:
Please don't double post your questions in more locations; we are monitoring both channels, and the time spent de-duplicating questions is better spent answering more user questions.
What version of gRPC and what language are you using?
1.8.4, grpc-core (proprietary binding)
What operating system (Linux, Windows, …) and version?
Linux, Ubuntu 16.04
What runtime / compiler are you using (e.g. python version or version of gcc)
What did you do?
Run a load test with the following setup (and other way around):
What did you expect to see?
Both server and client spend most of their time in tcp stack, transferring data to/from tcp stack and doing actual encryption. Effectively setup should be able to handle ~600-800 MBps per cpu core. (Non-tls grpc handles ~900 MBps per cpu core on the same hardware under the same load conditions).
What did you see instead?
Server spends 80% of cpu time inside BIO_read (based on pprof) and throughput is bounded by 50 MBps (~20 times performance degradation compared to disabled encryption).
Anything else we should know about your project / environment?
Apparently ssl_transport_security uses BIO_s_mem which has potentially O(n^2) runtime complexity for reading all data from buffer by using BIO_read operation.
It can be mitigated by using BIO_pair instead as it uses ring buffer and therefore upper bounds number of performed copies. It allows to achieve throughput of 300 MBps for the described above test.
More gains can be achieved by implementing zero copy protector interface to pass slices to boringssl directly using custom BIO implementation (similar to chrome's approach).