Performance #10
Labels
enhancement
New feature or request
help wanted
Extra attention is needed
performance
Throughput and/or latency issue
priority-medium
Medium priority issue
The performance as of 0.1.6 is worse than expected.
Between two virtual machines on the same virtualized network (capable of about 30 Gbps of throughput), Quincy only manages:
with an MTU of 1400 bytes.
Server -> Client
Initial profiling did not yield anything suspicious, other than the fact that
QuincyTunnel::process_inbound_traffic
takes more time (had more samples) thanQuincyTunnel::process_outbound_traffic
during theServer -> Client
data transfer, which is odd, as most of the data transfered should be going throughQuincyTunnel::process_inbound_traffic
.The CPU usage on the Server virtual machine is also only 60 % balanced across all cores, which could mean either either too much IO, or that the Client is the bottleneck.
The CPU usage on the Client is much higher, in the 90s.
Server flamechart:
![s2c-server](https://private-user-images.githubusercontent.com/18428519/252058222-16722a13-4fa0-4db1-9e84-0bbcb6b44508.svg?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjAwNzIxNDksIm5iZiI6MTcyMDA3MTg0OSwicGF0aCI6Ii8xODQyODUxOS8yNTIwNTgyMjItMTY3MjJhMTMtNGZhMC00ZGIxLTllODQtMGJiY2I2YjQ0NTA4LnN2Zz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MDQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzA0VDA1NDQwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBmMDRlMTY2ZTU3MTZkMGZiYzRmOTNlNzZhNjZhYWE5YjUyMTVhYTJkN2Q3YmNmYTFlYWVjNTE4OGNiMzdlNDQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.rGY5qgyeIZ4fY-v2RrpOlbMd3TKBvzIESKmcnvy2o54)
Client flamechart:
![s2c-client](https://private-user-images.githubusercontent.com/18428519/252058228-8b05e5e1-5ddb-4cfa-b896-86ecf3332e72.svg?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjAwNzIxNDksIm5iZiI6MTcyMDA3MTg0OSwicGF0aCI6Ii8xODQyODUxOS8yNTIwNTgyMjgtOGIwNWU1ZTEtNWRkYi00Y2ZhLWI4OTYtODZlY2YzMzMyZTcyLnN2Zz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MDQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzA0VDA1NDQwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVhZDFjMTk1Y2VmMTY3YjgyZjFiNTI2MGVhZjE4NjQwMTM2NDc4MWQzZTRiZWQyNGM5NjQ0MWZhYjM3OTc1ZjgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.MPc9nVRQI9okAo3KpVu2e60R4PM7Pd6UJ-pOOKUrBrk)
Client -> Server
Pretty much the same behaviour as above -
QuincyClient::process_inbound_traffic
takes more time thanQuincyClient::process_outbound_traffic
, which is, again, suspicious.The CPU usage on the Server side is above 90 %, on the Client side only ~ 70 %.
Server flamechart:
![c2s-server](https://private-user-images.githubusercontent.com/18428519/252058560-39997d26-ccf5-4542-8b87-7007edf6ad01.svg?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjAwNzIxNDksIm5iZiI6MTcyMDA3MTg0OSwicGF0aCI6Ii8xODQyODUxOS8yNTIwNTg1NjAtMzk5OTdkMjYtY2NmNS00NTQyLThiODctNzAwN2VkZjZhZDAxLnN2Zz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MDQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzA0VDA1NDQwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTViOTJhZGVhMGE3M2EwNDU1MzE5MGY2M2M2OTliMjBiMWMxYTkzOWRmZTUzMGY3MTQyYWI0NDFhOTQ5MWIyZTgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.xSZlEBE8oGvodYn8Yl10aWIHcNNQXFFUQ6z_0Tg9Jag)
Client flamechart:
![c2s-client](https://private-user-images.githubusercontent.com/18428519/252058594-3b3c681a-a55d-4d78-9f55-e32e249e6ecd.svg?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjAwNzIxNDksIm5iZiI6MTcyMDA3MTg0OSwicGF0aCI6Ii8xODQyODUxOS8yNTIwNTg1OTQtM2IzYzY4MWEtYTU1ZC00ZDc4LTlmNTUtZTMyZTI0OWU2ZWNkLnN2Zz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MDQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzA0VDA1NDQwOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTU5ZGEyYjYwOGMzZTYyZDE3N2Q1ZmJiNmI1NDg3YzI5NjNmZDUzMGZlMTc4MTI4MzVhNjlmMDFkYjk2NzQxZjUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.IOj80V6GxxS6L1XKrbrst1XvgwAEWOcLJJo-m9w6s3g)
Initial conclusions
It seems that the CPU usage on the receiving side is quite high, and that the receiving side spends more time in their respective
process_inbound_traffic
methods, which is highly suspicious (most of the data transfered should be handled by the respectiveprocess_outbound_traffic
methods, at least that is my initial assumption).Further investigation will be needed as to where Quincy client and server spend too much time.
TODO
process_inbound_traffic
andprocess_outbound_traffic
The text was updated successfully, but these errors were encountered: