Collect stats about message sizes in JITServer

[mpirvu]

The goal is to determine which messages are larger and whether such messages can be reduced in size. For instance, the message that sends the compilation request is probably on the large side because it sends the entire ROMClass to the JITServer. If we find it worthwhile, we may be able to keep some cache at the client that memorizes which ROMClasses have already been sent to the server so that we don't have to send them again.

[mpirvu]

I am thinking that the best place for collecting message sizes is:

void
CommunicationStream::readMessage(Message &msg)
   {
   msg.clearForRead();

   // read message size
   uint32_t serializedSize;
   readBlocking(serializedSize);
   msg.setSerializedSize(serializedSize);

   // read the rest of the message
   uint32_t messageSize = serializedSize - sizeof(uint32_t);
   readBlocking(msg.getBufferStartForRead() + sizeof(uint32_t), messageSize);

   // rebuild the message
   msg.deserialize();
   }

Here we have access to the total message size and the type of the message can be obtained with msg.type()

[mpirvu]

Attn: @EmanElsaban

[mpirvu]

Stats could be collected with utilities from omr/compiler/infra/Statistics.hpp
class TR_Stats
or class TR_StatsHisto when the spread is large

[EmanElsaban]

This Excel sheet contains the stats collected from Acmeair benchmark:
AcmeAir Message Statistics.xlsx

[mpirvu]

Thanks.
These are the messages sent by the server and received by the client. It appears that the message with the compiled code can be pretty big, 1K-65K, but the rest are typically less than 100 bytes.

@EmanElsaban could you please generate the same stats from the server point of view (messages received by the server)? In this case you may need to add the print routine to another place.

[EmanElsaban]

I included the message stats of Acmeair on the client and server side in this excel sheet, Thank you.
AcmeAir MsgStats on client and server side.xlsx

[EmanElsaban]

I ran it a second trial on Acmeair to print out the total number of messages on server side, since I didn't print it the first trial:
AcmeAir MsgStats on client and server side.xlsx

[mpirvu]

The compilation request is the largest contributor: it has the largest messages (115KB), the latest average message size (17.5 KB) and the largest number of bytes being received (95 MB). This represents 55% of the total bytes transferred from client to server.
Given this, it may be worthwhile to look into reducing the size of this message as mentioned in the first comment of this issue

[mpirvu]

Resolved by #9870