Realtime messaging not working properly

[dakikifr]

Hi,

I'm writing a multiplayer game which uses gpg RealtimeMultiplayer API.
I'm facing multiple issues with it:

1/ Reliable messages are not arriving in order

2/ I often get an error during the send, but the message still arrives

1/ Here is the log of what I'm sending (burst send)
cocos2d: sending chunk of size 4 to p_CPDEjqzHuPOlYRAB
cocos2d: sending chunk of size 1400 to p_CPDEjqzHuPOlYRAB
cocos2d: sending chunk of size 1400 to p_CPDEjqzHuPOlYRAB
cocos2d: sending chunk of size 1400 to p_CPDEjqzHuPOlYRAB
cocos2d: sending chunk of size 1400 to p_CPDEjqzHuPOlYRAB
cocos2d: sending chunk of size 1400 to p_CPDEjqzHuPOlYRAB
cocos2d: sending chunk of size 826 to p_CPDEjqzHuPOlYRAB
cocos2d: sending chunk of size 4 to p_CPDEjqzHuPOlYRAB
cocos2d: sending chunk of size 5 to p_CPDEjqzHuPOlYRAB
The message is printed right before calling RealTimeMultiplayer().SendReliableMessage().

And here is the log on the receiver side:
cocos2d: OnDataReceived from p_CNnB1Na874r21gEQAQ (Reliable): 4 bytes
cocos2d: OnDataReceived from p_CNnB1Na874r21gEQAQ (Reliable): 1400 bytes
cocos2d: OnDataReceived from p_CNnB1Na874r21gEQAQ (Reliable): 1400 bytes
cocos2d: OnDataReceived from p_CNnB1Na874r21gEQAQ (Reliable): 1400 bytes
cocos2d: OnDataReceived from p_CNnB1Na874r21gEQAQ (Reliable): 1400 bytes
cocos2d: OnDataReceived from p_CNnB1Na874r21gEQAQ (Reliable): 1400 bytes
cocos2d: OnDataReceived from p_CNnB1Na874r21gEQAQ (Reliable): 4 bytes
cocos2d: OnDataReceived from p_CNnB1Na874r21gEQAQ (Reliable): 826 bytes
cocos2d: OnDataReceived from p_CNnB1Na874r21gEQAQ (Reliable): 5 bytes
The message is printed at the begining of the OnDataReceived() method.

How is that possible, the documentation specifies that the Reliable data are sent and received in order (should be if it's using TCP). Is there a bug in the gpg sdk which calls the callback not in order (another thread might be receiving the data, and not calling the callback correctly?).
Another question, who is responsible for flow control? Is it the SendReliableMessage() API that bufferize my data if it cannot immediatly send it, or should I do it myself (if so, how do I know if the receiver if "full"?)

2/ I check the RealTimeMultiplayer().SendReliableMessage() status from the lambda passed to the function, and in my previous example, I got in the log for the very first chunk I sent:
cocos2d: Message sending status: FAIL (-3)
-3 being ERROR_NOT_AUTHORIZED
But the chunk arrived to the receiver.

Why did I get this error (only for the first message, but I sometimes have this error multiple time in a row)? Should I ignore send errors (not very safe, I want my game to stop if a player is not receiving the data)?

I did all my tests using an iPhone6 and an iPad4, using the same build.
I didn't even try using an iOS device with an Android one... yet.
Every run, I have some messages out of order and/or send errors. In fact on my 50+ runs, I never got the full message in order on the receiver.

Thanks!

[dakikifr]

I tried using my android device (receiver), along with an iphone as sender. I have the same issue (not on the first message that I'm sending, but on another one right after).

I managed to capture the traffic from my iPhone, but I couldn't identify the TCP traffic from my application. Is gpg using SSL or something for p2p communication?
How can I help debugging this very important issue?
Thanks

[dakikifr]

I managed to produce this critical issue using the ButtonClicker source code.
Here is what I've done:

Add this method to ButtonClickerEngine.mm:

void ButtonClickerEngine::SendPacket(int val) {
    std::vector<uint8_t> v;
    v.resize(1400);
    v[0] = static_cast<uint8_t>(val);
        const std::vector<gpg::MultiplayerParticipant> participants =
        room_.Participants();
        for (gpg::MultiplayerParticipant participant : participants) {
            service_->RealTimeMultiplayer().SendReliableMessage(
                                                                room_, participant, v, [val](gpg::MultiplayerStatus const & status) {
                                                                    if(gpg::IsSuccess(status) == false)
                                                                        LOGI("Failed to send message idx %d (%d)", val, status);
                                                                });
        }
}

Then edit GameModel.mm and change the startGame method so it looks like that

- (void)startGame {
  if (_gameState == BCGameStateWaitingToStart) {
    _gameState = BCGameStatePlaying;
    _startTime = CACurrentMediaTime();
      for(int i=0;i<10;i++)
          ButtonClickerEngine::GetInstance().SendPacket(i);
  }
}

Then run the game on 2 devices (I've used an iPhone and an iPad), and you'll often have out of order packets in the log like this log (as well as Errors during send):

2015-05-15 21:58:00.996 ButtonClicker[3301:1080849] Failed to send message idx 0 (-3)
2015-05-15 21:58:00.997 ButtonClicker[3301:1080849] Failed to send message idx 0 (-2)
2015-05-15 21:58:00.997 ButtonClicker[3301:1080849] Failed to send message idx 1 (-2)
2015-05-15 21:58:00.998 ButtonClicker[3301:1080849] Failed to send message idx 2 (-2)
2015-05-15 21:58:00.999 ButtonClicker[3301:1080849] Failed to send message idx 3 (-2)
2015-05-15 21:58:00.999 ButtonClicker[3301:1080849] Failed to send message idx 4 (-2)
2015-05-15 21:58:00.999 ButtonClicker[3301:1080849] Failed to send message idx 5 (-2)
2015-05-15 21:58:01.000 ButtonClicker[3301:1080849] Failed to send message idx 6 (-2)
2015-05-15 21:58:01.003 ButtonClicker[3301:1080849] Failed to send message idx 7 (-2)
2015-05-15 21:58:01.003 ButtonClicker[3301:1080849] Failed to send message idx 8 (-2)
2015-05-15 21:58:01.003 ButtonClicker[3301:1080849] Failed to send message idx 9 (-2)
2015-05-15 21:58:01.170 ButtonClicker[3301:1080877] VERBOSE: RoomServiceStateMachine: Reliable message received : 1va148kzpa99y2cc6jtavw9n1j@public.talk.google.com/games_ios61CD4D53
2015-05-15 21:58:01.171 ButtonClicker[3301:1080849] Got message of size 1400 (0)
2015-05-15 21:58:01.173 ButtonClicker[3301:1080877] VERBOSE: RoomServiceStateMachine: Reliable message received : 1va148kzpa99y2cc6jtavw9n1j@public.talk.google.com/games_ios61CD4D53
2015-05-15 21:58:01.174 ButtonClicker[3301:1080849] Got message of size 1400 (1)
2015-05-15 21:58:01.195 ButtonClicker[3301:1080877] VERBOSE: RoomServiceStateMachine: Reliable message received : 1va148kzpa99y2cc6jtavw9n1j@public.talk.google.com/games_ios61CD4D53
2015-05-15 21:58:01.196 ButtonClicker[3301:1080849] Got message of size 1400 (3)
2015-05-15 21:58:01.198 ButtonClicker[3301:1080877] VERBOSE: RoomServiceStateMachine: Reliable message received : 1va148kzpa99y2cc6jtavw9n1j@public.talk.google.com/games_ios61CD4D53
2015-05-15 21:58:01.198 ButtonClicker[3301:1080877] VERBOSE: RoomServiceStateMachine: Reliable message received : 1va148kzpa99y2cc6jtavw9n1j@public.talk.google.com/games_ios61CD4D53
2015-05-15 21:58:01.199 ButtonClicker[3301:1080849] Got message of size 1400 (2)
2015-05-15 21:58:01.199 ButtonClicker[3301:1080849] Got message of size 1400 (4)
2015-05-15 21:58:01.224 ButtonClicker[3301:1080877] VERBOSE: RoomServiceStateMachine: Reliable message received : 1va148kzpa99y2cc6jtavw9n1j@public.talk.google.com/games_ios61CD4D53
2015-05-15 21:58:01.224 ButtonClicker[3301:1080849] Got message of size 1400 (5)
2015-05-15 21:58:01.232 ButtonClicker[3301:1080877] VERBOSE: RoomServiceStateMachine: Reliable message received : 1va148kzpa99y2cc6jtavw9n1j@public.talk.google.com/games_ios61CD4D53
2015-05-15 21:58:01.232 ButtonClicker[3301:1080849] Got message of size 1400 (6)
2015-05-15 21:58:01.235 ButtonClicker[3301:1080877] VERBOSE: RoomServiceStateMachine: Reliable message received : 1va148kzpa99y2cc6jtavw9n1j@public.talk.google.com/games_ios61CD4D53
2015-05-15 21:58:01.236 ButtonClicker[3301:1080849] Got message of size 1400 (7)
2015-05-15 21:58:01.677 ButtonClicker[3301:1080877] VERBOSE: RoomServiceStateMachine: Reliable message received : 1va148kzpa99y2cc6jtavw9n1j@public.talk.google.com/games_ios61CD4D53
2015-05-15 21:58:01.677 ButtonClicker[3301:1080877] VERBOSE: RoomServiceStateMachine: Reliable message received : 1va148kzpa99y2cc6jtavw9n1j@public.talk.google.com/games_ios61CD4D53
2015-05-15 21:58:01.678 ButtonClicker[3301:1080849] Got message of size 1400 (8)
2015-05-15 21:58:01.678 ButtonClicker[3301:1080849] Got message of size 1400 (9)
201

Please advice since this is a critical issue preventing any real usage of your Realtime APIs.
Maybe you could expose the sockets for advanced users (both TCP and UDP)?
Thanks!

[dakikifr]

Hi,

have you managed to reproduce this bug using the modified ButtonClicker sample?
Thanks!

[dakikifr]

I tried the sendReliableMessageBlocking method of the SDK, using a specific thread to send the data, and sync my main thread with the sending thread (so it continues only when the sendReliableMessageBlocking method returns), and it's working a little better.
Let me explain:
The burst send of 1400 chunks are now almost always arriving in order (and I still sometimes have an error returned by the send method, although the message is arriving fine and in order).
But when the receiver answers by send 2 small messages (the 1st is 4 bytes, the second is 5 bytes), it almost always arrives out-of-order (the same code is used to send the chunks, using sendReliableMessageBlocking and a thread).

Please advice?
Have you tried to reproduce this using the sample?
Did you succeed?
Realtime multiplayer is a core part of my game and I cannot ignore it.
I'll have no choice but to use another Gaming framework if this one is not stable enough.
Thanks for your help

[dakikifr]

Still no answer (at least an acknowledgment of the issue being reproduced) after more than a month.
Is the c++ gpg framework abandoned?