Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thread, Memory & CPU leak using mobicents RTP #21

Open
mcacker opened this issue Jun 25, 2015 · 18 comments
Open

Thread, Memory & CPU leak using mobicents RTP #21

mcacker opened this issue Jun 25, 2015 · 18 comments
Assignees
Labels

Comments

@mcacker
Copy link

mcacker commented Jun 25, 2015

Hi,

i'm using mobicents RTP to implements a media server and SBC monitor, by setting up a call and sending and receiving RTP. Unfortunately, whenever the call is stopped and restarted there is a Thread leak, memory leak and slowly increasing CPU usage, to the point that eventually the app crashes.

I've based it on the RTPChannelTest class.

the call setup is as follows:

    sendSource = new Sine(scheduler);
    sendSource.setFrequency(100);        

    // set peer on sending channel. receiving channel has already been bound
    getSendChannel().setPeer(new InetSocketAddress(mediaServerAddress, sendRemotePort));

    getSendChannel().setFormatMap(AVProfile.audio);
    getRecvChannel().setFormatMap(AVProfile.audio);

    // setup an audio component to receive the input
    sendComponent=new AudioComponent(1);
    sendComponent.addInput(sendSource.getAudioInput());
    sendComponent.updateMode(true,true);

    // setup an audio mixer to receive the input; mix the send component and send channel
    sendAudioMixer=new AudioMixer(scheduler);
    sendAudioMixer.addComponent(sendComponent);
    sendAudioMixer.addComponent(getSendChannel().getAudioComponent());

and the tear down:

        sendSource.deactivate();
        sendChannel.close();
        recvChannel.close();
        sendAudioMixer.stop();
        udpManager.stop();
        scheduler.stop();

the problem appears to be in the Scheduler class, specifically the CriticalWorkerThread and the Worker thread where they remain in the WAITING state on the take statement (line 413) in

                try
                {
                    current=waitingTasks.take();
                }
                catch(Exception ex)
                {

                }                   

the Scheduler.stop() method only sets the 'active' boolean for the threads which can never be tested while the threads are in the WAITING state.

What is the correct way to stop the scheduler to prevent this thread leak?

I've thought of adding something to interrupt the threads, but there is an empty catch block which would just swallow the resultant interrupt.

Any guidance would be much appreciated as this is it makes my head hurt trying to get to grips with this code and how to fix my app or modify the code.

thanks, Mitchell

@mcacker
Copy link
Author

mcacker commented Jun 30, 2015

I've fixed the resource leaks. Let me know if you're interested in them. I can push my branch if you like.

@hrosa
Copy link
Contributor

hrosa commented Jul 1, 2015

Hi Mitchell,

Thanks for reporting and contributing with a fix. Much appreciated.
Please push your branch so I can review.

Best Regards,
Henrique

@hrosa
Copy link
Contributor

hrosa commented Jul 1, 2015

Mitchell,

Can you please sign the Contributor License Agreement (http://www.telestax.com/open-source/#Contribute) so I can accept your contribution?

Also, would you like to be acknowledged for your contribution? (http://www.telestax.com/open-source/acknowledgments/)

Regards

@mcacker
Copy link
Author

mcacker commented Jul 1, 2015

I signed the contributor agreement, I guess I now wait for an invite or something?

Sure, if you like my contribution, you can acknowledge it. You could also acknowledge a fix to jain-sip, issue 21, see https://java.net/projects/jsip/pages/Acknowledgements

@hrosa
Copy link
Contributor

hrosa commented Jul 3, 2015

Thanks for signing the agreement. We're now trying to reach you over email.

@hrosa
Copy link
Contributor

hrosa commented Jul 3, 2015

Mitchell,

Can you please do a pull request so I can review your work?

Thank you

@hrosa hrosa self-assigned this Jul 8, 2015
@hrosa hrosa added the bug label Jul 8, 2015
@hrosa
Copy link
Contributor

hrosa commented Jul 20, 2015

Hi Mitchell,

Just checking on your progress, any news?

Best Regards

@mcacker
Copy link
Author

mcacker commented Jul 20, 2015

hey Henrique,

sorry, been swamped here. I was planning on waiting until you had
completed issue #23, as you suggested, before doing the merge. I just
checked it's status, and see that it is still open. Is that still the
recommended course of action?

Mitchell

On Mon, Jul 20, 2015 at 8:00 AM, Henrique Rosa notifications@github.com
wrote:

Hi Mitchell,

Just checking on your progress, any news?

Best Regards


Reply to this email directly or view it on GitHub
#21 (comment)
.

@hrosa
Copy link
Contributor

hrosa commented Jul 21, 2015

Sure, take your time.
You can pull the master and apply the fix, the issue 23 will NOT impact the Scheduler, UdpManager nor the PortManager.

Thanks

@hrosa hrosa added this to the 3.0.3 milestone Aug 20, 2015
hrosa added a commit that referenced this issue Aug 25, 2015
… when it is started. Upon stopping the manager, these resources will be cleaned.

Code cleanup of PortManager and UdpManager.
hrosa added a commit that referenced this issue Aug 25, 2015
Properly naming the threads spawned by the Scheduler.
hrosa added a commit that referenced this issue Sep 3, 2015
… when it is started. Upon stopping the manager, these resources will be cleaned.

Code cleanup of PortManager and UdpManager.
@hrosa hrosa removed this from the 3.0.3 milestone Oct 5, 2015
@thanhthuy40
Copy link

hi mcacker,
how to fix leak resourse ? can you show me?

@mcacker
Copy link
Author

mcacker commented Jun 2, 2016

I would use the latest branch. The offending class has been completely refactored since I patched it. I can't verify that it no longer has a resource leak as i'm still using my older patched version. If you're stuck on that version for some reason, you can find my patch at #24

@thanhthuy40
Copy link

Hi Mcracker ,
i have been write a test with MGCP driver ,
When i'm using 100 TPS connection to MMS server. I see in the MSS server , RAM increment very big but not decrements RAM when connection release (DLCX). This is my class test. Can you help me about how to fix it ? or idea because to long time i can't fixed it!
public class Test implements MgcpListener {

private static final RequestedAction[] actions = new RequestedAction[] { RequestedAction.NotifyImmediately };

private static final RequestedEvent[] requestedEventsWithoutDtmf = {
        new RequestedEvent(new EventName(AUPackage.AU, MgcpEvent.oc), actions),
        new RequestedEvent(new EventName(AUPackage.AU, MgcpEvent.of), actions) };

private static final RequestedEvent[] requestedEvents = {
        new RequestedEvent(new EventName(AUPackage.AU, MgcpEvent.oc),
                actions),
        new RequestedEvent(new EventName(AUPackage.AU, MgcpEvent.of),
                actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("0")),
                actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("1")), actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("2")), actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("3")), actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("4")), actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("5")), actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("6")), actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("7")), actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("8")), actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("9")), actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("A")), actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("B")), actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("C")), actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("D")), actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("*")), actions),
        new RequestedEvent(new EventName(PackageName.Dtmf, MgcpEvent.factory("#")), actions) };

public static final String sdp = "v=0" + "\r\no=- 1436175570632 1 IN IP4 127.0.0.1" + "\r\ns=Mobicents Media Server"
        + "\r\nc=IN IP4 192.168.146.103" + "\r\nt=0 0" + "\r\nm=audio 8984 RTP/AVP 97 8 0 101 116"
        + "\r\na=rtpmap:97 l16/8000" + "\r\na=rtpmap:8 pcma/8000" + "\r\na=rtpmap:0 pcmu/8000"
        + "\r\na=rtpmap:101 telephone-event/8000" + "\r\na=fmtp:101 0-15" + "\r\na=rtpmap:116 telephone-event/8000"
        + "\r\na=fmtp:116 0-15\n";

MarvelMgcpProvider mgcpProvider = null;

EndpointIdentifier mgcpEndpointId = null;
CallIdentifier callID = null;
Logger logger = Logger.getLogger(Test.class);
MgcpSession mgcpSession = null;

public Test(MarvelMgcpProvider mgcpProvider) {
    this.logger = Logger.getLogger(Test.class);
    this.mgcpProvider = mgcpProvider;
    mgcpSession = new MgcpSession(mgcpProvider, this);
}

@Override
public void onTransactionTimeout(TransactionTimeout timeoutEvent) {

    logger.info("Timeout Event:\n" + timeoutEvent.getJainMgcpCommandEvent());
}

@Override
public void onDeleteConnection(DeleteConnection request) {
    logger.info("onDeleteConnection\n" + request);
    // send response 200 OK
    DeleteConnectionResponse response = new DeleteConnectionResponse(request.getSource(),
            ReturnCode.Transaction_Executed_Normally);
    response.setTransactionHandle(request.getTransactionHandle());
    logger.info("Sending DeleteConnectionResponse [" + response + "] to [" + request + "]["
            + request.getTransactionHandle() + "] [" + response.getTransactionHandle() + "]");

    mgcpProvider.sendMgcpEvents(new JainMgcpEvent[] { response });
}

@Override
public void onModifyConnection(ModifyConnection request) {
    logger.info("onModifyConnection\n" + request);

}

@Override
public void onNotificationRequest(NotificationRequest request) {
    logger.info("onNotificationRequest\n" + request);

}

@Override
public void onNotify(Notify request) {
    logger.info("onNotify\n" + request);
    NotifyResponse response = new NotifyResponse(request.getSource(), ReturnCode.Transaction_Executed_Normally);
    response.setTransactionHandle(request.getTransactionHandle());

    mgcpProvider.sendMgcpEvents(new JainMgcpEvent[] { response });
    logger.info("Send NotifyResponse:\n" + response);

    this.deleteConnection(mgcpEndpointId);
}

@Override
public void onCreateConnectionResponse(CreateConnectionResponse response) {
    logger.info("onCreateConnectionResponse\n" + response);
    mgcpEndpointId = response.getSpecificEndpointIdentifier();
    // this.deleteConnection(mgcpEndpointId);

    this.sendRQNT("file:///D:\\audio\\ann.wav", true, response.getSpecificEndpointIdentifier());

    // if (response.getSecondConnectionIdentifier() == null) {
    // this.sendCrcx(response.getSpecificEndpointIdentifier(), callID);
    // } else {
    //// this.deleteConnection(response.getSecondEndpointIdentifier());
    //// this.deleteConnection(mgcpEndpointId);
    // this.sendRQNT("file:///D:\\audio\\ann.wav", true,
    // response.getSecondEndpointIdentifier());
    // }
}

@Override
public void onDeleteConnectionResponse(DeleteConnectionResponse response) {
    logger.info("onDeleteConnectionResponse\n" + response);

}

@Override
public void onModifyConnectionResponse(ModifyConnectionResponse response) {
    logger.info("onModifyConnectionResponse\n" + response);

}

@Override
public void onNotificationRequestResponse(NotificationRequestResponse response) {
    logger.info("onNotificationRequestResponse\n" + response);

}

@Override
public void onNotifyResponse(NotifyResponse response) {
    logger.info("onNotifyResponse\n" + response);

}

public void sendCrcx() {
    try {
        EndpointIdentifier endpointID = new EndpointIdentifier("mobicents/ivr/$", "192.168.146.103:2427");
        callID = new CallIdentifier(Test.getUniqueCallId());
        CreateConnection createConnection = new CreateConnection(this, callID, endpointID, ConnectionMode.SendRecv);
        int txID = mgcpProvider.getUniqueTransactionHandler();

        createConnection.setTransactionHandle(txID);
        createConnection.setRemoteConnectionDescriptor(new ConnectionDescriptor(sdp));

        logger.info("Send MGCP CRCX\n" + createConnection);
        mgcpSession.sendJainMgcpCommandEvent(new CreateConnection[] { createConnection });
    } catch (Exception e) {
        logger.error("Error send CRCX", e);
    }
}

public void sendCrcx(EndpointIdentifier endpointID, CallIdentifier callID) {
    try {
        EndpointIdentifier secondEndpoint = new EndpointIdentifier("mobicents/ivr/$", "192.168.146.103:2427");
        CreateConnection createConnection = new CreateConnection(this, callID, endpointID, ConnectionMode.SendRecv);
        createConnection.setSecondEndpointIdentifier(secondEndpoint);
        int txID = mgcpProvider.getUniqueTransactionHandler();

        createConnection.setTransactionHandle(txID);
        // createConnection.setRemoteConnectionDescriptor(new
        // ConnectionDescriptor(sdp));

        logger.info("Send MGCP CRCX\n" + createConnection);
        mgcpSession.sendJainMgcpCommandEvent(new CreateConnection[] { createConnection });
    } catch (Exception e) {
        logger.error("Error send CRCX", e);
    }
}

public void deleteConnection(EndpointIdentifier mgcpEndpointId) {
    DeleteConnection deleteConnection = new DeleteConnection(this, mgcpEndpointId);
    // deleteConnection.setCallIdentifier(this.getCallIdentifier());

    deleteConnection.setTransactionHandle(mgcpProvider.getUniqueTransactionHandler());
    mgcpProvider.sendMgcpEvents(new JainMgcpEvent[] { deleteConnection });
    logger.info("Sent DLCX request endpoidID: " + mgcpEndpointId);
    logger.info("Delete connections: \n" + deleteConnection);
}

public void sendRQNT(String mediaPath, boolean receiverDtmf, EndpointIdentifier mgcpEndpointId) {
    try {
        NotificationRequest notificationRequest = new NotificationRequest(this, mgcpEndpointId,
                mgcpProvider.getUniqueRequestIdentifier());

        EventName[] signalRequests = {
                new EventName(AUPackage.AU, MgcpEvent.factory("pa").withParm("an=" + mediaPath)) };
        notificationRequest.setSignalRequests(signalRequests);

        if (receiverDtmf) {
            notificationRequest.setRequestedEvents(requestedEvents);
        } else {
            notificationRequest.setRequestedEvents(requestedEventsWithoutDtmf);
        }
        notificationRequest.setTransactionHandle(mgcpProvider.getUniqueTransactionHandler());

        NotifiedEntity notifiedEntity = new NotifiedEntity("192.168.146.103", "192.168.146.103", 2727);
        notificationRequest.setNotifiedEntity(notifiedEntity);

        mgcpSession.sendJainMgcpCommandEvent(new JainMgcpCommandEvent[] { notificationRequest });
        logger.info("NotificationRequest sent:\n" + notificationRequest);
    } catch (Exception ex) {
        logger.error("CALL05: Error while send RQNT", ex);
    }
}

public static String getUniqueCallId() {
    return Long.toHexString(new Random().nextLong() ^ System.currentTimeMillis());
}

public static void main(String[] args) throws InterruptedException {
    MgcpStack.loadLog4j();
    MgcpStack stack = new MgcpStack();

    while (true) {
        int count = 0;
        for (int i = 0; i < 20; i++) {
            Test test = new Test(stack.getMgcpProvider());
            test.sendCrcx();
            count++;
        }

        if (count > 1000) {
            Thread.sleep(10000);
            count = 0;
        } else {
            Thread.sleep(1000);
        }

        Logger logger = Logger.getLogger("Main");
        logger.info("connections: " + stack.handleManager.connections);
        logger.info("endpoints: " + stack.handleManager.endpoints);
        logger.info("transactions: " + stack.handleManager.transactions);
    }

    // Thread.sleep(Integer.MAX_VALUE);

    // ConnectionIdentifier conn1 = new ConnectionIdentifier("1");
    // ConnectionIdentifier conn2 = new ConnectionIdentifier("1");
    // System.out.println(conn1.equals(conn2));
}

}

@mcacker
Copy link
Author

mcacker commented Jun 3, 2016

Sorry, this is beyond my knowledge of the mediaserver. I recommend you open a new Issue, perhaps one of the contributers can help.

@hrosa
Copy link
Contributor

hrosa commented Jun 3, 2016

Hi @thanhthuy40 please open a thread in the public forum: https://groups.google.com/forum/#!forum/restcomm

Thank you

@thanhthuy40
Copy link

thank you hrosa

@thanhthuy40
Copy link

Hi Mcracker ,
this is my test result with 100 TPS , but when application is stop , the RAM still increment , all end point is stop , do you have any ided about this task !

21818 thanhhn5 20 0 4974880 197056 17952 S 6.5 3.4 0:15.94 java
21818 thanhhn5 20 0 4974880 197056 17952 S 10.6 3.4 0:16.26 java
21818 thanhhn5 20 0 4974880 197056 17952 S 10.0 3.4 0:16.56 java
21818 thanhhn5 20 0 4976928 201432 17952 S 49.2 3.5 0:18.04 java
21818 thanhhn5 20 0 4976928 310056 17952 S 161.1 5.3 0:22.89 java
21818 thanhhn5 20 0 4976928 344128 17952 S 96.4 5.9 0:25.79 java
21818 thanhhn5 20 0 4976928 405472 17952 S 106.2 7.0 0:28.99 java
21818 thanhhn5 20 0 4976928 405612 17952 S 58.5 7.0 0:30.75 java
21818 thanhhn5 20 0 4976928 411876 17952 S 45.2 7.1 0:32.11 java
21818 thanhhn5 20 0 4976928 460436 17952 S 40.9 7.9 0:33.34 java
21818 thanhhn5 20 0 4976928 463528 17952 S 42.2 8.0 0:34.61 java
21818 thanhhn5 20 0 4976928 518320 17952 S 70.1 8.9 0:36.72 java
21818 thanhhn5 20 0 4976928 518320 17952 S 52.2 8.9 0:38.29 java
21818 thanhhn5 20 0 4976928 518460 17952 S 34.9 8.9 0:39.34 java
21818 thanhhn5 20 0 4976928 528364 17952 S 59.2 9.1 0:41.12 java
21818 thanhhn5 20 0 4976928 523116 17952 S 33.2 9.0 0:42.12 java
21818 thanhhn5 20 0 4976928 523116 17952 S 28.3 9.0 0:42.97 java
21818 thanhhn5 20 0 4976928 523256 17952 S 34.9 9.0 0:44.02 java
21818 thanhhn5 20 0 4976928 549192 17952 S 32.3 9.5 0:44.99 java
21818 thanhhn5 20 0 4976928 549192 17952 S 21.9 9.5 0:45.65 java
21818 thanhhn5 20 0 4976928 549192 17952 S 22.6 9.5 0:46.33 java
21818 thanhhn5 20 0 4976928 549192 17952 S 21.6 9.5 0:46.98 java
21818 thanhhn5 20 0 4976928 549192 17952 S 22.3 9.5 0:47.65 java
21818 thanhhn5 20 0 4976928 549192 17952 S 22.3 9.5 0:48.32 java
21818 thanhhn5 20 0 4976928 549192 17952 S 22.6 9.5 0:49.00 java
21818 thanhhn5 20 0 4976928 566780 17952 S 27.9 9.8 0:49.84 java
21818 thanhhn5 20 0 4976928 566780 17952 S 22.9 9.8 0:50.53 java
21818 thanhhn5 20 0 4976928 566780 17952 S 23.2 9.8 0:51.23 java
21818 thanhhn5 20 0 4976928 566780 17952 S 18.9 9.8 0:51.80 java
21818 thanhhn5 20 0 4976928 566780 17952 S 22.6 9.8 0:52.48 java
21818 thanhhn5 20 0 4976928 566780 17952 S 22.6 9.8 0:53.16 java
21818 thanhhn5 20 0 4976928 596020 17952 S 28.9 10.3 0:54.03 java
21818 thanhhn5 20 0 4976928 596020 17952 S 22.9 10.3 0:54.72 java
21818 thanhhn5 20 0 4976928 596020 17952 S 22.9 10.3 0:55.41 java
21818 thanhhn5 20 0 4976928 596020 17952 S 21.3 10.3 0:56.05 java
21818 thanhhn5 20 0 4976928 602112 17952 S 33.6 10.4 0:57.06 java
21818 thanhhn5 20 0 4976928 621932 17952 S 61.2 10.7 0:58.90 java
21818 thanhhn5 20 0 4976928 627928 17952 S 62.5 10.8 1:00.78 java
21818 thanhhn5 20 0 4976928 708968 17952 S 70.8 12.2 1:02.91 java
21818 thanhhn5 20 0 4976928 709992 17952 S 123.6 12.2 1:06.63 java
21818 thanhhn5 20 0 4976928 710144 17952 S 77.0 12.2 1:08.95 java
21818 thanhhn5 20 0 4976928 710220 17952 S 49.5 12.2 1:10.44 java
21818 thanhhn5 20 0 4976928 749584 17952 S 58.8 12.9 1:12.21 java
21818 thanhhn5 20 0 4976928 750044 17952 S 47.8 12.9 1:13.65 java
21818 thanhhn5 20 0 4976928 750100 17952 S 46.9 12.9 1:15.06 java
21818 thanhhn5 20 0 4976928 750100 17952 S 43.9 12.9 1:16.38 java
21818 thanhhn5 20 0 4976928 750100 17952 S 33.9 12.9 1:17.40 java
21818 thanhhn5 20 0 4976928 750100 17952 S 35.2 12.9 1:18.46 java
21818 thanhhn5 20 0 4976928 750892 17952 S 40.2 12.9 1:19.67 java
21818 thanhhn5 20 0 4976928 750892 17952 S 29.9 12.9 1:20.57 java
21818 thanhhn5 20 0 4976928 750892 17952 S 39.9 12.9 1:21.77 java
21818 thanhhn5 20 0 4976928 750892 17952 S 35.2 12.9 1:22.83 java
21818 thanhhn5 20 0 4976928 750892 17952 S 33.2 12.9 1:23.83 java
21818 thanhhn5 20 0 4976928 774792 17952 S 41.9 13.3 1:25.09 java
21818 thanhhn5 20 0 4976928 774792 17952 S 34.2 13.3 1:26.12 java
21818 thanhhn5 20 0 4976928 774808 17952 S 41.2 13.3 1:27.36 java
21818 thanhhn5 20 0 4976928 774808 17952 S 34.5 13.3 1:28.40 java
21818 thanhhn5 20 0 4976928 776628 17952 S 41.5 13.4 1:29.65 java
21818 thanhhn5 20 0 4976928 776628 17952 S 36.2 13.4 1:30.74 java
21818 thanhhn5 20 0 4976928 776628 17952 S 35.9 13.4 1:31.82 java
21818 thanhhn5 20 0 4976928 778640 17952 S 42.2 13.4 1:33.09 java
21818 thanhhn5 20 0 4976928 778640 17952 S 37.2 13.4 1:34.21 java
21818 thanhhn5 20 0 4976928 767204 17952 S 40.5 13.2 1:35.43 java
21818 thanhhn5 20 0 4976928 767204 17952 S 35.9 13.2 1:36.51 java
21818 thanhhn5 20 0 4976928 767204 17952 S 34.9 13.2 1:37.56 java
21818 thanhhn5 20 0 4976928 770912 17952 S 40.9 13.3 1:38.79 java
21818 thanhhn5 20 0 4976928 770912 17952 S 34.2 13.3 1:39.82 java
21818 thanhhn5 20 0 4976928 770912 17952 S 35.6 13.3 1:40.89 java
21818 thanhhn5 20 0 4976928 760444 17952 S 40.5 13.1 1:42.11 java
21818 thanhhn5 20 0 4976928 760444 17952 S 36.2 13.1 1:43.20 java
21818 thanhhn5 20 0 4976928 759932 17952 S 40.6 13.1 1:44.42 java
21818 thanhhn5 20 0 4976928 759932 17952 S 35.9 13.1 1:45.50 java
21818 thanhhn5 20 0 4976928 759932 17952 S 34.6 13.1 1:46.54 java
21818 thanhhn5 20 0 4976928 779944 17952 S 41.9 13.4 1:47.80 java
21818 thanhhn5 20 0 4976928 779944 17952 S 35.2 13.4 1:48.86 java
21818 thanhhn5 20 0 4976928 779944 17952 S 35.2 13.4 1:49.92 java
21818 thanhhn5 20 0 4976928 807160 17952 S 79.5 13.9 1:52.31 java
21818 thanhhn5 20 0 4976928 807160 17952 S 35.6 13.9 1:53.38 java
21818 thanhhn5 20 0 4976928 807160 17952 S 35.9 13.9 1:54.46 java
21818 thanhhn5 20 0 4976928 808924 17952 S 36.5 13.9 1:55.56 java
21818 thanhhn5 20 0 4976928 808924 17952 S 36.2 13.9 1:56.65 java
21818 thanhhn5 20 0 4976928 808924 17952 S 36.2 13.9 1:57.74 java
21818 thanhhn5 20 0 4976928 811520 17952 S 37.5 14.0 1:58.87 java
21818 thanhhn5 20 0 4976928 811520 17952 S 36.2 14.0 1:59.96 java
21818 thanhhn5 20 0 4976928 811520 17952 S 35.2 14.0 2:01.02 java
21818 thanhhn5 20 0 4976928 814776 17952 S 36.5 14.0 2:02.12 java
21818 thanhhn5 20 0 4976928 814776 17952 S 34.6 14.0 2:03.16 java
21818 thanhhn5 20 0 4976928 814776 17952 S 35.5 14.0 2:04.23 java
21818 thanhhn5 20 0 4976928 814792 17952 S 36.2 14.0 2:05.32 java
21818 thanhhn5 20 0 4976928 814792 17952 S 36.9 14.0 2:06.43 java
21818 thanhhn5 20 0 4976928 814792 17952 S 34.3 14.0 2:07.46 java
21818 thanhhn5 20 0 4976928 814800 17952 S 36.6 14.0 2:08.56 java
21818 thanhhn5 20 0 4976928 814800 17952 S 35.2 14.0 2:09.62 java

@nhanth87
Copy link

nhanth87 commented Jun 6, 2016

Chào Thuy,
Bạn có thể cung cấp email của bạn để bên mình có thể hỗ trợ tốt hơn?
Bạn có thể gửi email của bạn cho mình ở tran.nhan@telestax.com

Br,
TN

@thanhthuy40
Copy link

Dear Nhanth87,
my mail is hangocthanh3107@gmail.com , can you send me mail?

hrosa pushed a commit that referenced this issue Feb 9, 2018
fixed cannot parse string payload format

Approved-by: Henrique Rosa <henrique.rosa@telestax.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants