Capstone Simulator Latency

[wkhattak]

No description provided.

[wkhattak]

Facing severe latency/lag when using the simulator for the capstone project. This happens specifically when the camera is turned on. Also, when the generated waypoints in the waypoint-updater node are set to the default 200 (camera off), the same latency appears (seems kind of OK if the waypoints are reduced to 50).
I have a Windows 7 machine with i7 / 32 GB of RAM and a GPU. I first used the simulator inside the VM (with 6 CPU allocated + around 12GB RAM). Then I moved on to a native install by converting the provided VM to a physical drive via Clonezilla. However, I don't see any decrease in latency even after moving to a native install.
I contacted support & was told that there are no known issues but obviously there are. Apart from getting frustrated and wasting a lot of time, this is stopping me from completing the final project in a timely fashion. A quick search over Slack & the forum reveals that a number of students are facing the same issue. I request for a speedy resolution of this issue. Thanks

[aup1907]

I am having the same issue on two different machines

[mvirgo]

Capstone project has now been moved to a workspace to help isolate any particular issues - please re-open if you see the same issues arising within the workspace.

[MyCodeBits]

@mvirgo I have been using the workspace recently.
Things stay good till the time camera is NOT ON.
With camera being ON, Car barely manages to stay on track for 10 seconds and then goes off.
So workspace does not seem to be the solution for the problem reported. Not sure if its dependent on combined workspace load across other workspaces in the cluster where its hosted.

BROWSER: Chrome

Video at 640x480 with Graphics being Simple.

• CAMERA ON at July 24th 6:50 PM PST - Car goes off track
  https://drive.google.com/file/d/1NrsOC8JU-d9X2XXB3pPnrqZMm-JCDIrX/view?usp=sharing

However, even with CAMERA OFF at 6:50 PM, car goes off track.

• CAMERA OFF at July 24th 6:50 PM PST - Car goes off track https://drive.google.com/open?id=1c9bLY9H7j_jPJpIFvrhqt5MDT-tN45Xu

I tried ~ 2 hours later and now with same code base car stays in track with CAMERA OFF.

• CAMERA OFF at July 24th 8:58 PM PST - Car stays in track https://drive.google.com/open?id=18BykAoUnGduUNMfvxvoeVApSb4rdN8PS
  -- Most probably load by this time on cluster would have reduced (my conjecture).

I have provided timelines if someone wants to analyze cluster health based on how many workspaces were active.

CC @awbrown90

[mvirgo]

@MyCodeBits Can you post your internet connection speed as well as what browser you are using?

[MyCodeBits]

@MyCodeBits Can you post your internet connection speed as well as what browser you are using?

Hi @mvirgo. Thanks for quick response. Earlier videos/tests were using Chrome Browser in last comment.

Internet connection speed

Chrome on Host m/c (mac OS)

Firefox inside Workspace

Created another video of what the current state is at the workspace using Firefox on host (Mac OS) accessing Workspace at July 25th 3:00 PM PST

• Car without CAMERA ON goes off track. https://drive.google.com/open?id=1mSMFCYRNuoPm-sZUnOWjaLejragU5FU9

I would be happy if you want me to try any instrumented code for any reason.

Further please note that there are other people in my team who are facing the same issue who are geographically at different locations and one person has commented in Slack #general also for this after i posted for the same issue.

My understanding is that given the workspace is remote, if one has a decent enough internet connection, the connection speed in remote workspace only should be of consideration. Please comment if i am missing/over simplifying.

CC @SIakovlev

[jonam85]

On the observation for Workspace getting struck: till the manual operation is present, the terminal runs at nominal pace. when the manual checkbox is unchecked, then the program runs at a slower pace.

For each image topic received

With automatic mode (manual checkbox unchecked, camera checked)
Actual Time Delta time from previous
1532625293.0018000000 0.4056799412
1532625293.5656900000 0.5638899803
1532625293.9795300000 0.4138400555
1532625294.5536900000 0.5741598606
1532625295.1094800000 0.5557899475
1532625295.5538500000 0.4443700314
1532625295.9139400000 0.3600900173
1532625296.2937000000 0.3797600269
1532625296.8977800000 0.6040799618
1532625297.3851800000 0.4874000549
1532625297.7796800000 0.3945000172
1532625298.2115300000 0.4318499565
1532625298.6865300000 0.4750001431
1532625299.1741400000 0.4876098633
1532625299.5141200000 0.3399801254
1532625300.0766600000 0.5625398159
1532625300.5452900000 0.4686300755
1532625300.8820800000 0.3367900848
1532625301.2525100000 0.3704299927
1532625301.6422000000 0.3896899223
1532625302.0498000000 0.4075999260
1532625302.4078500000 0.3580501080
1532625302.7898200000 0.3819699287
1532625303.1111800000 0.3213601112
1532625303.4052600000 0.2940800190
1532625303.9557500000 0.5504899025
1532625304.3985300000 0.4427800179
1532625304.7305400000 0.3320100307
1532625305.1275000000 0.3969600201
1532625305.5279200000 0.4004199505
1532625305.9520800000 0.4241600037
1532625306.3499500000 0.3978700638
1532625306.7005000000 0.3505499363
1532625307.1803400000 0.4798400402
1532625307.6294700000 0.4491300583
1532625308.0477500000 0.4182798862
1532625308.4741300000 0.4263799191
1532625308.8001100000 0.3259801865
1532625309.1900000000 0.3898899555
1532625309.6295500000 0.4395499229
1532625310.0781400000 0.4485900402
1532625310.3754400000 0.2972998619
1532625310.8115100000 0.4360702038
1532625311.2482000000 0.4366898537
1532625311.6494300000 0.4012300968
1532625311.9611100000 0.3116800785
1532625312.4449300000 0.4838199615
1532625312.8244400000 0.3795099258
1532625313.2389800000 0.4145400524
1532625313.6539400000 0.4149599075
1532625314.0785500000 0.4246101379
1532625314.7061800000 0.6276299953
1532625315.1100900000 0.4039099216
1532625315.3070800000 0.1969900131

With manual mode (manual checkbox checked, camera checked)
1532625315.3842100000 0.0771300793
1532625315.4616300000 0.0774199963
1532625315.5367000000 0.0750699043
1532625315.6145000000 0.0778000355
1532625315.6913000000 0.0767998695
1532625315.7682300000 0.0769300461
1532625315.8403500000 0.0721199512
1532625315.9141200000 0.0737700462
1532625315.9960000000 0.0818800926
1532625316.0744300000 0.0784299374
1532625316.1645600000 0.0901300907
1532625316.2793400000 0.1147799492
1532625316.3689000000 0.0895600319
1532625316.4614500000 0.0925500393
1532625316.5559000000 0.0944499969
1532625316.6602500000 0.1043498516
1532625316.7534200000 0.0931701660
1532625316.8396400000 0.0862197876
1532625316.9276100000 0.0879700184
1532625317.0184100000 0.0908000469
1532625317.1038900000 0.0854799747
1532625317.2116200000 0.1077301502
1532625317.2978300000 0.0862100124
1532625317.4037900000 0.1059598923
1532625317.4947600000 0.0909700394
1532625317.5928800000 0.0981199741
1532625317.6964200000 0.1035399437
1532625317.7957400000 0.0993199348
1532625317.8946900000 0.0989501476
1532625317.9857900000 0.0910999775
1532625318.0801800000 0.0943899155
1532625318.1683300000 0.0881500244
1532625318.2614400000 0.0931100845
1532625318.3467600000 0.0853199959
1532625318.4351000000 0.0883400440
1532625318.5383200000 0.1032199860
1532625318.6375100000 0.0991899967
1532625318.7277300000 0.0902199745
1532625318.8281400000 0.1004099846
1532625318.9242300000 0.0960900784
1532625319.0219600000 0.0977299213
1532625319.1298200000 0.1078600883
1532625319.2173200000 0.0874998569
1532625319.3214600000 0.1041400433
1532625319.4047700000 0.0833098888
1532625319.5123000000 0.1075301170
1532625319.6167000000 0.1043999195
1532625319.7118300000 0.0951299667
1532625319.8026800000 0.0908501148
1532625319.8751700000 0.0724899769
1532625319.9713200000 0.0961499214
1532625320.0650900000 0.0937700272
1532625320.1593000000 0.0942101479
1532625320.2509800000 0.0916798115
1532625320.3423000000 0.0913200378
1532625320.4300000000 0.0877001286
1532625320.5175900000 0.0875899792

1532625320.6145500000 0.0969600677
1532625320.7099100000 0.0953598022
1532625320.8370000000 0.1270899773

The top rows are when Manual is unselected and Camera is selected. Bottom rows are when both Manual and Camera selected. So the camera processing is not creating any problem. When manual is selected, the delay between two topics received by our program is about 0.5 seconds whereas in the other case it is <0.1 seconds.

Left side is the clock info, right column is the delta between previous to current clock.

[mvirgo]

Couple items for further clarification and testing on our end -
1 - How is the performance of the simulator within each browser?
2 - Are you running a detection model when you unselect Manual mode? That is when the code starts calling to a detection model (or attempting to). Depending on how you've coded this call, or how large the model is (if one is included), can have a big impact here.

[swapanshridhar]

@mvirgo

Couple items for further clarification and testing on our end -
1 - How is the performance of the simulator within each browser?

I did try with Safari and Chrome with not much changes. Performance was bad on both.

2 - Are you running a detection model when you unselect Manual mode? That is when the code starts calling to a detection model (or attempting to). Depending on how you've coded this call, or how large the model is (if one is included), can have a big impact here.

When I am testing my code, there is no detection code deployed. You can assume it to be walkthrough code getting tested, where car will move forward ignoring Traffic Lights.

CC @SIakovlev

[lukavia]

I have the same problem. Since I see the issue is closed, is there a solution to the problem or not?

[pkpp1233]

I have the same problem as well. In workspace and on my laptop.

[ivovandongen]

Performance is still very bad. To the point of being unusable. Any way to get better performance out of the simulator when the camera is turned on?

[Elgeweily]

Same issue here....I am not doing any TL classification yet, and the simulator is still very slow when I turn the camera on.

[papiot]

I am also facing the problem to the point that I cannot use the workspace to work on the final project. Any ideas/fixes?

[Elgeweily]

I solved it by switching to native Linux installation, I heard Docker also works pretty well, these are your 2 options, forget about the VM or the Workspace.

[papiot]

@Elgeweily Thanks, I will do the same then!

[loewner]

I have the same problems. Is there any other solution beside installing on native Linux? (I do not wanna install ros on my client ubuntu machine)

[lukavia]

I've never found one. At the end I did some optimizations and submitted on the blind

[Elgeweily]

@loewner try Docker then....this issue is not addressed properly by Udacity

[loewner]

@Elgeweily Thanks, Docker worked for me as well.

[deybvagm]

I have the same problem with the workspace, I tried the docker option locally but the car still goes off, although I don't have a GPU. My system info is:

OS: Ubuntu 18.04
Processor: intel core i7 @ 2.00 GHz
RAM: 8 GB

My question is: is a GPU required in order to be able to run the project?

[johnadams2076]

I have the same problem.

[heikoschmidt1187]

Same here, lags with VM ROS and local simulator, but somehow at least for a part of the track usable if waypoint frequency is reduced to 10Hz. Workspace really bad, Docker only usable without Camera.

I'm using Arch Linux locally, so in theory I could run native ROS, but unfortunately Opencv3 in ROS doesn't work.

My machine is a Intel Core i7 Notebook with 16GiB RAM and an nvidia gtx 950m

[haritaparekh]

Any solution for the problem?

[marcin-sielski]

Hopefully this article can help
https://medium.com/@marcin.sielski/capstone-project-for-self-driving-car-nanodegree-program-3c54c3c704bf

[drugescu]

This remains an issue to this date, our whole team is stuck for now. Really disappointed that Udacity did not even care enough to answer. So what if it's 1 in a 100?

Some HW combinations seems to trigger this, others which are slightly off, do not. I guess we will submit with "hacks" and hope Carla handles it fine.