Gamepads sometimes disconnect from DS following ESD event #703
Comments
Windwoes
changed the title
|
Update: this bug bit me in a real match in Detroit (Ochoa Q-115)... Not only do I have a commented log from both the driver station and the robot controller, but here's a timestamp to the livestream a few seconds before the event for full context. |
|
One notable difference between the behavior I saw in Detroit and that I saw in testing is that in Detroit, not only did the gamepads disconnect, but the OpMode also stopped - in my previous testing, the gamepads disconnected but the OpMode continued to run... |
|
OpModes stopping when a gamepad disconnects is probably a good thing. |
|
@AustinShalit whether it is a good thing or not, it is not the normal behavior. Besides, the question here is why are the gamepads disconnecting from the DS.... |
Unless you are running in autonomous, in which case it's an unnecessary disaster. |
|
@gearsincorg as I mentioned to you in Detroit, it seems that this issue was triggered by protected void onPause()
{
super.onPause();
RobotLog.vv(TAG, "onPause()");
analytics.unregister();
gamepadManager.clearGamepadAssignments();
mInputManager.unregisterInputDeviceListener(this);
initDefaultOpMode();
}Notice that the gamepads are dropped and the default OpMode (StopRobot) is started. |
|
OK, I've processed the two log files from issue #702 and created a combined log with corrected timestamps. Sequence of events This indicates that the RC clock is ahead of the DS clock by approx. 3.3 sec (+/- 0.015 sec) From this I think we can conclude:
Based on this, my conclusion would have to be that the DS phone/gamepad connection was disrupted by the ESD event, but not through the robot, but rather this points to the disruption being transmitted via a EM pulse (ie: radio interference) that is somehow seen at the Gamepad interface, and/or is locking up the WiFi, and confusing the WiFi state machine.. Anyone else see a trigger I'm missing? |
|
@gearsincorg Interesting analysis, a few points of my own:
Edit for clarity: in testing the OpMode did not stop by itself, I stopped it manually. In Q115, the OpMode stopped by itself. |
@FROGbots-4634 Since you have the originals can you do that? Double check my time synching as well. |
I can try, though I'm not entirely sure how you determined the time difference? Did you assume the command arrived at the RC at exactly the time it was sent from the DS? Also, what exactly does this message mean?
|
|
Re: WiFiState change. Yes, I also noticed that. To determine time difference I found an RC log line that must occur between two close DS log lines, and split the 30mS time difference. RC 12.944 - DS 9.65 = 3.3 sec (approx) |
|
Ok I think I understand. One other thing I suppose I could do to test whether EMI is really at play would be to back up say 12FT from the field while driving around and see if I can still reproduce. |
|
@cmacfarl @rgatkinson @tomeng70 What can cause this DS log message: V StateMachine: { 4610 13.977} State with no transitions: WifiState It's happening around the time of the ESD/Gamepad disconnection. Is this expected under any situations, or is it indicative of a problem? |
Yes, that would definitely help to determine if the Gamepad is seeing the EMI, but it would not prevent the WiFi from being effected. |
|
Since you are doing more testing, it would help if you re-synched your phones. It should be enough to just let them both connect to the internet for a short while to get them to do a time update. |
|
Ok, so assuming that the DS is ~115ms ahead of RC, here is a cleaned-up and the time-synced log for Q115: |
|
As I mentioned before, the |
|
Something I noticed on the live stream video.... Your GREEN LED strip appears to go off the split second that your dispenser touches the rim of the lander. This would imply that there is no timeout involved with this issue.... How is this strip driven? Is it a REV Blinky, or some other method? Just seeing if this provides any additional clues as to the trigger event. Does the arm retract require powered control, or is it reflex? |
|
The LED strip is based on the APA102 chipset and is controlled using a very simply SPI protocol. I'm controlling it by using an I2C to SPI bridge which the GDC surprisingly allowed this year. The strip turning off is indeed an indication that the comms link was still very much alive, as the program turns off the strip before stopping: /*
* A stop was requested and we're about to get out of dodge;
* turn off the LED strip so as to not waste power.
*/
for(DotstarPixel pixel : robot.dotstarStrip.getAllPixels())
{
//Set each pixel to off
pixel.queueRgb(0,0,0);
}
robot.dotstarStrip.apply();The dumper arm does not require power to retract; gravity is more than enough to bring it down from full extension. During normal operation I actually power it down in closed-loop mode to prevent it from crashing down unnecessarily hard. I think the reason it didn't come down all the way after the program stopped in the live stream is that power to the dumper servo was cut and thus it wasn't holding position, and I think the dumper caught on the slides as it was going down. |
|
Let me recap... So, from what you are saying, an ESD would not naturally be able to turn ALL the LEDs off. (I assume they hold their last state if there is no more SPI data). From this you deduce that the ESD must have induced the DS to stop the opmode running.... So once again we are left to wonder how the DS knew there was an ESD... Seems like we need some more comms diagnostics to see the actual trigger. |
Yes that's correct.
Well, I'm not just deducing it, you can see from the logs that the DS sends the stop command...
I'm not sure whether we need more comms diagnostics or Android diagnostics. That call to |
|
@FROGbots-4634 Purely on the topic of the Gamepad becoming disconnected, and ignoring for the moment any possible change in app lifecycle.... Over the last cold dry week in MD, I have witnessed several gamepad disconnected that appear to synch with metal-metal contact on the robot. Symptoms: So in these instances is seems to be purely a DS/Gamepad issue, somehow magically being caused by the robot ESD. This is my moment of brilliance: If this IS being induced by high levels of EM radiation, with the gamepad cable being a great receiving antenna, then I wonder if adding Ferrite Chokes to the Gamepad cables might help suppress the effect. One at each end of the cable might be extra effective. Just a thought. |
|
@gearsincorg Well that's actually good that you were able to reproduce somewhat, as that means I'm not entirely out to lunch. When I see it, the OpMode does not always stop. I'm not sure why it would stop sometimes and not others. Oooh, that is an interesting idea. And it certainly can't hurt. I find it hard to believe that EMI from the robot could be strong enough to affect DS several feet away, but without putting a scope on the gamepad wires we're just guessing right now... |
|
The issue is: I don't know what needs to happen protocol-wise on the USB for the phone to consider the gamepad disconnected. It's hard to believe that a single bad packet transmission would be enough of a problem to disconnect. But perhaps a handshake gets corrupted and something times out. I still wish we could do automatic reconnects. |
|
@gearsincorg So here's something interesting..... I'm not sure if the fact that it's a totally new robot has anything to do with it, but this year we're using Xbox controllers and the same USB hub on the DS and I have yet to see the gamepads drop, even across dozzens of audible arcs to/from the robot. |
|
Need to go back to Logitech and see if dropouts re-appear.
|
|
I was FTA at a qualifier on Dec 7, where in one match we had three robots become disabled. Two of them exhibited common symptoms of USB disconnection between the RC phone and REV hub, but the third robot lost driver control and ran across the field. The team was able to stop the opmode. After the match, we started the opmode again, and the robot continued to drive straight, without any driver input. We stopped the opmode, disconnected and reconnected the USB hub to the DS phone, and restarted the opmode, and the robot was back to normal. So evidently there was a malfunction of the controller, USB hub, or DS phone, which persisted across opmode restart, and was cleared by disconnecting the USB hub from the DS phone, and it happened during a match where circumstantially it seemed like other robots were having static-related issues. I'm checking to see if the team still has logs from that match. |
|
I've seen that fairly often, I've attributed it to a flaky connection between the gamepad and the DS phone. It temporarily disconnects, and when it reconnects it re-zero's the gamepad at whatever point everything is at that moment. So all of a sudden full forward becomes 0 and neutral becomes full backwards. My process for helping teams during a match with it is below, usually taking 10-15 seconds.
This is also mentioned in the Control System Troubleshooting Guide, section 11.2.3. |
Yeah, I should try that at some point.... I still have yet to see the Xbox 360 controllers drop from DS after many, many ~1500ms or so USB dropouts on the robot due to ESD. |
and others
This has got to be the weirdest bug I have ever seen, but I have been able to reproduce it nearly 10 times so I'm very confident in my analysis.
Sometimes, right after an ESD event, the gamepad will inexplicably disconnect from the driver station. All that is required to reproduce it is to drive the robot around the field to build up static charge, and then drive it so that it touches an object at a lower potential (or a grounded object); thus creating an ESD event. I have been able to reproduce this with 1) ESD from the field wall 2) ESD from the lander leg 3) ESD from a wheel of the robot touching the metal base of a floor lamp. I actually managed to catch that last one on video. Please ignore the warning message on the driver station; I had a webcam disconnected during this test.
Logs from the DS and RC of this encounter can be found on my other issue that I opened today. (At first I thought that previous issue and this were related and thus posted the logs there; but I now believe they are separate and apart issues)
The text was updated successfully, but these errors were encountered: