GM camera ACC: LKAS faults

[sshane]

• Fault when relay opens (we send counter of 1, should copy camera's counter) - GM camera ACC: reduce LKAS faults on startup #26039
• Fault when relay opens with above fix - We can be sort of unlucky and try to send a message just before the relay opens, blocking it, then reading an updated counter from the camera (that doesn't reach the car), and end up skipping a counter. It's rare, but still a possibility. - GM camera ACC: reliable relay open init #27163
• Random LKAS faults -> perm faults - GM: enforce steering command message timing #27250
• Fault when relay closes (our counter doesn't sync up with camera's) - GM camera ACC: reliable relay closing #27164

[ncoop23]

0def4a390f6fe5c0|2022-10-25--19-17-40--35

Just had this happen today driving along straight and level near the end of the drive. Also had a little steering tune issue as well.

2022 EUV Premier non SC

I think this was the other time it happened.

0def4a390f6fe5c0|2022-10-22--14-22-47--23

[dominicdoty]

I'm in a 2017 Bolt on an opgm/newcombo variant, but in case it helps your debugging, I had an LKAS fault while driving today on the segments:
82a0516f27e12844\|2022-11-04--18-03-04--15
82a0516f27e12844\|2022-11-04--18-03-04--16

[abueide]

@sshane we ran into this on our bolt euv while driving on a straight road (roughly 3 hours of drive time after we installed the comma 3) and it scared the crap out of us xD. Is there anything we can do to help fix this? We're pretty new to openpilot development so any advice would be appreciated! also we were on 0.9.0 stable in a 2022 bolt euv

[minorOffense]

Just happened to me today on the bolt-torque branch (0.9.1) on a 2022 Bolt EUV

[dominicdoty]

Had another LKAS fault recently, my branch which is synced to comma master LKAS sync code.

I've had LKAS faults on this exact segment of road under similar conditions 3 or 4 times, leading me to think there's some reproducible issue there.

Its a long left hand sweeping on ramp with a meter light at the end. OP disengaged, heavy traffic, stop and go at 0-10 mph.

82a0516f27e12844|2022-12-08--17-07-17--22

[ncoop23]

It seems these errors are happening more often on the EUV. They do not always clear on a vehicle power cycle. I turn off the car, open then close the door, wait for the dash to go dark and then power on the car. The comma still says LKAS fault. Then I drive for a while as I have somewhere to go and don't care about the comma working or not and the cruise control won't work so I unplug the comma and there's a terribly annoying beeping and the cruise control works. This is very frustrating and makes me want to return the comma since at this point about 25% of my drives, it doesn't work. The LKAS error came up at some point when the vehicle was sitting charging tonight.

[abueide]

@ncoop23 you need to wait at least 3 minutes between closing the door and turning the bolt back on, seems to fix the error every time for me at least. This is cause the car doesn't completely shut off until after 3 minutes of being "off".

[sshane]

That's correct, I've observed 2-3 minutes with the car fully off to reset the fault. @ncoop23 when do these faults mostly happen for you? While driving? When first turning the car on? Or shortly after turning the car off?

[sshane]

@dominicdoty I've requested upload, can you put your device on wifi?

@abueide let me know if you have a segment where it happens and I can take a look

[ncoop23]

That's correct, I've observed 2-3 minutes with the car fully off to reset the fault. @ncoop23 when do these faults mostly happen for you? While driving? When first turning the car on? Or shortly after turning the car off?

I generally get them when driving for a while. Some have been near turns, other on the straight and level. I think there may be a buffer overflow or a command that needs an ack that doesn't get it like a calibration or check every so often. About to find and upload the latest one. Car is syncing. They are random. The strange thing is something latches as if I disconnect the comma there is a terribly annoying beeping coming from somewhere and I don't know if it's the comma.

[abueide]

@sshane i'll figure out how to use the segments system and keep my eye out for it happening again and post back here if I do. so far i had one single fault happen while driving (this was on stable 0.9.0 no experimental) in about 400 miles. we also had one that happened shortly after parking turning the car off, then realizing i needed to repark and turning it on again (not enough time for a full restart) it happened again.

[dominicdoty]

@sshane Looks like the first one I posted I must have unpreserved, the one from 12/8 should have uploaded now - apologies for the delay.

Let me know if there's anything I can do to help out. These errors are pretty rare for me, but would be nice to get them fixed.

[jelimoore]

@sshane this is @abueide 's recent LKAS failure route ID: c9dc069b00b41849|2022-12-14--14-11-59--26. It happens right at the end so you may want c9dc069b00b41849|2022-12-14--14-11-59--27 as well.

[dominicdoty]

another one, back to opgm fork, exact same location and conditions
82a0516f27e12844|2022-12-15--18-01-41--17

[minorOffense]

Happened to me again at 10:22 to 10:23am on Thursday dec 22nd.

6afbe9126b0b488e

I went to look for the video on connect but it’s not showing up. Maybe the logs still exist but I can see them. If it shows up I’ll post a proper link here.

[minorOffense]

Got one uploaded finally.

6afbe9126b0b488e|2023-01-02--12-35-07--2

Check around 12:37 that’s when it triggered.

I set the flag on connect to preserve this drive.

And in case I forgot to mention it I drive a 2022 Bolt EUV Premiere (No SC). Running on master-ci branch.

[minorOffense]

And another at 10:56

6afbe9126b0b488e|2023-01-03--09-42-55--0

[minorOffense]

And again at 13:58

6afbe9126b0b488e|2023-01-03--12-54-29--0

[ncoop23]

So what happens to correct this issue? It's kind of a big deal since it causes the comma to abort and need a car restart.

[minorOffense]

bump

[sshane]

I've been busy with other work recently, but will look at all the issues. Make sure if you report a segment to upload all logs and preserve

[sshane]

@minorOffense I can't guarantee this will work, it's just a hypothesis I have, but can you try this branch?

https://github.com/commaai/openpilot/tree/gm-full-rate-fault
Installer URL: https://installer.comma.ai/commaai/gm-full-rate-fault

The two faults you posted the car had a temporary fault followed by a permanent fault, it could be possible that openpilot slowing its send of the inactive steer frame could be exacerbating the issue during the temp fault, leading to a perm fault.

Please drive on it for a few days and let me know with a route if you get any more temporary faults, and if it recovers or not.

[ncoop23]

@minorOffense I can't guarantee this will work, it's just a hypothesis I have, but can you try this branch?

https://github.com/commaai/openpilot/tree/gm-full-rate-fault Installer URL: https://installer.comma.ai/commaai/gm-full-rate-fault

The two faults you posted the car had a temporary fault followed by a permanent fault, it could be possible that openpilot slowing its send of the inactive steer frame could be exacerbating the issue during the temp fault, leading to a perm fault.

Please drive on it for a few days and let me know with a route if you get any more temporary faults, and if it recovers or not.

I'm driving around on this now. So far so good. The steering tuning isn't damped enough but that's something else.

[ncoop23]

Had what looked like 2 steering errors in quick succession on your test build yesterday. Nothing caused comma to fault, only warn. 0def4a390f6fe5c0|2023-01-20--21-51-56--12

Please share if this is the actual error. I flagged a couple other things also.

[minorOffense]

I’ve installed it and I’ll let you know.

[jettoblack]

@sshane I have been testing this change (applied to another branch of SunnyPilot) for a few days and thought the LKAS fault had been resolved, unfortunately just now I got the error TAKE CONTROL NOW! LKAS Fault - Restart car to engage.

48c21cb1105d4d1c|2023-01-23--19-30-58--5

Happened around 19:36:10. I was driving mostly straight at the time. 2022 Bolt EUV. Let me know if there's anything else I can do to help.

[ncoop23]

Got a take control now error yesterday on straight and level freeway with this branch. Same non SC EUV Premier

0def4a390f6fe5c0|2023-01-22--20-13-34--70

[minorOffense]

I’ve been stuck at home with COVID all week so I haven’t had a chance to test. Hopefully next week I’ll get a chance.

[sshane]

I have a branch now to reduce the send rate of the LKAS message to the EPS, as we suspect that sending two LKAS messages too close together is causing faults (undetectable by openpilot at just 100Hz). This branch changes the 50Hz message down to just 33Hz, so steering may be impacted, but if this hypothesis is correct, faulting should be reduced.

Let me know how this branch does @ncoop23 @jettoblack @minorOffense

Installer URL: https://installer.comma.ai/commaai/gm-reduce-rate

https://github.com/commaai/openpilot/tree/gm-reduce-rate

[fredcredeur]

Are you able to see how many people are currently using a branch? Just curious.

I am installing the gm reduce rate branch and will report issues.

[ncoop23]

I have a branch now to reduce the send rate of the LKAS message to the EPS, as we suspect that sending two LKAS messages too close together is causing faults (undetectable by openpilot at just 100Hz). This branch changes the 50Hz message down to just 33Hz, so steering may be impacted, but if this hypothesis is correct, faulting should be reduced.

Let me know how this branch does @ncoop23 @jettoblack @minorOffense

Installer URL: https://installer.comma.ai/commaai/gm-reduce-rate

https://github.com/commaai/openpilot/tree/gm-reduce-rate

I can load it to try. Did you see that in mine that double errored?

[sshane]

I just merged two fixes to help significantly reduce LKAS faults when starting and stopping your vehicle a few times. Please try out master branch and let me know if those faults are gone now! #27164

[fredcredeur]

I just merged two fixes to help significantly reduce LKAS faults when starting and stopping your vehicle a few times. Please try out master branch and let me know if those faults are gone now! #27164

Will you update the GM Reduce Branch or should we uninstall this branch to try out the master branch? On my 3rd day with GM Reduce Branch and have had zero LKAS Faults.

[sshane]

Haven't tried that branch yet myself, how is the steering control? Hopefully it should be the same. And yes I'll update it

[fredcredeur]

All week so far I thought the steering felt better. Actually noticed and thought about it multiple times that it just felt perfect like it centered the lane very well. Until my drive home a while ago. For some of the trip it felt like it just kept hugging the line opposite of the center line. A couple of times I made a point to nudge the vehicle back center and it would move back. Made a point to try this in the left lane and right lane and it did the same thing in both. While it did this it felt like it needed to make more corrections (like it ping-ponged a bit more). Maybe it was the weather. Roads are wet today? All week up until my drive home just a while ago id say it's been flawless. Even with the hugging of the line Im satisfied without getting that LKAS Fault.

[Verylukyguy]

I know that I am not a Camera Car, but I just loaded a fresh fork of Comma Master and now I am getting a "LKAS Fault: Restart Car to Engage" error on startup that clears on its own without a restart.

Edit: The fault was/is a Cruise Fault. I was confused because the Cruise Fault did not usually say “Restart the car to engage.” Which is what I normally see with LKAS Faults.

[sshane]

Can you send a segment @Verylukyguy?

Edit: that's a cruise fault, not related to recent changes

[sshane]

@fredcredeur updated that branch. Please try gm-reduce-rate again, making sure to update, for the next week to stress test it. Let me know if ever get a fault from starting and stopping your car many times, that should be gone!

[sshane]

@fredcredeur can you send your dongle ID?

[minorOffense]

I got three temporary failures in a row but they didn’t trigger the lkas fault which is good. But yeah each time I tried to rengage the same cruise setting it would temporary fault and I’d take over.

I’ll upload the logs and maybe there’s something about the cruise state I was in that it didn’t like (speed maybe had weird rounding or decimal errors, who knows). Cause once I changed the speed it stopped temporary faulting.

[sshane]

I got three temporary failures in a row but they didn’t trigger the lkas fault which is good. But yeah each time I tried to rengage the same cruise setting it would temporary fault and I’d take over.

I’ll upload the logs and maybe there’s something about the cruise state I was in that it didn’t like (speed maybe had weird rounding or decimal errors, who knows). Cause once I changed the speed it stopped temporary faulting.

Please send a route with issues so I can check it out

[minorOffense]

6afbe9126b0b488e|2023-02-02--19-17-59--0

There was three faults on this route. Around 19:20 I think they started. I was actually on the old gm branch you had created though at the time.

[sshane]

Hey everyone, the random permanent LKAS faults should be fixed in master, master-ci, and nightly branches. Please let us know if you try these branches and see anything strange! Final PR: #27250

[minorOffense]

I’m still on that other branch and got some process communication failures. Not sure if it’s related.

6afbe9126b0b488e|2023-02-09--10-04-26--0

Anyways I’ll switch back to master and test there too. Thanks.

[ncoop23]

Got another straight line red error a couple days ago:
0def4a390f6fe5c0|2023-03-09--10-36-17--93
I posted in tuning, is there a good place to be able to get better at describing an error? The car isn't always in a place where I can SSH into it but I am connected to connect to pull logs and such.

[sshane]

@ncoop23 that's not an LKAS fault, that's a temp ACC fault. With an LKAS fault you would not have been able to re-engage. Looks like this ACC fault was caused by panda keeping controls allowed high when you pressed the main button to disengage, then re-engaged 2 seconds later. Cross posted to this issue to track this: #27479

[kjgarrett23]

Hey, I’ve got a 2020 Silverado that uses sunny pilot. I get LKAS when driving long ways for work. It’ll pop up anytime 45 min to two hours but will always happen on a drive longer than 2 hours. Any thoughts?