Add core OP support for CRC validation, with extra support for Volkswagen MQB #836
Conversation
patches (commaai#582)
Pull in changes from OP 0.5.11
Bring current with upstream OP 0.6.1
Merge changes for OP 0.6.2
* Add 2018 Civic Hatchback 1.0l CVT support test_fingerprints returns fine
3d90c78 Fix wrong event enum f4c92cf add none/invalid usb power mode 12bc189 Add fields and states required for robust Volkswagen safety compliance (#8) 16b5850 let's spell wolkswagen 6243622 Safety cereal (#9) 5cf3c2c change order of UsbPowerMode to preserve panda lib behavior 18a2917 usbPowerMode is more useful to log and it comes from panda 1526784 add ldw to visual hud alerts (#5) 24f6fa7 merge commaai/msgq#3 after fixing conflicts 4d5862c added event about internet connection ca070f9 Added usbPowerOn to health e2eaa76 add HW type for UNO e528a20 Read RPM from fan connected to Panda 8472175 Added invalidGiraffeToyota event 71be970 placeholders for mazda, nissan and vw safety models git-subtree-dir: cereal git-subtree-split: 3d90c78
30c7ca8 bump version to 1.5.3 9403dbe Need to fix wifi test before re-enabling. 0812362 GPS UART fix until boardd is refactored (commaai#294) ffbdb87 python2 -> 3 fixes to pedal flasher (commaai#292) 78b75ef Added build type to release version strings 736c2cb Fixed sending of bytes over PandaSerial 0894b28 Fixed USB power mode on black (commaai#291) 4b3358c patch to be able to switch from EON to PC with a Panda that has EON b… (commaai#290) a95c44a Made setting of NOOUTPUT on no heartbeat more efficient (commaai#287) 9486836 UART instability fix with high interrupt load (commaai#283) 9a9e9d4 Fix usb_power_mode missing initialization (commaai#289) af0960a DFU fix (commaai#288) 70219d7 match safety enum in cereal (commaai#285) a338d39 Fix build for jenkins test 78ef4a6 Stop charge (commaai#284) 5266a40 Fix typo (commaai#286) f4787ec Revert "turn on CDP when ignition switches on (commaai#281)" d37daee Revert "NONE and CLIENT should be the same thing in white/grey pandas" e97b283 NONE and CLIENT should be the same thing in white/grey pandas 8c1df55 turn on CDP when ignition switches on (commaai#281) 847a35d Fix bullet points fac0277 Misra update (commaai#280) 5a04df6 Added description of regression tests to README c4aabae Fixed some python3 bugs in the test scripts and PandaSerial 9af0cb3 Bump version c4ac3d6 Disable GPS load switching on black pandas 078ee58 This is the correct table, actually 578b95e Misra table of coverage added d383a26 bump panda b98ca01 fix sdk build in python3 env (commaai#279) 63d3dc7 Set python3 env before runnign get_sdk, so we know if it fails e951d79 legacy code we don't control can remain python2 11b7151 Merge pull request commaai#276 from commaai/python3 9893a84 Merge pull request commaai#277 from zorrobyte/patch-1 d326869 Revert "revert back esptool to python2 and force to build esptools with python2" 875e760 revert back esptool to python2 and force to build esptools with python2 9c40e62 needed to install python3 ed2ac87 Also moved safety tests to python3 6842b2d move esptool sdk installation before python3 env is set. Kind of a cheat b5a2cab this hopefully fixes build test 6280509 Fixes safety replay 2c220b6 this fixes language regr test fdbe789 use python 3 in Docker container ee1ae4f Better hash print 0de9ef7 Revert "Final 2to3 on the whole repo" c92fd3b Final 2to3 on the whole repo 5f2bc44 better b2a30fd make works! b74005d fix sign.py fe72770 read file as byte and no tab before sleep 32a344e Update README.md 2dc3409 2to3 applied ffa68ef undo unnecessary brackets for print dbc2480 Fix all the prints with 2to3, some need to be undo 5a7aeba xrange is gone 982c4c9 one more python3 env 1e2412a env python -> env python3 git-subtree-dir: panda git-subtree-split: 30c7ca8
f3b573559 move generator to python3 9efff4086 2019+ New Prius Steer Angle (commaai#189) f3461d143 add units and a couple new signals for toyota (commaai#188) 16033b3ff Remove non ascii characters de162d8c2 Toyota no dsu: fix steer angle factor, it's 1% of a rad 2fb524554 Lexus CT200H seems to have the safetyParam 1 instead of 0.73 b2fde8b11 toyota time signal (commaai#187) 6cfe0c432 Fixed brake signal unit in Bosch Honda git-subtree-dir: opendbc git-subtree-split: f3b573559f1619346f082554a72cebcc43da5275
|
Nice. I would like to merge this in. 3 things:
As we discussed, I agree I don't like canonicalizing the dbc files. On the other hand, we can come back to this after we have a MVP VW port. Thanks! |
|
Thanks for the review. Per our DM discussion I will push a commit to rename things to CHECKSUM for now, and we'll follow up with support for the correct MSG_NAME_[CRC|BZ] scheme later. I already have the conflicts fixed, and I will look at adopting the start bit generalization from your comment. It may be a few days until I can follow up, because my CAN bus implodes when I try to run 0.6.5 for reasons I have not yet identified, and I don't have cycles to hack on it further today. |
# Conflicts: # common/params.py # selfdrive/boardd/boardd.cc # selfdrive/car/toyota/values.py
# Conflicts: # selfdrive/can/process_dbc.py
Done per our discussion.
The PR has been brought current with OP 0.6.5-devel.
I've refactored parser_dbc.py to be cleaner, hopefully in the way you were thinking. We can't really verify exact start bits per se, because those messages move around for at least some cars. Honda seems to do this a lot. The existing code was really just checking alignment for the most part. The refactor doesn't check it in precisely the same way (it will allow a Honda checksum to start on bit 7, or a Honda counter to start on any odd bit) but I trust it complies with the spirit of your request. There's really no other way to do this in a manufacturer-generic way. The refactored code does the alignment check based on endianness, and along the way it checks the DBC for correctness. It found two issues in the 2016 Accord Touring DBC, and you'll notice the Travis build for this PR fails at that point. I've submitted commaai/opendbc#193 to fix it properly. |
|
Two final comments: First, my 0.6.5 merge for conflict resolution seems to have added some new-in-0.6.5 files to the PR. My git-fu is not that strong and I'm not sure of the right way to pull them out, I wonder if deleting them might make things worse? Let me know if you can fix this, or if I should withdraw and resubmit the PR from a clean 0.6.5 fork. Second, you'll notice some extra instrumentation to print the name of the DBC when a fault occurs. In the past, depending on when and why process_dbc.py ran, you might get no information at all about which DBC failed to process, or worse, something extremely misleading if there's more than one file with problems. Here's an example of two files broken, Honda and VW. The make is triggered by the Honda DBC change, but the process_dbc failure is actually from the VW file:
Example of a situation where you'd normally have no context at all, but the refactored code tells you what DBC it's working with now:
|
|
RE: added files: yeah, just delete them. You also want to add the opendbc changes to this PR (at least the accord fix), so tests won't fail. Don't worry about how clean is the history of commits of the PR, I'll squash all your commits before merging. As long as the diffs are good and there are no conflicts reported, we are good. And nice addition printing the dbc file that fails. |
Done.
Done.
Looks good now.
Hulk squash!
Thank you for your support through this process! |
* fix ALCA module * Update ALCA_module.py * Promote panda fixes from alpha to devel (#98) * Fix for mock car crash * fix ALCA module * do not use * radar_interface logic improvements * remove radar logging * radar_interface refactoring * Log radar_interface CAN error, Process Comm Errors (#95) * Log radar_interface CAN error * Add support for process comm error logging * Improve Process Comm error logging * Can and CommError throttling * Update radar_interface.py * fixing push per Raf's comments * ALCA and radar_interface work (#96) (#97) * do not use * radar_interface logic improvements * remove radar logging * radar_interface refactoring * Update radar_interface.py * fixing push per Raf's comments * fix process comm issue * switch radard from lane_parser to model * radar/ALCA changes * fix lane width * switched back to not deleting CarParams to avoid radar failure when rebooting EON while driving * cleanup some debug info to be able to restart ALCA research * tweaks on safety_tesla.h for can bus isolation * few more cleanups to safety_tesla.h * fix start signal logic * Increase LEXUS_IS safetyParam to 77 (#841) * use list comprehension (#839) * use list comprehension * oops index range fix * simplify encode/decode in dbc (#840) * simplify encode/decode * 2019 Highlander Hybrid Limited Platinum (#843) fingerprint from ogdragonzypher on Discord * Allow to lock safety mode to keep gm/tesla cars supported (#844) * Force subaru to assume the stock camera is connected (#849) * Revert to using the sum of the lane line probabilities instead of the the product * Add core OP support for CRC validation, with extra support for Volkswagen MQB (#836) * Generalized core OP CRC support plus extra bits for Volkswagen MQB. * Update ref to process_replay regression test * Remove unneded use of replace, causing json to fail loading the alert * Add supported car for 2019 Toyota Altis Hybrid (#859) * add supported car for 2019 Toyota Altis Hybird / Corolla hybrid 2020 (US market) * remove unnecessary comprehensions (#863) * Added fingerprint from Sienna XLE AWD (#848) * Added fingerprint from Sienna XLE AWD * Update driver_monitor.py * Update driver_monitor.py * Update driver_monitor.py * fix stupid merge issue * Adds 2019 Lexus ES (non hybrid) (#866) * Adds 2019 Lexus ES (non hybrid) * Updated test route * merge issues * Update README with now supported Subaru Harness * Disable LDW for 4 seconds after blinker transition from on to off (#112) If we want to change lanes with a blinker tap (3 blink mode), then we need to disable LDW for enough time to allow a smooth manual lane change. 4 seconds as it seems the freq is actually around 50Hz, not 100Hz. * Disable HSO for 3 seconds after blinker has been active (#111) Avoid HSO will deactivate and try to take the car back into the lane during a manual lane change with blinker tapped (3 blinks mode) * params learner to only learn @ cons speed Acceleration/deceleration in turns affect yaw as measured by phone and will create potential issues with learner. This code changes the learner to only adjust parameters when speed is constant * Update to match new version of python in eons * python 3 fixes for test client / server * async/await server * Don't log for action message as it's too often, and we throttle it internally. * fix merge issue in thermald * fixes and tweaks; code simplification * Saving P,I,F parameters from pid controller (#116) Added separate save/load functions, try catch - and saving the right constants to the params file. * Feature/tinklad throttling + linting (#118) * Throttle tinlkad's attemptToSendPendingMessages * Remove get_git_remote from registration. It's defined in version already. * tinklad linting * tesla linting * Prevent multiple readings of the pid params file (#119) ..only load the pid values when we initiate the LoC * missing V_PID_FILE constant (#121) * show cars in IC for non-radar users (#120) * bug fixes * Fix issue where user info was being dropped on no connectivity (#122) * move pisParams, fix GPS logging * Bugfix tinklad throttling (#124) * ALCA4 (still in dev) * ALCA4 tuning * ALCA4 alpha release * ALCA4 tweak for turns * ACC fix; LDW mite during ALCA * ALCA polishing * fix enter_canloader.py for python 3 * should_ldw changes moved the should_ldw code closer together and added status messages for when LDW is active and inactive * thermald and ALCA * ALCA sinplified logic
|
Hello,i make the correction table CRC DataIDList VW MQB. |
|
@MikhailTik55 I don't quite understand what you're asking for. To the extent I've solved the CRC pad byte for a given message, you can already find the solution in this PR. Each message requires a different solution; I have not found a general approach. I have not bothered to solve the pad byte for all messages, only the ones I needed. I have some code that helps me with the per-message brute force process, but I'm not planning to publish it, because it's a write-only disaster full of cruft from when I was trying to find the overall algorithm, and would not be usable for anyone else. If you have an openpilot related need to solve a certain message, I'm willing to do one or two on your behalf. If you need it for something else, or want a solution for every message, there's sufficient information in this PR to trivially write your own brute force tester. Further questions would best be asked in #volkswagen on Comma's Discord. |
|
Hello! Thank you for your interest to my message. |
…agen MQB (commaai#836) * Generalized core OP CRC support plus extra bits for Volkswagen MQB.
Fourth of five PRs to upstream the Volkswagen, Audi, SEAT, and Škoda community port. This one reduces the size and scope of the final OP PR by giving you a look at the core CRC stuff separated from the traditional vehicle port.
This code teaches OP generally about the signal name "CRC", like it knows "CHECKSUM" today. It also teaches OP about the specific algorithm used by Volkswagen MQB, which is CRC-8/AUTOSAR with a twist. It enables OP to do CRC and counter validation on all Volkswagen MQB received messages it relies on, similar to Honda and Toyota, and to set the CRC on transmitted messages.
I wrote the overarching CRC signal stuff with extension to other vehicle ports in mind. It could see use for Pedal, Hyundai, Mazda, and Nissan at a minimum. The wrapper function init_crc_lookup_tables() exists for the purpose of adding other CRC lookup tables as needed. In the beginning I had them embedded as static lookup tables, but switched to computing them at startup on informal feedback from @rbiasini.
Note: this code requires the CRC signal be named exactly "CRC", just as CHECKSUM and COUNTER work today. OP would need some nontrivial refactoring to support magic signal names that vary by message. As such, our PR commaai/opendbc#191 changes those signal names to CRC and COUNTER respectively for select Volkswagen MQB messages we rely on. I do not like this and would much prefer to keep the signal names canonical, and am willing to talk more about it if you see a reasonable path to fix it.