fuzzy fingerprint: track number of matching ECUs, not FW versions

[sshane]

On Hyundai, for example, it is common for many ECUs to respond to more than one of our queries, and if we have both in the database, it will treat it as multiple ECU matches, when the original intention of the logic was to treat it as one ECU match. This changed when we logged and returned all FW versions instead of just the last per ECU.

Semi pre-requisite for #26939, so match count is accurate (two FW versions for camera and radar will both have the same prefix).

This PR still attempts to use all FW versions available for the brand to both look for valid matches, and multiple invalid matches (disqualifying).

[sshane]

master:

batman@workstation-shane:~/openpilot/selfdrive/debug/internal$ ./fuzz_fw_fingerprint.py 
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [04:02<00:00,  4.13it/s]

AUDI A3 3RD GEN {'VOLKSWAGEN GOLF 7TH GEN'}
HONDA ACCORD 2018 {'HONDA ACCORD HYBRID 2018'}
SEAT LEON 3RD GEN {'VOLKSWAGEN GOLF 7TH GEN'}
SKODA KODIAQ 1ST GEN {'VOLKSWAGEN TIGUAN 2ND GEN'}
SKODA OCTAVIA 3RD GEN {'VOLKSWAGEN GOLF 7TH GEN'}
SKODA SUPERB 3RD GEN {'VOLKSWAGEN PASSAT 8TH GEN'}
VOLKSWAGEN GOLF 7TH GEN {'AUDI A3 3RD GEN', 'SEAT LEON 3RD GEN', 'SKODA OCTAVIA 3RD GEN'}
VOLKSWAGEN PASSAT 8TH GEN {'SKODA SUPERB 3RD GEN'}
VOLKSWAGEN TIGUAN 2ND GEN {'SKODA KODIAQ 1ST GEN'}

Total fuzz cases: 155000
Correct matches:  0
Wrong matches:    541
batman@workstation-shane:~/openpilot/selfdrive/debug/internal$

fuzzy-fp-count-ecu-matches-not-fw-matches:

batman@workstation-shane:~/openpilot/selfdrive/debug/internal$ ./fuzz_fw_fingerprint.py 
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [03:58<00:00,  4.19it/s]

AUDI A3 3RD GEN {'VOLKSWAGEN GOLF 7TH GEN'}
HONDA ACCORD 2018 {'HONDA ACCORD HYBRID 2018'}
SEAT LEON 3RD GEN {'VOLKSWAGEN GOLF 7TH GEN'}
SKODA KODIAQ 1ST GEN {'VOLKSWAGEN TIGUAN 2ND GEN'}
SKODA OCTAVIA 3RD GEN {'VOLKSWAGEN GOLF 7TH GEN'}
SKODA SUPERB 3RD GEN {'VOLKSWAGEN PASSAT 8TH GEN'}
VOLKSWAGEN GOLF 7TH GEN {'AUDI A3 3RD GEN'}
VOLKSWAGEN PASSAT 8TH GEN {'SKODA SUPERB 3RD GEN'}
VOLKSWAGEN TIGUAN 2ND GEN {'SKODA KODIAQ 1ST GEN'}

Total fuzz cases: 155000
Correct matches:  0
Wrong matches:    496
batman@workstation-shane:~/openpilot/selfdrive/debug/internal$

[sshane]

I ran this on real fuzzy fingerprinting routes in the field for the last year and the diff is very minimal, but interesting to note (it's something I'll try to capture in a test).

The data breakdown:

last_n_days=360
Total routes with live fingerprints checked: 34979
Total dongles checked: 677
Brands checked: {'honda': 11417, 'subaru': 2378, 'hyundai': 9881, 'toyota': 5364, 'chrysler': 4336, 'volkswagen': 1113, 'mazda': 317, 'ford': 165, 'nissan': 8}

How I got the matches:

 # original function
  matches = set()
  for brand in VERSIONS.keys():
    fw_versions_dict = build_fw_dict(CP.carFw, filter_brand=brand)
    matches |= match_fw_to_car_fuzzy(fw_versions_dict, log=False)
 
  excluded_matches = set()
  for brand in VERSIONS.keys():
    fw_versions_dict = build_fw_dict(CP.carFw, filter_brand=brand)
    excluded_matches |= match_fw_to_car_fuzzy(fw_versions_dict, log=False, exclude=platform)

  # new function
  # ...

First was to test the number of matches for all fuzzy FP routes didn't change.

There were some differences, but filtering out Hyundai cases where the car's transmission ECU had two FW in the database leaves two routes, where the ABS had two matches instead:

len of matches diff! ('f2f6a4fecb99bb45|2022-10-10--13-51-25--0', 'HYUNDAI SANTA FE 2022', {'HYUNDAI SANTA FE 2022'}, set())
ecus in DB: [2000, 2001, 2001, 2004]
len of matches diff! ('7ff92471720e8674|2022-08-05--17-39-12--0', 'HYUNDAI VELOSTER 2019', {'HYUNDAI VELOSTER 2019'}, set())
ecus in DB: [2001, 2001, 2004]

Checked routes: 35001
Expected loss of fuzzy FP: 696, {'HYUNDAI SONATA HYBRID 2021', 'KIA OPTIMA 4TH GEN FACELIFT', 'HYUNDAI ELANTRA HYBRID 2021'}
Unexpected loss of fuzzy FP: 2, {'HYUNDAI SANTA FE 2022', 'HYUNDAI VELOSTER 2019'}

Next, there was no diff in the number of matches when the platform was excluded, which means no extra false positives:

if len(excluded_matches) != len(excluded_matches_new):
    print('len of excluded matches diff!', (segment, platform, excluded_matches, excluded_matches_new))

Next, there was also no diff in the fuzzy FP matches, when there was a match:

  if len(matches) == 1 and len(matches_new) == 1:
    match = list(matches)[0]
    match_new = list(matches)[0]
    if match != match_new:
      print('mismatch new vs. old func!', (segment, platform, match, match_new))

Finally, there was no diff when checking the matches when excluding the correct platform

(one KODIAQ fingerprints as a TIGUAN with both functions. @jyoung8607 if you're interested and have access: 60f84fabbe664da0|2022-09-06--19-10-59):

if len(excluded_matches) == 1 and len(excluded_matches_new) == 1:
    match = list(excluded_matches)[0]
    match_new = list(excluded_matches_new)[0]
    print('match with platform excluded!', (segment, platform, match, match_new))

(below commits are just tweaking tests)