Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Micro bounty: visiond PC version #544

wants to merge 36 commits into
base: devel


None yet
9 participants
Copy link

nanamiwang commented Mar 2, 2019

A visiond version for linux, tested on ubuntu 16.04.


  • Connect to Eon streaming server (loggerd --stream), decode the hevc YUV420 frames using libavcodec API.
  • Emulate a rear camera, dispatch YUV420 frames to the processing thread.

  • A clone of
  • Launch the emulated camera, run driving model and posnet model, so that all the rear camera vision pipeline can work as normal.
  • Removed dependencies on android ion, visionbuf and visionimg.
  • Removed streaming server support.
  • Removed front camera and drive monitoring features.

Other changes

  • Add some missing header inclusions in, and
  • Use CPU mode for x86_64 in
  • Fixed a memory leak in
    Have tested the code on Eon, so that all the changes will not affect the android version of visiond.

Build Prerequisites

  • Install SNPE and set SNPE_ROOT env.
  • Build and install zmq & czmq.
  • Install libavcodec-dev.

How to use

On Eon
Launch loggerd and visiond

loggerd --stream

On ubuntu
Launch visiond, set the target Eon IP address using env EON_IP:

EON_IP= ./visiond

Launch calibration daemon to handle posnet inference result and do live calibration:


Launch openpilot-tools replay UI to see the decode frames and model inference result:

python openpilot-tools/replay/.py

How to verify the model inference result

espes and others added some commits Jul 29, 2017

Merge pull request #121 from commaai/devel
openpilot v0.3.5 release
Merge pull request #126 from commaai/devel
openpilot v0.3.6.1
Merge pull request #146 from commaai/devel
openpilot v0.3.8.2
Merge pull request #165 from commaai/devel
openpilot v0.4.0.1
Merge pull request #176 from commaai/devel
openpilot v0.4.0.2
Merge pull request #184 from commaai/devel
openpilot v0.4.1 hotfix
Taiwanese Prius Prime fingerprint was identical to the regular prime (107 msgs.), but should really have 110.  I have the correct one in there now.
openpilot (#220)
* Honda Pilot 2017 Port (#161)

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update Pilot Fingerprint

* Update

* Give pilot its own definition and not use ILX

* add pilot argument

* Add Pilot interface

* Add pilot argument

* Update

* Parse Different gear on pilot

* Add steer max

* Fixed duplication of steer max value

* Adjust PID's for steering

* Update

* Change Steer Ratio and wheelbase

* Update Steer fault values

Steer fault value of 3, does not seem to effect anything

* Update Kp,Ki Ratio

* Update

* Update readme for Pilot

* add pilot

* Update

* Update

* add signals

* add signal

* fix restricting video upload to wifi

* Dibs on SAFETY_GM numerical value

To match Panda repo.

* Safety Reference for Honda Bosch

* Update (#210)

Consolidated my fingerprint and removed duplicates and ordered the fingerprint for the RAV4H. Double Checked.

* Interpolate ki/kp for steering PID loop (#200)

* Interpolate ki/kp for steering PID loop

Very much needed for the Volt port: car ping-pongs with low kp
on high speeeds, and the loop is unstable with high kp on
low speeds.

Also, removes "number or array?" logic from PIController,
now that all the callers use interpolation ofr ki/kp.

* Pass speed to steering PID loop for ki/kp interpolation

* Remove unused numbers import

* Slight changes to UI and Fingerprint for Odyssey Elite (#196)

* Adding back drive time to UI

* Add fingerprint for Odyssey Elite

* Removed extended fingerprint for Elite

* Revert "Adding back drive time to UI"

This reverts commit b9b02f7.

* Squashed 'panda/' changes from 98f29a4..67d5208

67d5208 fix signedness issue in toyota safety
fe15d3f bump pandacan
11c2b08 add fault invalid
2c26e45 add sleep
27c7637 forgot the counter
3a6d7db don't hang
bfa7d2e canloader works
b259e2a can flasher is close to working
83f2edf isotp can support in softloader
7ae7c79 typo
e85cc47 forgot the selfs
190b4f6 start work on canflasher
5c655c9 add recover support
ae3457f usbflash is reliable
f7a0ab0 pedal usbflash works
585d0f9 add way to call isotp
be82899 despite it being bad code, move isotp
000715b start work on pedal canloader
626e312 pedal has a bootstub now
3662d1e redundant check
81e6b0d fix bug
083cd12 should have bounty to refactor that ish
b65d30c bad asserts
b2e6c3f isotp untested support for subaddr
30fd66a Merge pull request #93 from vntarasov/volt
06f5109 Merge pull request #94 from gregjhogan/can-printer-hex
c7d098c Merge pull request #95 from gregjhogan/setup-script
22fe250 Merge pull request #99 from gregjhogan/bit-transition-example
ba16ba3 Merge pull request #100 from gregjhogan/j2534-troubleshooting-instructions
ad08ea4 Merge pull request #90 from gregjhogan/can-forwarding
f3b6f5d added j2534 troubleshooting instructions
858d150 added script to find bits that transition from 0 to 1
c6acac8 added checking pedal interceptor message length
f7226ff added brake safety checks
d0c2634 added gas safety checks
d378e4a removed bosch safety forwarding restriction on 29 bit addresses
5c7ef9e added bosch safety hooks and forwarding
90c64b6 add note
23de8d4 Merge pull request #97 from commaai/pedal_improvements
0261641 added missing python packages
b92b235 fix bytearray encode issue
2434f1c Tweak Volt's brake pedal stickiness
e2f73d2 enable has a whole byte to itself
d5a9e1e correct checksum
f8ed9fa better names
986a14c don't alias pointers
9b8472e add watchdog support
8f0add9 handle faults
1d917f8 split gas set into 2 values, and have a fault state
1b77026 j2534 isn't alpha anymore
fbcc872 Merge pull request #92 from commaai/pedal
8a6f44b pedal is sending messages
08f464c python 3 bro is bad bro
9390961 kline checksum algo was broken...
3b7c33b add kline debug support
aa622bc init values
631ea9f better refactor
eb1fd75 add PEDAL adc sets
ccaa310 don't build with usb
8d4d763 debug console works
bd09883 comma pedal is building
75a29d5 Merge pull request #84 from gregjhogan/j2534-hds
eece37d only the panda has gmlan
9f43abe Merge pull request #89 from vntarasov/volt
5364d43 Merge pull request #88 from vntarasov/smaller-firmware
377a1ec bump version for descriptor fix
4fabdf0 Merge pull request #87 from gregjhogan/usb-multi-packet-control
8580773 fix sending WinUSB Extended Properties Feature Descriptor
6908feb Chevy Volt safety
786a004 Enable optimization to reduce firmware size
d70f43b hack to fix thinkpad
95ab1ae fixed flow control message padding
bbd04d1 updated installer
62216d0 single standalone DLL for J2534 driver
5c9138d fixed 11 bit address issue
f3b0ad2 fix LOOPBACK getting set when DATA_RATE is set
b750d36 updated README
a9a097f lowered CPU utilization
7c26a70 TIS needs unsupported protocols to return an error
42692b4 TIS doesn't like ChannelID being zero
cf126bb SET_CONFIG return error for reserved parameters
2e99dbf fix HDS issues
8203cc8 add is_grey
e946a54 add insecure_okay flag
4363b3e check webpage
4f59ded add secure mode note
6b11fb5 add autosecuring to tests
b27d185 Merge pull request #86 from commaai/better_pairing
4b53b42 elm wifi isn't an automated test
99f85cb Merge pull request #85 from gregjhogan/usb-wcid
0d38060 auto-install WinUSB device driver
c6653ca from python import
38cc0ee add wifi_secure_mode, boots in insecure mode

git-subtree-dir: panda
git-subtree-split: 67d52089a1300b86800d897f2b271e0a24cf6dd6

* Squashed 'opendbc/' changes from 81d9871..aa067f7

aa067f7 Chevy Volt tweaks (#83)
a60c6c4 Toyota: change signal name in EPS_STATUS msg
ce70b1a extra setme field toyota LKAS_HUD
df2a552 toyota missing ACC_CONTROL checksum
48bb293 Revert "Toyota Prius: added a comma specific message to control the speed sent to the EPS"
5f42439 Toyota Prius: added a comma specific message to control the speed sent to the EPS
6f5e8b6 Pedal Interceptor: fault state VAL moved to _comma
efd5f5c add setme to honda ACC_HUD
97fc335 add interceptor to civic
6f40f16 update generator script to allow for multiple imports
9ca956b add setme to toyota STEERING_IPAS
e5afa57 run generator for ipas scaling
8bd1182 Toyota IPAS: proper steer angle unit
f57511e acceleration pedal for gasPressed
c8d1dbc high beams also. likely dashboard message.
9f1c78b high beams for genericToggle
f037d42 turn signal lights (and thus hazard lights)
b35bb08 turn signals
78986cf Revert "turn signals"
ba946c9 turn signals
2af3ecc Speed, braking, and distance signals
f40ab87 Set packet lengths, adding steering rate, adjusted speed
cd59bfa units for speed_right
c2fcce2 speed of right vs left side of car
4ef5fae value table for gear status
97c48e2 tighten up speed bits. brake pressue max comment.
a0cbfd1 add gear status PRNDL
0c82865 initial signals for chrysler pacifica 2017 hybrid
5ed0540 add set me to toyota LKAS_HUD
aecac5d add set me fields to toyota ACC_HUD
5417013 update toyota ACC_CONTROL fields
e91e967 Comma Pedal: made GAS_COMMAND 6 bytes
d04434a Comma Pedal: added state byte and enable bit
c30b2cd Comma Pedal: sending 2 tracks on 0x200
8f72467 Volt doors and belts status (#70)
60f8b6c add set me to lkas hud honda
3c9e335 fix honda pcm gas message size
7ca471d Add 2018 Toyota CHR dbc (#78)
637fe00 set scaling to 1 for brake and gas which have no real unit
62a88d4 Volt: switch to parsing ACC buttons from powertrain CAN (#74)
3fdd47b Volt's gas pedal only and combined gas/acc (#76)
45ec9c9 Add 2017 Honda Ridgeline (#77)
cbd186a Add 2018 Camry Hybrid DBC's (#73)
974eeaf Toyota: re-generated the files after cfbc9ae363f98ef
19ea195 Toyota: more vals for LKA_STATE
cfbc9ae fixed inconsistent factor for speed in Honda dbc files
e7db803 convert all line endings to unix style

git-subtree-dir: opendbc
git-subtree-split: aa067f7079aa12617f7a37d85233e51af44e1bb2

* openpilot v0.4.3 release

* Squashed 'panda/' changes from 67d5208..3125232

3125232 bump version
703c0b4 Gas Interceptor: another fix to gas pressed logic
196d383 Interceptor: fixed gas pressed logic

git-subtree-dir: panda
git-subtree-split: 31252324d98e701c33cb6aeda20af6b549175764

* Squashed 'opendbc/' changes from aa067f7..91e882d

91e882d Updating bosch dbcs to use new format and bringing in new honda changes (#82)
9b32e2e Fix Checksum errors for CH-R (#86)

git-subtree-dir: opendbc
git-subtree-split: 91e882d4a04c129e12d39bcff0bbe56b75166e0f

* openpilot v0.4.3 release

* openpilot v0.4.3.1 release

* fix bug in canpacker for Toyotas with DSU connected (#221)

* update year on civic
Revert "openpilot (#220)"
This reverts commit 640ab12.
Merge pull request #226 from commaai/revert-220-devel
Revert "openpilot"
Merge pull request #239 from commaai/devel
openpilot release
Vehicle Researcher
openpilot v0.5.1 release (#315)
Merge commit '2cfdc676101c387524246f789c8429647069b827' into release2
Revert the changes to GM in 0.5.4 (#380) (#386)
* Revert 0.5.4 changes for GM for 18ers

* Redo the refactor of stock control msgs


* Remove spacing

* Need candidate array idx
Merge pull request #410 from commaai/devel
Improve Toyota radar filtering (#409)
Merge pull request #434 from commaai/devel
calibration improvements and devel synch up

rbiasini and others added some commits Dec 18, 2018


This comment has been minimized.

Copy link

icmma commented Mar 2, 2019

@nanamiwang you have done it again bro hats off.


This comment has been minimized.

Copy link

arne182 commented Mar 3, 2019

Am busy loading up 16.04 to test this. Great job.


This comment has been minimized.

Copy link

rbiasini commented Mar 11, 2019

@nanamiwang , this looks great!
Quick questions:

We still have to test it. Tnx!


This comment has been minimized.

Copy link

nanamiwang commented Mar 12, 2019


  • I was thinking a bit of performance gain if doing stream reception and hevc decoding in visiond directly instead of going through PyAv and zmq queue. I can update the PR, remove and remove frame publishing stuff in

  • Do you mean moving to openpilot-tools repo? shares model inference and OpenCL code with, seems not be easy to be moved out.
    BTW, A suggestion on code structure, currently the posenet code is duplicated in the two visiond cc files, should be refactored out to a dedicate file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.