Collect and evaluate affordable camera and processing options and requirments

[mslavescu]

Post here ideas with affordable cameras, image sensors, lenses, image processing boards, to build an open source large scale deployable smart camera.

We need to test a lot of options, eBay, Aliexpress, DigiKey etc links would be very helpful.

Stereo setups will be probably required for front and back car views.

Fast frame per second for accurate and reliable speed and distance calculation.
We will add more requirments later.

[ingenieroariel]

Here are my thoughts:

1. Use a jetson TX2 as the brain. ($300).
2. Find out how we can get a gopro knockoff that has a 4 channel MIPI interface. (2.5Gb/s per lane, 10GBs total). My personal desire is to put at least two cameras, one that does VGA at 300fps and another one that does 4Mp at 12fps. I call them the HOF (vga) and HOG (4Mp) cameras. ($350 from e-consystems: e-CAM40_CUTK1 is a 4.0 Mega Pixel, 4-lane MIPI CSI-2 Camera solution for NVIDIA Tegra K1 CPU. This is based on the OV4682 CMOS Image Sensor, a popular RGB-IR pixel format CMOS image sensor from OmniVision. The e-CAM40_CUTK1 has 4MP camera module and an adapter board to connect the camera with the Jetson TK1 development kit.).

[bigsnarfdude]

These look interesting http://antmicro.com/products/tx1-tx2-kit/
This thread has list of compatible cameras http://elinux.org/Jetson/Cameras#Gigabit_network_cameras_known_to_be_working

[mslavescu]

Thanks @ingenieroariel and @bigsnarfdude!

Here is another a list of low cost options:

http://www.learnopencv.com/embedded-computer-vision-which-device-should-you-choose/

@ingenieroariel how would you connect cameras located in different places in the car when using TX1/TX2?

I assume we need multiple TX2 boards:

1-2 for 2-3 cameras front
1 for 2-3 cameras back
1 for 2 cameras left side
1 for 2 cameras right side

Total: $5500

5 TX2 * $300 ~ $1500 +
10 cameras * $350 ~ $3500 +
10 lenses and mounts * $50 (should be less) ~ $500

This would be a high priced setup.

If we could build a similar one with RPi3/Odroid or Custom FPGA + RPi 8Mpx camera at a 1/4th of that price would be super for large scale of Open Source SDC deployment/testing.
The single camera+board option will probably give us better modularity, connectivity options and more redundancy.

I'll start to capture these requirements in the wiki here.

[mslavescu]

We should also look into zoom capable cameras, like in this Mercedes SDC demo:
https://twitter.com/GTARobotics/status/851953288052396032

[ingenieroariel]

I'd say:

• 4 Jetson TX2 (each of those can host 3 cameras with 4 lanes or 12 cameras with one lane): $1200.
• 3 cameras for the front, 2 for each side, 3 for the back: 10 cameras: We should find a way to get them under $200 each with lenses and mounts: $2000.

So the budget would be around $3500 - $4000 when you factor other small items. This is the same price users pay when changing between trims on a car (to get brake by wire and gas by wire I had to pay this on my Ford Fusion). It's not unreasonable in my opinion and could come way down in the future potentially with a Qualcomm Snapdragon 835? running OpenCL/TensorRT. CommaAI is already trying to do the dirt cheap ($1000), Dataspeed does the crazy expensive ($100000) there is a spot to be filled by smart cameras that can work with that system, ROS or anything else, cheap can mean between 3K-5K.

If you end up choosing the Jetson I will be happy to collaborate as I already purchased a few along with cameras. If you go the custom FPGA route, or smaller boards I really recommend looking into RISC-V designs, they are very fast and you could tweak the design to add more UARTs (can bus) and other interfaces.

[mslavescu]

Thanks @ingenieroariel!
Would love to try the TX2 with some of the cameras you mention above. Your Autti.co startup would set a nice precedent :-)

For large scale, thousands of setups around the world, I still think we should look into lower cost solutions too.

Here are a few more off the shelf boards:
http://www.tongfamily.com/2016/03/raspberry-pi-3-and-alternatives/

Regarding Comma.ai, can you change the lens or is just the way the phone comes?

I also think the smart camera should have only the minimum hardware to accelerate part of the image processing pipeline and provide fast communication lanes (uncompressed image if possible) to a central processing unit, which initially would be a single or dual GPU (Nvidia 1070 level).

We need to create benchmarks defined for each step in the computer vision pipeline and then see which one makes sense to offload to the end points (smart camera, smart sensors etc), and we may also find that we need different algorithms for different camera views points.

[ingenieroariel]

I doubt you can change the lenses on the oneplus. Might be possible though to create and addon that makes it fish eye.

…

-a

On Apr 11, 2017, at 7:54 PM, Marius Slavescu ***@***.***> wrote: Thanks @ingenieroariel! Would love to try the TX2 with some of the cameras you mention above. Your Autti.co startup would set a nice precedent :-) For large scale, thousands of setups around the world, I still think we should look into lower cost solutions too. Here are a few more off the shelf boards: http://www.tongfamily.com/2016/03/raspberry-pi-3-and-alternatives/ Regarding Comma.ai, can you change the lens or is just the way the phone comes? I also think the smart camera should have only the minimum hardware to accelerate part of the image processing pipeline and provide fast communication lanes (uncompressed image if possible) to a central processing unit, which initially would be a single or dual GPU (Nvidia 1070 level). We need to create benchmarks defined for each step in the computer vision pipeline and then see which one makes sense to offload to the end points (smart camera, smart sensors etc), and we may also find that we need different algorithms for different camera views points. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[mslavescu]

One option for smart camera should be stereo camera, this model may be a good design choice FPGA stereo processing:
https://nerian.com/products/sp1-stereo-vision/

More options here:
http://rosindustrial.org/news/2016/1/13/3d-camera-survey

This one is nice also:
http://files.carnegierobotics.com/products/MultiSense_S7/MultiSense_Stereo_brochure.pdf

If you used stereo cameras for self driving car scenarios (in full size cars), please post the models here with pros and cons and, the price of these cameras.

More info here:
http://robotics.stackexchange.com/questions/6508/low-power-computer-for-stereo-vision

[mslavescu]

Nice article on stereo camera with FPGA and ARM CPU, covers well stereo:

Real-time and Low Latency Embedded Computer Vision Hardware
Based on a Combination of FPGA and Mobile CPU

https://www.inf.ethz.ch/personal/pomarc/pubs/HoneggerIROS14.pdf

[mslavescu]

Very cool technology, we could use it for free space calculation (same authors as previous paper):

Voxblox: Building 3D Signed Distance Fields for Planning
https://m.youtube.com/watch?v=PlqT5zNsvwM

[mslavescu]

More smart camera FPGA related papers here:
https://twitter.com/GTARobotics/status/852034046464937985

[bigsnarfdude]

Found this on multi-camera and CPU http://antmicro.com/blog/2016/05/epson-s2d13p04-and-toradex-nvidia-tegra-t30/

[mslavescu]

Thanks @bigsnarfdude! These guys are amazing!
Do you know anyone there?
Maybe they will design a low cost open source module​ for us.

Here are some nice modules (some used by them):

https://www.toradex.com/computer-on-modules/apalis-arm-family/nvidia-tegra-k1

This a nice one, has CAN bus interface also:

https://www.toradex.com/products/carrier-board/apalis-evaluation-board

[mslavescu]

This setup would be cool to test, see paper:

Reactive Avoidance Using Embedded Stereo Vision for MAV Flight

http://helenol.github.io/publications/icra_2015_reactive_avoidance.pdf
...
We use a Samsung Exynos 4412 System on Chip Module
with a built in Cortex-A9 Quad Core mobile CPU. The FPGA
system is based on a Xilinx Artix7 XC4A100T module and
is connected to the dedicated camera interface of the mobile
CPU. The two cameras are equipped with MT9V034 CMOS
image sensors from Aptina with global shutter. Images with
640x480 pixels resolution and corresponding disparity map
are provided to the mobile CPU with 60 frames per second
update rate.
...

[mslavescu]

MIPI Alliance expands popular CSI-2 camera spec
http://www.ept.ca/2017/04/mipi-alliance-expands-popular-csi-2-camera-spec/?custnum=2089006

[mslavescu]

This looks very nice, I'm not sure what cameras and board they use:

VI-Sensor at ICRA 2014
https://m.youtube.com/watch?v=jcjB_Pflu5A

#A Synchronized Visual-Inertial Sensor System with FPGA Pre-Processing for Accurate Real-Time SLAM" at ICRA 2014

[mslavescu]

We need to look into very flexible and affordable stereo cameras, that can be modified (zoom, focus, direction, relative position) and recalibrated on the fly (at the end with FPGA).

Basically calculate a "disparity index" and maximize its consistency, while using cues from each monocular camera image.

We should also integrate in this automatic reconfiguration and recalibration, feedback from cheap LIDAR, ultrasound and radar, also using known (3D position and model) static landmarks along the road.

A few relevant article/videos to spark the discussions:

Sensors: Powerful, Cheaper and Multipurpose
http://analysis.tu-auto.com/autonomous-car/sensors-powerful-cheaper-and-multipurpose

Automatic Camera Re-Calibration for Robust Stereo Vision
https://m.youtube.com/watch?v=2QGnOwfQKYo

Real-time unknown-baseline fully-dense image correspondence
https://m.youtube.com/watch?v=BYUMgrDvaRc

Hardware Accelerated Stereo Vision
https://m.youtube.com/watch?v=tKJJ_IrZ834

Although not funded, it describes very nice what FPGAs are and how they can be used in robotics/computer vision:
https://www.kickstarter.com/projects/989837359/logitraxx-xilinx-fpga-powered-tracked-robot

More videos with stereo cameras/processing:

Making Cameras Smarter: FPGA Based Image Pre-processing Unleashed
https://m.youtube.com/watch?v=zVqbIxre-cA

Real-time Stereo Matching on GPU with Hierarchical AD-Census
https://m.youtube.com/watch?v=Egb9EgANo24
Published on Oct 23, 2015 Hierarchical AD-Census Stereo Matching Algorithm, 30fps for VGA input @ NVIDIA GTX580
Code: http://www.cs.columbia.edu/~fyun/SGM.zip

FPGA Version: https://nicsefc.ee.tsinghua.edu.cn/media/publications/2015/IEEE%20TCSVT_151.pdf

"Walker" HD - 3D reconstruction using VisLab stereovision
https://m.youtube.com/watch?v=N1ro2YE5rY8

AIT 3D Stereo Vision
https://m.youtube.com/watch?v=XgHB85Ccky8

Urban 3D Semantic Modelling Using Stereo Vision, ICRA 2013
https://m.youtube.com/watch?v=6aYmx4_LUms

Real-Time Stereo Vision For ADAS : Stixel 160311
https://m.youtube.com/watch?v=0KUAfZqiT-w

FPGA stereo camera: off-road video I and II
https://m.youtube.com/watch?v=9KE2Eq0CP54
https://m.youtube.com/watch?v=7c4WjOcw5Wc

[mslavescu]

A very nice video that emphasis the idea of using cheap LIDAR for automatic calibration and improving disparity computation:

Integrating LIDAR into Stereo for Fast and Improved Disparity Computation, Hernan Badino, Daniel Huber and Takeo Kanade, 3DIMPVT, 2011.
https://www.ri.cmu.edu/pub_files/2011/5/3DIMPVT2011.pdf

Dense Stereo Reconstruction of a Road Environment (YouTube video)

[mslavescu]

This event is very interesting, exactly what we need for smart camera with OpenCV for ADAS features acceleration:

UK event on May 18 to discuss Machine Vision with FPGA & SoC Platforms

https://forums.xilinx.com/t5/Xcell-Daily-Blog/UK-event-on-May-18-to-discuss-Machine-Vision-with-FPGA-amp-SoC/ba-p/760271

NMI, a non-profit organization dedicated to improving electronic engineering and manufacturing in the UK, is organizing a one-day, machine-vision event for May 18 titled “Implementing Machine Vision with FPGA & SoC Platforms.” MBDA Missile Systems is hosting the event in its Stevenage location in the UK. (That’s roughly midway between London and Cambridge for those of us who are geographically challenged.)

 Key themes for the event will include: OpenCV with FPGAs and SoCs, ADAS, Robotic Guided Vision/Drones, Industry 4.0, Defense, and Machine Learning.

[mslavescu]

FPGA ADAS solutions from Altera:
https://www.altera.com/solutions/industry/automotive/applications/drive-assistance/driver-assist.html

Forward camera:
https://www.altera.com/solutions/industry/automotive/applications/drive-assistance/forward-camera.html

[mslavescu]

Recommend affordable boards that could be used to achieve (some of) these:

Responsive and Reconfigurable Vision Systems
https://www.xilinx.com/products/design-tools/embedded-vision-zone.html#design

Design Examples for Machine Learning and Computer Vision

The reVISION Stack includes four initial design examples (with more to come) that are intended to get you up-and-running in a very short period of time. These design examples will help you easily see the distinct advantage Xilinx All Programmable SoCs have in high performance Embedded Vision applications. The following is a brief description of these four design examples.

LK Dense Optical Flow @ 4K60 - Real-time dense implementation of optical flow, detecting object motion for every single pixel. This example uses non-iterative, non-pyramidal implementation on 4K60 FPS input coming from a Sony IMX274 sensor via the MIPI interface
Stereo Vision - Real-time stereo disparity map calculation including remap, rectification and local block matching. It can process dual 1080p30 stereo camera input via USB3
Deep Learning: GoogleNet - GoogleNet benchmark with INT8 demonstrated using standard ImageNet inputs.
Autonomous Perception Design - Combination of dense optical flow, stereo vision and deep learning example designs (the three described above). Using combination of MIPI sensors and USB3 cameras, the design represents a real-life use case of autonomous vision system by combining dense optical flow, stereo vision and CNN network into a single design example

These design examples will be available in May 2017. Get started today designing your computer vision system around Zynq SoCs/MPSoCs and FPGAs by leveraging existing Xilinx and ecosystem design hardware, modules, and production-ready Systems on Module (SOMs).

[mslavescu]

Very good presentation from 2016:
Open-Source tools for FPGA development

[mslavescu]

Links to Jetson TX2 Resources

[mslavescu]

This kit is very interesting, it covers a lot of the ADAS features.
I would like to have some of the functions at camera end, but we can learn from this design what cameras and boards are used currently for ADAS:

http://www.logicbricks.com/Solutions/Xylon-ADAS-Development-Kit.aspx

Check the data sheet PDF for more details, and post here alternatives that we could use for an affordable OSSDC Smart Camera:

http://www.logicbricks.com/Documentation/Datasheets/HW/logiADAK_hds.pdf

This kit is pretty expensive: $14K

http://www.logicbricks.com/Products/logiADAK.aspx

The hardware platform includes:

• 1x Xilinx Zynq-7000 SoC ZC706 Evaluation Kit (OEM Version)
• 1x Xylon LVDS FMC add-on board for up to 6 camera connections
• 5x Xylon waterproof aluminum camera housings
• 5x Xylon LVDS serializer boards for camera sensors
• 5x Omnivision OV10635 1Mpix camera sensors
• 4x Sunex DSL219 fisheye and 1x DSL947 miniature lenses
• 1x power supply and cabling for vehicle installations

The board alone costs $4K:

https://www.digikey.ca/product-detail/en/xilinx-inc/EK-Z7-ZC706-G/122-1904-ND/4147282

The camera sensor is $23:
https://www.digikey.ca/product-detail/en/OmniVision-Technologies-Inc/OV10635-N29Y-PB/OV10635-N29Y-PB-ND/4377186

If we use better image sensors with cheaper lower end FPGAs, to do just part of processing, we should reduce the cost a lot (I hope) and we could improve the algorithms easier, because of the scale of the deployment, the more of us can try these in road scenarios the better.

[mslavescu]

Not sure how much they are, but may be a good option to transport high res/speed videos with low latency:

http://www.vision-systems.com/articles/print/volume-19/issue-1/features/coaxpress-cameras-and-frame-grabbers-tackle-high-speed-imaging.html

As an asymmetric point-to-point serial image transmission standard, the CXP standard is scalable. Using a single coaxial cable, downlink speeds of up to 6.25Gbps per cable can be achieved. With a 20.833Mbps uplink for communications and control, the standard also allows 13W of power to be supplied over each cable in Power-over-CXP mode.

[mslavescu]

It becomes easier to build solutions with FPGAs:

https://www.digikey.ca/en/articles/techzone/2017/jan/build-better-video-bridging-solutions

Dual camera
[]

[mslavescu]

Affordable FPGAs dev kits:

https://www.quora.com/What-is-the-best-affordable-FPGA-dev-kit-for-a-starter

[mslavescu]

Found this a while ago, has lots of info with stereo cameras for ADAS:

Stereo vision - Facing the challenges and seeing the opportunities for ADAS applications�

http://www.ti.com/general/docs/lit/getliterature.tsp?baseLiteratureNumber=spry300

[mslavescu]

Thanks @bigsnarfdude for this list!

https://twitter.com/BigsnarfDude/status/853142311026409472

https://joelw.id.au/FPGA/CheapFPGADevelopmentBoards

Lost of options here, we just need to find a few that will be best for offloading part of computer vision pipeline into the smart camera.

A board that has MIPI interface for cameras and we could also connect or build easily a coax transceiver to transport high res/fps uncompressed video to the central unit at 5-15m (maybe even farther for trailers and boats later for example).

[bigsnarfdude]

@mslavescu following up on a project that can be a baseline for experiments on FPGA. Here is code for a CNN on FPGA https://github.com/dgschwend/zynqnet

[mslavescu]

Thanks @bigsnarfdude!
That is a very interesting project, I'm wondering how much we can scale down that approach on a <$100 FPGA chip/board, maybe even handling two images at once, from a stereo camera. We really need some magic happening, but I feel is possible.

The FPGA he used is $3400, just for the chip:

https://www.digikey.ca/product-detail/en/xilinx-inc/XC7Z045-2FFG900I/122-1897-ND/3925782

[mslavescu]

This sensors seems to be pretty good OV04689-H67A:
http://www.ovt.com/sensors/OV4689
https://octopart.com/ov04689-h67a-omnivision+technologies-46507254

Spec OV04689-H67A
Package CSP
Technology OmniBSI-2™
HDR
Interface MIPI
Shutter Type Rolling Shutter
Resolution 4MP
CFA (Chroma) RGB Bayer
Analog / Digital Digital
Power Requirement Active: 163 mA (261 mW)
Standby: 1 mA
XSHUTDOWN: <10 µA
Output Format RAW
Operating Temperature -30°C to +85°C
Optical Format 1/3"
Frame Rate Full @ 90 fps
Pixel Size 2.0 µm
Image Area 5440 x 3072 µm
Package Dimensions 6630 x 5830 µm
Product Guide PDF

[mslavescu]

This may be good to hack on and connect it to an FPGA if possible.

It is an outdoor security camera for $38-$46 that uses the Sony IMX322 image sensor + Nextchip 2441 processor, has infrared and zoom also which is pretty nice, I think is similar with the ones used by http://Autox.ai:

https://m.aliexpress.com/item/32413015582.html

http://m.ebay.com/itm/231679288350?_mwBanner=1

Features:
1/3" CMOS Sony sensor, horizontal resolution 1200TVL
72PCS IR LEDs, infrared distance up to 262 feets
IR-CUT automatic switching function, good day and night vision effect
2.8-12mm manual vari-focal lens for different view angle
Can be easily mounted on the wall or ceiling, with the adjustable bracket
IP66 waterproof for both outdoor and indoor use
Metal body material, durable and stable performance
Easy installation, plug and play
Can be connected to DVR, TV, computer and so on
AGC, BLC, Auto/manual white balance, No distortion, No shadow

If we could connect an FPGA (to the internal board) for advanced processing would be really cool.

The videos look very good, including night time:
https://m.youtube.com/results?q=TOVERN%20T755-VHN%201080P&sm=3

This seems to be the board and image sensor $23, although for double, we get also case and zoom lens:
https://m.aliexpress.com/s/item/32587189101.html?trace=storeDetail2msiteDetail

[mslavescu]

This is a really nice camera, does anyone know how much it costs?

http://www.e2v.com/products/imaging/cmos-image-sensors/lince5m/

Lince5M incorporates a high speed 5.2M CMOS active pixel image sensor providing global electronic shutter and High Dynamic Range (HDR) features.
5.2 Megapixels, 1-inch, 250fps,
global-shutter CMOS image sensor

 2,560 x 2,048 active pixel

• 5μm pixel-pitch with pinned photodiode
• Optical Format: 1”
• Global-shutter with programmable exposition time
• Maximum frame rate: 250fps @ 2,560 x 2,048 in 12-bit
mode
• 60dB dynamic range (DR)
• 42dB SNRMAX
• 0.5% PRNU
• 0.1% FS DSNU
• 74% Fill Factor x Quantum Efficiency1
• Sensitivity2
6.0 V/(lux-sec)
102,800 DN12-bit /(uJ/cm2
)
• Sensitivity*3
11.0 V/(lux-sec)
95,500 DN12-bit/( uJ/cm2
)
• 16.6 Gbit/sec throughput
• High-speed data output via LVDS ports (x24) or CMOS
buffers (x48)
Image sensor block diagram
Miscellaneous
Key features
*1 (λ=550mn, μ-lens)
*2 (λ=550mn, Monochromatic, gain x1, μ-lens)
*3 (3200K + CM500S IRCF, gain x1, μ-lens

[mslavescu]

Excellent presentation!

Passive stereo vision with deep learning
https://www.slideshare.net/mobile/yuhuang/passive-stereo-vision-with-deep-learning

Must try for OSSDC Stereo Smart Camera!

https://twitter.com/GTARobotics/status/853615370342674433

[mslavescu]

This is interesting, maybe we should look at other camera interfaces then MIPI, any suggestions?

https://www.parallella.org/2015/06/01/the-open-camera-project-1000-bounty-for-open-firmwaredrivers-for-raspberry-pi-camera-module/

Josy Boelen says:
June 2, 2015 at 11:23 am

MIPI is a closed specification and bears a cost of US$ 8,000.- (per year). I don’t think you can incorporate any ‘derived’ works in an open source project.

http://mipi.org/specifications/camera-interface
http://mipi.org/join-mipi

There are other image sensors with non-proprietary/closed-spec interfaces, but I don’t think the Raspberry Camera price …

[mslavescu]

Lots of boards to connect MIPI cameras over USB, may not best for top #3-mvp-smart-camera, but still a good way to test easily lots of cameras:

https://parallella.org/forums/viewtopic.php?f=23&t=2514&sid=f7e01fb1d53931420ed2d2d316a720d8&start=50

http://mrfdsp.com/mipi-csi2/index.html

[]

[mslavescu]

Check out this video https://twitter.com/GTARobotics/status/853645496581337088?s=09

To the experts in this area please advise what boards should we use to approach building an affordable OSSDC stereo #3-mvp-smart-camera.