Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stereo mode #41

Open
wezzoid opened this issue Mar 27, 2018 · 36 comments
Open

stereo mode #41

wezzoid opened this issue Mar 27, 2018 · 36 comments

Comments

@wezzoid
Copy link

@wezzoid wezzoid commented Mar 27, 2018

Hi, how do I use this with the Compute Module to publish nodes for 2 cameras?

@realizator
Copy link

@realizator realizator commented Oct 17, 2018

+1, the same question.
Raspbian supports stereoscopic mode out of the box (https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=85012&sid=ec557617fabfa9942e5baeec16bf862c)

Raspberry Pi Compute module supports 2 cameras, and using a Pi devboard or ready-to-use boards like StereoPi you have native stereoscopic image.
I can help to develper by providing hardware for tests (StereoPi engineering version). @rohbotics, please contact me if you need hardware set for tests and stereoscopic implementation to your driver.

@rohbotics
Copy link
Member

@rohbotics rohbotics commented Oct 17, 2018

We use the same firmware and kernel as raspbian, so that side of stereoscopic processing should work.

Can you use our image (https://downloads.ubiquityrobotics.com/pi.html) , and see if you can get PiCamera to output stereo?

@realizator
Copy link

@realizator realizator commented Oct 17, 2018

@rohbotics, it just works! :-) The only thing I did is put dt-blob.bin (device tree for StereoPi) in a boot partition to enable two cameras support.
Thank you!

scrnsht

@realizator
Copy link

@realizator realizator commented Oct 17, 2018

@rohbotics, concerning @wezzoid question - how do you think, what is the simplest way to publish nodes for stereo processing?

@rohbotics
Copy link
Member

@rohbotics rohbotics commented Oct 17, 2018

I'll have to look into the mmal format for stereo, but the way to do it is basically publish 2 separate images and use stereo_image_proc for combining them into a stereo_msgs/DisparityImage in ROS.

A couple concerns:

  • I think we will be limited to raw publishing, or will have to find a way to cut a jpeg in half to publish it separately
  • Calibration, how do we save calibration data for each camera separately?

Also PRs are welcome :)

@realizator
Copy link

@realizator realizator commented Oct 17, 2018

As an idea - you can use two cameras separately. In raspivid I use -cs 0 and -cs 1 to use cameras independently and simultaneously (in my case it is front view and rear view on a drone, not stereoscopic mode). The only disadvantage is not so good image synchronization in comparison with -3d single image capture mode.

But as first step for tests it may be Ok.
Is it possible to load and use two instances of your driver and two cameras simultaneously? Is this solve problem with separate calibration data for each camera?

@rohbotics
Copy link
Member

@rohbotics rohbotics commented Oct 17, 2018

Being able to do select a camera for each instance of the node would be a really good first step in supporting stereo.

I'll look into it.

@davecrawley
Copy link
Contributor

@davecrawley davecrawley commented Oct 17, 2018

@wezzoid
Copy link
Author

@wezzoid wezzoid commented Oct 19, 2018

I used the picamera Python lib to get stereo working for my project (https://hackaday.io/project/159212-augmented-mekamon-robot) I put the camera in stereo mode, loaded the data in as an array, chopped it in half and published the left & right frames to ROS, basically. Doing it in Python is slow. To get around this I put the camera in top-bottom stereo mode, which means chopping the image data in half is just one 'cut' instead of one cut per row.

Here's my (amateurish) code: https://pastebin.com/RQ4DMBJj

@davecrawley the normal Pi 3 doesn't have the right pins exposed. Only the compute module allows this. Why Raspberry don't want to add a second CSI connector to their revised boards I don't know.

@realizator
Copy link

@realizator realizator commented Oct 21, 2018

@wezzoid thank you for your sample code! It is right this example that we were looking for! I will try to combine it with @rohbotics camera selection support from his last pull request after tests.

@realizator
Copy link

@realizator realizator commented Nov 20, 2018

@wezzoid thank you for your code sharing. After minor changes it works fine:

screenshot from 2018-11-20 10-31-23

BTW I had tried to contact you on hackaday.io but did not received any answer. Are you still there?

@rohbotics
Copy link
Member

@rohbotics rohbotics commented Nov 21, 2018

@realizator That is very cool! Looks like I am going to have to pick up a StereoPi 😄

@realizator
Copy link

@realizator realizator commented Nov 21, 2018

@rohbotics StereoPi will be available after our coming-soon crowdfunding (here is pre-launch page), but I can send you one of my current revision board for experiments right now. I just need your address for parcel.
I wrote you email to send2rohan@... but did not receive a response (possibly my message was marked as a spam). You can write me to realizator[at]gmail[dot]com
I wonder if it is possible to make stereoscopic driver for ROS like other stereoscopic cameras have. It will be compatible with original Raspberry Pi CM3 devboard and with StereoPi. Also we plan to do StereoPi schematic open source, and I hope some companies like Adafruit will help us to make this board mass-producted and available for DIY enthusiasts.

@wezzoid
Copy link
Author

@wezzoid wezzoid commented Nov 21, 2018

@realizator - awesome stuff!
And that StereoPi board looks VERY interesting!
I clearly haven't been checking my Hackaday, oops.

@realizator
Copy link

@realizator realizator commented Nov 26, 2018

@wezzoid, @rohbotics - I put all my files used for tests here: https://github.com/realizator/StereoPi-ROS-depth-map-test
May be they will be useful.

@nirvenesh20
Copy link

@nirvenesh20 nirvenesh20 commented Mar 1, 2019

I'm very sorry guys, i'm actually developing a visual for monocular visual impaired patients on which i am developing a depth map, i'd like to experiment with your code but i am not sure how to enable the stereoscopic camera setup on the compute module. would really appreciate that

@realizator
Copy link

@realizator realizator commented Mar 1, 2019

@nirvenesh20 If you are talking about Raspberry Pi Development board with Compute Module and two cameras - here is the manual from RPi Foundation: https://www.raspberrypi.org/documentation/hardware/computemodule/cmio-camera.md

I'd like to notice that by default the Devboard have 1 adapter for camera and another for connecting display. So you need to buy second adapter to connect two cameras (https://uk.farnell.com/raspberry-pi/rpi-cmdk-adapter/adaptor-compute-module-camera/dp/2433318?st=raspberry%20adapter). Looks like it is out of stock now, so you need to google it.

@nirvenesh20
Copy link

@nirvenesh20 nirvenesh20 commented Mar 1, 2019

@nirvenesh20 If you are talking about Raspberry Pi Development board with Compute Module and two cameras - here is the manual from RPi Foundation: https://www.raspberrypi.org/documentation/hardware/computemodule/cmio-camera.md

I'd like to notice that by default the Devboard have 1 adapter for camera and another for connecting display. So you need to buy second adapter to connect two cameras (https://uk.farnell.com/raspberry-pi/rpi-cmdk-adapter/adaptor-compute-module-camera/dp/2433318?st=raspberry%20adapter). Looks like it is out of stock now, so you need to google it.

Thank You very much for the swift reply
I've gotten a PiZero camera cable as an alternative, I'm not sure if it affects in any way , also I'm booting with an image link you provided for ROS and using a Waveshare compute module IO board with CM3L. Any further advice on my configuration?

@realizator
Copy link

@realizator realizator commented Mar 1, 2019

@nirvenesh20 Ok, now I understand your problem.
If you use our image, you also use our dt-blob.bin This file explains to Raspbian how the second camera is connected. So you need to connect wires as explained in Raspberry manual, and put dt-blob.bin from their manual to /boot partition instead of our dt-blob.bin. It should work.
The easiest way to test if it is everything Okay with cameras are these 3 steps:

  1. Open terminal
  2. type
    raspistill -cs 0
    You should see 5 seconds preview from your first camera
  3. type
    raspistill -cs 1
    You should see 5 seconds preview from your second camera.

UPD> Just noticed you use Waveshare IO board. I have no any experience with it, but hope it is pin-to-pin compatible with original Raspberry's devboard, and you can use Raspberry's guide.

@nirvenesh20
Copy link

@nirvenesh20 nirvenesh20 commented Mar 1, 2019

@realizator Thank you very much, will try it and keep you informed
You're a saviour

@nirvenesh20
Copy link

@nirvenesh20 nirvenesh20 commented Mar 1, 2019

@realizator
Copy link

@realizator realizator commented Mar 1, 2019

@nirvenesh20 try to do several steps back. There are lot of possible reasons of fail (wires commutation at devboard, bad cameras, bad or wrong ribbon connections etc.)

  1. Use clear Raspbian image, without any dt-blob.bin
  2. Try to preview picture from one of your cameras.
    UPD> Do not forget to run raspi-config and turn on camera support before using raspistill on stock Raspbian.
    UPD2> On stock raspbian without dt-blob.bin use raspistill command instead of raspistill -cs 0. cs option means "Camera Select" and works on two-cameras systems only.
  3. If step 2 is Ok, try to connect another camera to the same CSI connector and take second picture.
  4. If 2 and 3 is Ok - both cameras and ribbons are ok, go to the next step.
  5. Add Raspberry's dt-blob.bin and check both cameras. Notice: please use appropriate dt-blob.bin for your wires commutation on the devboard.
  6. If 5 is Ok, copy Raspberry's dt-blob.bin, and try to obtain images from each camera. When it will work, just put the same dt-blob.bin to our image.
@realizator
Copy link

@realizator realizator commented Mar 1, 2019

@nirvenesh20 By the way I see another wallpaper on your photo. I can suggest you did not use our full linux image, but just our files on Github. That is camera support on your image can be disabled. Try to start sudo raspi-config and check if it is turned on.

@nirvenesh20
Copy link

@nirvenesh20 nirvenesh20 commented Mar 2, 2019

@realizator Thank you for your detailed instructions, everything works absolutely well. I am able to run the publisher as well with no errors. But how do i excute step 3 onwards from https://github.com/realizator/StereoPi-ROS-depth-map-test . If everything works well, I will purchase the Stereopi , when it starts shipping in April . Also i really appreciate your time and effort, its really nice of you to guide a person like me

Also the image i was previously using was the latest image provided on your page published on the 19/2 /2019. However I'm now using an older image from the same page.
Thank You

@realizator
Copy link

@realizator realizator commented Mar 4, 2019

@nirvenesh20 Could you please clarify your question.
If you already put all files in your home folder, you need just copy-paste command from step 3 in your console and run it. And, as you may guess, you need to run several console instances (4), and run each command in a separate console window.
p.s. Image you use is not my image, but image from Ubiquiti Robotics. :-) I'm one of the StereoPi developers, it is another team. We use this image because it has great support, always updated and works out-of-the-box.

UPD> I added terminal instances notice in my Github repository

@nirvenesh20
Copy link

@nirvenesh20 nirvenesh20 commented Mar 5, 2019

@realizator I did run the command on a different console/separate console window . But i get an error [rospack] Error: package 'stereo_image_proc' not found . I have never got a proper link to search for this package as well.

@realizator
Copy link

@realizator realizator commented Mar 5, 2019

@nirvenesh20 I think you should install it something like this:
sudo apt-get install ros-kinetic-stereo-image-proc

@nirvenesh20
Copy link

@nirvenesh20 nirvenesh20 commented Mar 6, 2019

@realizator thank you very much. Everything worked perfectly. Also is there anyway i could enlarge the disparity map so that it could display in full screen.

UPD >also i think it should be ROS_NAMESPACE=stereo rosrun stereo_image_proc stereo_image_proc instead of ROS_NAMESPACE=stereo rosrun stereo_image_proc stereo_ige_proc

@realizator
Copy link

@realizator realizator commented Mar 6, 2019

@nirvenesh20

  1. Thank you for your typo notice ("ige" instead of "image"), I've fixed it.
  2. I'm not a guru in ROS, so you should ask some more advanced guys about full-screen preview :-) I just want to notice, that in a real onboard robot depth map implementation it is better to turn off visualization at all, as it takes a lot of CPU resources.
  3. All my experiments was proof-of-concept. For better results I recommend you to do cameras calibration. It will dramatically increase depth map precision and smoothness.
@nirvenesh20
Copy link

@nirvenesh20 nirvenesh20 commented Mar 20, 2019

@realizator I've been very much intrested with the stereopi, but was just curious, if the stereo camera requires any GPIO jumpers to be connected. Its a concern as i plan to attach a 3.5 inch touch display (something like this https://www.cytron.io/p-3.5-inches-tft-touch-screen-for-rpi-3?currency=MYR&gclid=Cj0KCQjwpsLkBRDpARIsAKoYI8w13Jbbnhs0VCPdAhWUcp9P_4unY-9HX9MaTlGX6zm5806LN0B7ey8aAu0iEALw_wcB to the stereopi, and it wouldn't be possible if there's need for GPIO connectors for the cameras

@realizator
Copy link

@realizator realizator commented Mar 20, 2019

@nirvenesh20 Your question is a bit offtop here (as it is not about raspicam_node). Could you please ask it here on a Raspberry Pi forum? Thank you!

@Lashhev
Copy link

@Lashhev Lashhev commented Nov 9, 2019

I managed to adapt the new mmal api with support for stereo mode. You should try my fork https://github.com/Lashhev/raspicam_node.git (stereo branch)

@Lashhev Lashhev mentioned this issue Nov 9, 2019
@rohbotics
Copy link
Member

@rohbotics rohbotics commented Nov 9, 2019

@Lashhev That is awesome! Could you please make a PR so that we can get the necessary changes merged in to this node?

@Lashhev
Copy link

@Lashhev Lashhev commented Nov 16, 2019

@rohbotics, Oh. Ok.
In which branch should I make PR?

@rohbotics
Copy link
Member

@rohbotics rohbotics commented Nov 16, 2019

@tiralonghipol
Copy link

@tiralonghipol tiralonghipol commented Jun 4, 2020

hi @rohbotics
i followed the discussion and I was wondering if the stereo ros node can be applied to this stereo hat (https://www.arducam.com/product/b0195-synchronized-stereo-camera-hat-raspberry-pi/)
can you give a feedback on this ?

Thank you in advance!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants