Startup camera sync via initializing/deinitializing cycle does not work with GigE camera

[llvs]

I use the spinnaker_sdk_camera_driver and I experience an issue of which I don't know what the core of the issue might be.

Your ROS driver detects the camera and the flushing also seems to work (see log of roslaunch below) However, the camera seems to not start with frame ID 0.

The camera is a new Blackfly S with standard factory firmware. It is connected via GigE with Power-over-Ethernet network cable. The camera also works fine in spinview.

The test was done in VirtualBox on a fully updated Lubuntu 16.04.

What can I do, to have the camera start with a certain frame number?
Console output is as follows:

[ INFO] [1539259721.897108867]: [ OK ] USB memory: 1000 MB
20181011[ INFO] [1539259721.897183646]: *** PARAMETER SETTINGS ***
[ INFO] [1539259721.897203254]: ** Date = 20181011
[ INFO] [1539259721.897739443]:   Save path set via parameter to: /home/pw
[ INFO] [1539259721.897782787]:   Camera IDs:
[ INFO] [1539259721.898293787]:     <serial no>
[ INFO] [1539259721.898791570]:   Camera Aliases:
[ INFO] [1539259721.898814759]:     <serial no> >> cam0
[ INFO] [1539259721.899740583]:   Unique time stamps for each camera: 0
[ INFO] [1539259721.900238189]:   color set to: 0
[ INFO] [1539259721.900932690]:   Exporting images to ROS: 1
[ INFO] [1539259721.901459004]:   Showing live images setting: 0
[ INFO] [1539259721.901925390]:   Showing grid-style live images setting: 0
[ INFO] [1539259721.902395142]:   Max Rate Save Mode: 0
[ INFO] [1539259721.902891314]:   Displaying timing details: 0
[ INFO] [1539259721.904583561]:   No. of images to skip set to: 20
[ INFO] [1539259721.906793675]:   Init sleep delays set to : 20 sec
[ WARN] [1539259721.907340806]:   'fps' Parameter not set, using default behavior: fps=20.00
[ INFO] [1539259721.907794313]:   'exp'=0, Setting autoexposure
[ INFO] [1539259721.908270332]:   Binning set to: 2
[ INFO] [1539259721.908731872]:   Using Software rate control, rate set to: 20
[ INFO] [1539259721.909187848]:   Saving images set to: 0
[ INFO] [1539259721.910053749]:   Camera coeffs not provided correctly, camera info messges will not be published.
[ INFO] [1539259721.910122159]: Creating system instance...
[ INFO] [1539259721.910468338]: Retreiving list of cameras...
[ INFO] [1539259723.542994586]: Numer of cameras found: 1
[ INFO] [1539259723.543034197]:  Cameras connected: 1
[ INFO] [1539259723.543082408]:   -<serial no>
[ INFO] [1539259723.545509510]: *** FLUSH SEQUENCE ***
[ INFO] [1539259723.545538521]: Initializing cameras...
[ INFO] [1539259724.172084859]: Deinitializing cameras...
[ INFO] [1539259725.952781566]: All cameras deinitialized.
[ INFO] [1539259725.952871993]: Initializing cameras...
[ INFO] [1539259726.425770815]: *** ACQUISITION ***
[FATAL] [1539259726.495451653]: ASSERTION FAILED
    file = /home/pw/ros/ws/spinnaker_ws/src/spinnaker_camera_driver/src/camera.cpp
    line = 59
    cond = frameID_ == 0
    message = [FATAL] [1539259726.495545610]: First frame ID was not zero! Might cause sync issues later...
[FATAL] [1539259726.495617142]: 

You suggested in private conversation:

We dont have any GigE versions at our lab, so haven't tested with them. The assertion for the frame number is because when using multiple camera we want to make sure they are synced. If you are only using a single camera, you should be able to safely comment out the assertion at:
Link to source

Your suggestion above is not possible for us, as we want to use a stereo camera pair and want to have synced cameras.

[karanchawla]

Having the same issue, I commented out the assertion line, however, that leads to the node crashing eventually.

[ghost]

What is ur setup exactly? GIG E or ethernet. How many cameras?

[ghost]

@karanchawla the above question is for you.

[karanchawla]

I’m away from the computer at the moment, will send a complete bug report in a bit.

[llvs]

I asked the manufacturer FLIR about this and they told me (translated quote):

There actually is a difference between the GigE and USB3 variant: The GigE camera resets FrameID, if AcquisitionStop, AcquisitionStart is executed. The USB3 does this only on EndAcquisition, BeginAcquisition. The counter cannot be reset manually.

HTH

[ghost]

@llvs and @karanchawla are you guys using the GPIO cable to trigger the cameras as described in the ReadMe. If you are setting up the code for 2 camera and the trigger doesn't happen correctly, the code would timeout during the image acquisition on the slave camera and terminate.

@llvs before worrying too much about the frame id's it would be worth checking if it is an issue. If you are able to acquire images from 2 cameras, I suggest pointing them at a stopwatch and checking if they are synced.

[llvs]

No, as the problems using the node happen already with a single camera. However, in the future I assume that we will need to employ hardware triggering via GPIO.

[karanchawla]

My setup is as follows:
Camera: Blackfly S
Sensor: RGB
Connection: GIG E

I'm trying to use this repository to acquire images however, the node would crash after a few minutes after giving a few warnings about image might be incomplete!

[ghost]

@karanchawla which particular model are you using. How many megapixels camera is it?

[karanchawla]

Specs for the camera I'm using: https://www.ptgrey.com/blackfly-s-color-32-mp-gige-vision-sony-imx265

[ghost]

So does it work fine for a few mins?

[karanchawla]

Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://ThinkPad-L380-Yoga:43951/

SUMMARY
========

PARAMETERS
 * /acquisition_node/binning: 2
 * /acquisition_node/cam_aliases: ['cam0']
 * /acquisition_node/cam_ids: [18434273]
 * /acquisition_node/color: True
 * /acquisition_node/delay: 0
 * /acquisition_node/exp: 0
 * /acquisition_node/frames: 0
 * /acquisition_node/live: False
 * /acquisition_node/live_grid: False
 * /acquisition_node/master_cam: 18434273
 * /acquisition_node/max_rate_save: False
 * /acquisition_node/save: False
 * /acquisition_node/save_path: ~
 * /acquisition_node/save_type: bmp
 * /acquisition_node/skip: 30
 * /acquisition_node/soft_framerate: 15
 * /acquisition_node/time: False
 * /acquisition_node/to_ros: True
 * /acquisition_node/utstamps: False
 * /rosdistro: kinetic
 * /rosversion: 1.12.14

NODES
  /
    acquisition_node (spinnaker_sdk_camera_driver/acquisition_node)

ROS_MASTER_URI=http://localhost:11311

process[acquisition_node-1]: started with pid [4337]
[ INFO] [1539896282.597559438]: [ OK ] USB memory: 1000 MB
20181018[ INFO] [1539896282.597629787]: *** PARAMETER SETTINGS ***
[ INFO] [1539896282.597642561]: ** Date = 20181018
[ INFO] [1539896282.598052658]:   Save path set via parameter to: /home/skyryse
[ INFO] [1539896282.598085649]:   Camera IDs:
[ INFO] [1539896282.598526005]:     18434273
[ INFO] [1539896282.599001187]:   Camera Aliases:
[ INFO] [1539896282.599019405]:     18434273 >> cam0
[ INFO] [1539896282.599881492]:   Unique time stamps for each camera: 0
[ INFO] [1539896282.600393175]:   color set to: 1
[ INFO] [1539896282.600880416]:   Exporting images to ROS: 1
[ INFO] [1539896282.601281588]:   Showing live images setting: 0
[ INFO] [1539896282.601648232]:   Showing grid-style live images setting: 0
[ INFO] [1539896282.602022591]:   Max Rate Save Mode: 0
[ INFO] [1539896282.602444702]:   Displaying timing details: 0
[ INFO] [1539896282.602865232]:   No. of images to skip set to: 30
[ INFO] [1539896282.603289117]:   Init sleep delays set to : 30 sec
[ WARN] [1539896282.603729684]:   'fps' Parameter not set, using default behavior: fps=20.00
[ INFO] [1539896282.604159206]:   'exp'=0, Setting autoexposure
[ INFO] [1539896282.604585260]:   Binning set to: 2
[ INFO] [1539896282.604973468]:   Using Software rate control, rate set to: 15
[ INFO] [1539896282.605421087]:   Saving images set to: 0
[ INFO] [1539896282.606245677]:   Camera coeffs not provided correctly, camera info messages will not be published.
[ INFO] [1539896282.606282651]: Creating system instance...
[ INFO] [1539896282.606560907]: Retrieving the list of cameras...
[ INFO] [1539896284.732364455]: Numer of cameras found: 1
[ INFO] [1539896284.732453493]:  Cameras connected: 1
[ INFO] [1539896284.732554350]:   -18434273
[ INFO] [1539896284.739698339]: *** FLUSH SEQUENCE ***
[ INFO] [1539896284.739773712]: Initializing cameras...
[ INFO] [1539896285.389000848]: Deinitializing cameras...
[ INFO] [1539896287.166753612]: All cameras deinitialized.
[ INFO] [1539896287.166924233]: Initializing cameras...
[ INFO] [1539896287.645791210]: *** ACQUISITION ***
[ WARN] [1539896289.066276953]: Image incomplete with image status 9!
[ WARN] [1539896290.266493358]: Image incomplete with image status 9!
.
.
.
[ WARN] [1539896340.474110609]: Image incomplete with image status 9!
[FATAL] [1539896342.525220597]: Some exception occured. 
                                                         Exiting gracefully, 
  possible reason could be Camera Disconnection...
[ INFO] [1539896342.599320622]: Deinitializing cameras...
[ INFO] [1539896343.624576840]: All cameras deinitialized.
[ INFO] [1539896343.624668475]: Clearing camList...
[ INFO] [1539896343.624731904]: Releasing camera pointers...
[ INFO] [1539896343.624787122]: Releasing system instance...
terminate called after throwing an instance of 'std::bad_cast'
  what():  std::bad_cast

[karanchawla]

It works for less than a minute after which it crashes. I have posted the log in the comment above.

[ghost]

@karanchawla That issue seems to be with the Spinnaker SDK as opposed to the ROS driver. See
https://www.ptgrey.com/KB/11142 at the end under Troubleshooting.

• Can you run SpinView with similar frame rates and settings without any issues?
• Try running at lower frame rate just to see if this is related to the bandwidth.
• Also try to use binning to reduce image size. You can set it to 2 for testing.

[ghost]

@karanchawla and @llvs were you guys able to get the cameras to work? Do you have an update about the issues?

[mikedef]

Hi @shahvi I am getting the exact same issue as @karanchawla. I have a Blackfly BFLY-PGE-20E4C. I had a frameID error to start with as above. I then commented line 59 in camera.cpp. Once I did that the camera would run for under a minute and then would crash again. Same failure as above.

I can run spinview at any framerate available. I have set frame:=10 for the ROS node.

[ghost]

@mikedef Its difficult to debug without us having that type of camera. One thing you could try would be to launch the code with the debug turned on by changing the first line in

spinnaker_sdk_camera_driver/launch/acquisition.launch

Line 3 in fd3d35e

<env name="ROSCONSOLE_CONFIG_FILE" value="$(find spinnaker_sdk_camera_driver)/cfg/std_console.conf"/>

to debug_console.conf

See if the debug messages help...

[mikedef]

@shahvi changing to debug_console.conf did not give me any updated debug messages.

Here is my latest error message: Spinnaker: Failed waiting for EventData on NEW_BUFFER_DATA event [-1011]
I see see that there was a closed ticket for this error, but no solution to it. Do you have any updated information on this buffer data issue?
Thanks

[vik748]

@mikedef , I am really out of ideas. It seems to be happening in the Spinnaker_SDK and I am not sure what it might be. Hard to know without having one to play with.

[mikedef]

@shahvi Right I am not sure either. Also the color flag does not work for this camera as well.
Spinview works fine. So I am not sure why it's not working with this ROS app. Thanks for the help either way.

[mikedef]

Hi @shahvi looks like the Event Data Spinnaker error was solved by increasing UDP buffer size to 25MB by 'sudo sysctl -w net.core.rmem_max=26214400' and 'sudo sysctl -w net.core.rmem_default=26214400'. I found this on a different forum. Sorry about the back and forth. I figured since the camera worked fine with Spinview that it would work fine with the ROS node.

[vik748]

@mikedef thanks for the update. So does the driver work properly once you make this change?

[mikedef]

@shahvi The driver does not work in color mode. It does work when in black and white. I had to make a few changes, mostly commenting out of error checking code as highlighted above.

I get the following errors when setting the color flag to true
[FATAL] [1549565095.362445492]: Unable to set PixelFormat to BGR8 (entry retrieval). Aborting...
[FATAL] [1549565095.362619558]: Error: Spinnaker: LogicalErrorException NULL pointer dereferenced [-2005]

[vik748]

Setting the eumeration takes place in:

spinnaker_sdk_camera_driver/src/camera.cpp

Line 134 in fd3d35e

void acquisition::Camera::setEnumValue(string setting, string value) {

Maybe these GigE cameras have a different enumeration entry ? You can check in SpinView, what enumerations the PixelFormat option accepts.

[mikedef]

@shahvi Looks like my only option is BayerGB8. Where is this set exactly? I changed line 96 to PixelFormat_BayerGB8 and did a catkin_make, but I still get the BGR8 error as before.

[vik748]

You should be able to change it at

spinnaker_sdk_camera_driver/src/capture.cpp

Line 508 in fd3d35e

cams[i].setEnumValue("PixelFormat", "BGR8");

[mikedef]

@shahvi Sorry it's actually BayerBG8 not GB. Either way it works when I change the pixel format to BayerBG8.
Thanks for the help. The ROS node is now running with the default settings and the color flag set to true.

[llvs]

In the meantime we found the following regarding the errors with:

Image incomplete with image status 9!

The two cameras are flooding the IP connection with packets. To have more than one camera coexist in peace one has to divide the GigE bandwith between the cameras. This is done via the
PacketDelay parameter which may also be called GevSCPD (Gigevision Stream Channel Packet Delay) or the corresponding parameter DeviceLinkThroughputLimit (which is easier to understand and set a meaningful value).
You can set the DeviceLinkThroughputLimit to 61644209 (equals PacketDelay of 73760 for FLIR camera tick frequency) and this equals half of GigE bandwidth. Or you may use 40841495 (PacketDelay 147880) which is a third of GigE bandwidth to have some space.
Please note that limiting device bandwidth of course reduces the frame rate such that the necessary data transfer does not exceed the limit.

[angegu]

@llvs That solves the image incomplete and event buffer timeout issue thanks!

[alankwok97]

@llvs how would you set that up with ros in terminal exactly? I can find the parameters in spinview but changing in spinview doesn't seem to help in roslaunching the application as the parameters are reset the next time I open spinview.

Many thanks.

[kaalessi]

@alankwok97 You can save SpinView settings to your camera by typing user into the search box in SpinView. Change the parameter UserSetSelector to UserSet0 or UserSet1. Then click the execute button for UserSetSave. Now your settings are saved and all you have to do now is change UserSetDefault to match your desired UserSet. Everytime your camera is booted up, the settings saved will be used. If you want to go back to Default, you can select Default under UserSetSelector and click Execute for UserSetLoad.

[alankwok97]

@kaalessi Thanks for your reply!

[atyshka]

@llvs what could be causing this error if I’m using only one camera? You said in your case with multiple cameras they were flooding the network, but this shouldn’t happen with a single camera, right?

[llvs]

I can only guess here, but it may have to do with too small network buffers. The linux defaults are maybe too little. Or you may want to use Ethernet Jumbo Frames if the camera allows it to reduce network load. But those are only guesses.
HTH