arv-camera-test encounters missing packets for nearly all frames

[cctang]

Describe the bug
arv-camera-test appears to receive no frames from the camera because of missing packet errors. Representative debug information:

[GvStream::missing_packet_check] Resend request at dt = 6837, packet id = 1008 (1216 packets/frame)
[GvStream::send_packet_request] frame_id = 66 (from packet 1007 to 1007)
packet_type = cmd
packet_flags = none
command = packet-resend-cmd
size = 12
id = 8048
00000000 42 00 00 40 00 0c 1f 70 00 00 00 42 00 00 03 ef B..@...p...B....
00000010 00 00 03 ef ....
[GvStream::missing_packet_check] Resend request at dt = 6885, packet id = 1010 (1216 packets/frame)
[GvStream::send_packet_request] frame_id = 66 (from packet 1009 to 1009)
packet_type = cmd
packet_flags = none
command = packet-resend-cmd
size = 12
id = 8049
00000000 42 00 00 40 00 0c 1f 71 00 00 00 42 00 00 03 f1 B..@...q...B....
00000010 00 00 03 f1 ....
[GvStream::process_data_block] Received resent packet 187 for frame 66
[GvStream::process_data_block] Received resent packet 188 for frame 66
[GvStream::process_data_block] Received resent packet 189 for frame 66
[GvStream::process_data_block] Received resent packet 190 for frame 66
[GvStream::process_data_block] Received resent packet 191 for frame 66
[GvStream::process_data_block] Received resent packet 192 for frame 66
[GvStream::process_data_block] Received resent packet 193 for frame 66
[GvStream::process_data_block] Received resent packet 194 for frame 66
[GvStream::process_data_block] Received resent packet 195 for frame 66
[GvStream::process_data_block] Received resent packet 196 for frame 66
[GvStream::process_data_block] Received resent packet 197 for frame 66
[GvStream::process_data_block] Received resent packet 202 for frame 66
[GvStream::missing_packet_check] Resend request at dt = 7030, packet id = 1038 (1216 packets/frame)
[GvStream::send_packet_request] frame_id = 66 (from packet 1011 to 1037)
packet_type = cmd
packet_flags = none
command = packet-resend-cmd
size = 12
id = 8050
00000000 42 00 00 40 00 0c 1f 72 00 00 00 42 00 00 03 f3 B..@...r...B....
00000010 00 00 04 0d ....
[GvStream::check_frame_completion] Timeout for frame 54 at dt = 115629
[GvStream::close_frame] Close frame 54

To Reproduce
Steps to reproduce the behavior:

1. cloned this repository
2. checked out ARAVIS_0_8_1
3. compiled using meson compile
4. executed ./arv-camera-test --debug all:3 from build/tests directory

Expected behavior
I expect arv-camera-test to successfully receive frames from the camera with no errors.

Camera description:

• Manufacturer: Xenics Infrared Solutions
• Model: Bobcat 640
• Interface: GigE

Platform description:

• Aravis version: 0.8.1
• OS: ubuntu 18.04 LTS
• Hardware: ARMv8 (NVIDIA Jetson Xavier NX)

Additional context

• Camera and Jetson Xavier NX are connected by CAT6 cables to a Netgear gigabit switch
• I have encountered similar errors using different cameras (Allied Vision Technologies Manta G-504B and Prosilica GT1910)

Full output with debug information:

• Xenics Bobcat 640: camera_test_log_bobcat.txt
• Allied Vision Manta G-504B: camera_test_log_manta.txt
• Allied Vision Prosilica GT1910: camera_test_log_gt1910.txt

[EmmanuelP]

Hi,

Did you try the following tweaks:

• MTU increase
• Socket buffer size increase
• use of packet socket

What is the CPU consumption ?
What are the image acquisition parameters (PixelFormat, image size, frame rate) ?

[cctang]

Hi @EmmanuelP,

Thanks for your quick reply!

I must admit, I'm not certain how to tweak socket buffer size or how to use packet socket. I'll look into how to tweak those. If you have a guide you would share, I would appreciate it. For the MTU increase, I have tried changing the value from automatic to 8000, 9000, and 9014. None of those helped. I don't know how to change it on the camera. Is there a setting in Aravis I need to also change?

For CPU consumption, one of the 6 cores goes to 100% and the other 5 cores increase to around 50% when I run arv-camera-test.
The camera image size is 640 x 512 with 14-bit grayscale per pixel. The max frame rate of the camera is 100 Hz.

Thanks again!

[cctang]

I discovered arv-tool just now, but don't really know how to use it. I also found an example in the issue tracker for changing the camera MTU to match what I'm using in Ubuntu Network settings. That doesn't help. The only other thing I discovered how to set is the image size. I set it to the minimum (32x4) and now am receiving frames. It seems like I'm headed in the right direction by accident, but I'm not sure how best to proceed.

Here's a sample of the output:

➜  tests git:(56fe407) ✗ arv-tool-0.8 control GevSCPSPacketSize
Xenics-Bobcat-640-GigE-6265 (192.168.1.168)
GevSCPSPacketSize = 576 min:512 max:9196
➜  tests git:(56fe407) ✗ arv-tool-0.8 control GevSCPSPacketSize=9196
Xenics-Bobcat-640-GigE-6265 (192.168.1.168)
GevSCPSPacketSize = 9196 min:512 max:9196
➜  tests git:(56fe407) ✗ arv-tool-0.8 control Width
Xenics-Bobcat-640-GigE-6265 (192.168.1.168)
Width = 640 min:32 max:640 inc:16
➜  tests git:(56fe407) ✗ arv-tool-0.8 control Width=32
Xenics-Bobcat-640-GigE-6265 (192.168.1.168)
Width = 32 min:32 max:640 inc:16
➜  tests git:(56fe407) ✗ arv-tool-0.8 control Height  
Xenics-Bobcat-640-GigE-6265 (192.168.1.168)
Height = 512 min:4 max:512 inc:4
➜  tests git:(56fe407) ✗ arv-tool-0.8 control Height=4
Xenics-Bobcat-640-GigE-6265 (192.168.1.168)
Height = 4 min:4 max:512 inc:4
➜  tests git:(56fe407) ✗ ./arv-camera-test                          
Looking for the first available camera
vendor name           = Xenics
model name            = Bobcat-640-GigE
device id             = 6265
image width           = 32
image height          = 4
horizontal binning    = 0
vertical binning      = 0
payload               = 256 bytes
exposure              = 0 µs
gain                  = 0 dB
gv n_stream channels  = 1
gv current channel    = 0
gv packet delay       = 0 ns
gv packet size        = 9196 bytes
10528 frames/s -     2.7 MiB/s
10537 frames/s -     2.7 MiB/s
10528 frames/s -     2.7 MiB/s
4368 frames/s -    1.12 MiB/s - 7 errors
7615 frames/s -    1.95 MiB/s - 3 errors
10538 frames/s -     2.7 MiB/s
10538 frames/s -     2.7 MiB/s
6482 frames/s -    1.66 MiB/s - 4 errors
10539 frames/s -     2.7 MiB/s
10537 frames/s -     2.7 MiB/s
10538 frames/s -     2.7 MiB/s
10535 frames/s -     2.7 MiB/s
10533 frames/s -     2.7 MiB/s
10538 frames/s -     2.7 MiB/s
10545 frames/s -     2.7 MiB/s
Completed buffers = 144913
Failures          = 14
Underruns         = 39199

[EmmanuelP]

Please try:

./tests/arv-camera-test -d device,stream --auto -f 10

And:

sudo ./tests/arv-camera-test -d device,stream --auto -f 10

And attach the console output.

[cctang]

Console output running arv-camera-test without 'sudo': camera_test_log_bobcat_d_device_stream_auto_f_10.txt

Console output running arv-camera-test with 'sudo': camera_test_log_bobcat_sudo_d_device_stream_auto_f_10.txt

[cctang]

Hi @EmmanuelP, I was wondering if you had an opportunity to look at the console output you asked me to collect last week. Thanks!

[EmmanuelP]

When arv-camera-test runs using sudo, packet_socket is enabled. In this case there is not missing packets. But for some reason, the transfers stop after 80 images. May be there is a parameter that limits the acquisition sequence to 80 images.

Also, there can be an issue when you are trying to transmit bigger images, because in this case the actual packet size may exceed what is supported by the network hardware on the path to the camera.

Before running arv-camera-test, try to run tests/arv-auto-packet-size, which should sets the maximum camera packet size to what is possible on your network.

[cctang]

@EmmanuelP, thanks for helping me investigate this problem! As you suggested, I ran tests/arv-auto-packet_size_test before running arv-camera-test. The output said it set the packet size to 1500 bytes. I then set the MTU size of the network adapter to 1500 bytes before running tests/arv-camera-test with sudo. It looks like acquisition stops after 49-69 frames on multiple tries.

This is the output for when acquisition stopped after 69 frames:
camera_test_log_bobcat_auto_packet_size_sudo_d_device_stream_auto_f_10.txt

Where would you suggest I look to see if there is a parameter that's interfering with image acquisition?

[EmmanuelP]

Where would you suggest I look to see if there is a parameter that's interfering with image acquisition?

In the camera documentation.

Something that can explain a stop of the streaming is the heartbeat packet that must be sent by aravis to the camera before a given timeout. Aravis by default sends the heartbeat packet every second, but there is a compile time option to send it faster.

It would help if you can attach a network capture using wireshark.