InvalidHandleException at the first fetch

[ahirner]

Describe the bug
Sometimes fetch_buffer() throws InvalidHandleException at the first fetch. This exception is not caught within harvesters.

To Reproduce
We use a Basler USB3 camera and which is listed correctly:

h = Harvester()
h.add_cti_file(<USB3 cti file>)
h.update_device_info_list()
print(h.device_info_list)
[(id_='Basler daA2500-14uc (22841742)', vendor='Basler', model='daA2500-14uc', tl_type='U3V', user_defined_name=None, serial_number='22841742', version='Unknown')]

First fetch_buffer():

ia = h.create_image_acquirer(0)
ia.start_image_acquisition()
with ia.fetch_buffer() as ia_buffer:
    <...>

Stacktrace:

Dec 12 07:43:44 localhost start.sh[23607]:     with ia.fetch_buffer() as ia_buffer:
Dec 12 07:43:44 localhost start.sh[23607]:   File "/usr/local/lib/python3.6/site-packages/harvesters/core.py", line 1785, in fetch_buffer
Dec 12 07:43:44 localhost start.sh[23607]:     self._update_chunk_data(buffer=_buffer)
Dec 12 07:43:44 localhost start.sh[23607]:   File "/usr/local/lib/python3.6/site-packages/harvesters/core.py", line 1713, in _update_chunk_data
Dec 12 07:43:44 localhost start.sh[23607]:     if buffer.num_chunks == 0:
Dec 12 07:43:44 localhost start.sh[23607]:   File "/usr/local/lib/python3.6/site-packages/genicam2/gentl.py", line 4346, in _get_num_chunks
Dec 12 07:43:44 localhost start.sh[23607]:     return _gentl.Buffer__get_num_chunks(self)
Dec 12 07:43:44 localhost start.sh[23607]: _gentl.InvalidHandleException: GenTL exception: Given handle does not support the operation. (Message from the source: Given handle does not support the operation) (ID: -1006)

Expected behavior
The exception doesn't occur, is handled in _update_chunk_handler (https://github.com/genicam/harvesters/blob/master/src/harvesters/core.py#L1739) or can be traced back to an invalid initialization.

Linux:
CoreOS within docker container and shared /dev/bus/usb.

Additional context
The problem occurs frequently after restarting our container and an extended period of time (>4h) without problems. It's possible that a deinitialization routine wasn't called properly upon program exit, which might be the root cause if some resources aren't released by harvesters.

[kazunarikudo]

Hi Alexander,

Thank you for the report. Could you tell me the GenTL Producer that you use?

Regards,
Kazunari

[ahirner]

Thanks, it's from the latest of pylon5 for USB3: lib64/gentlproducer/gtl/ProducerU3V-1.4.0.cti

[kazunarikudo]

Cool! I didn't know that they had started providing GenTL Producer!

[kazunarikudo]

When the num_chunks property is called, Harvester calls the following DSGetBufferChunkData function in the backgrond:

GC_ERROR DSGetBufferChunkData( DS_HANDLE hDataStream, BUFFER_HANDLE hBuffer,
SINGLE_CHUNK_DATA * pChunkData, size_t * piNumChunks )

And the GenTL Standard says as follows regarding GC_ERR_INVALID_HANDLE:

(1) The handle hDataStream is invalid (NULL) or (2) does not reference an open Data Stream module retrieved through a call to DevOpenDataStream or (3) the handle hBuffer is invalid (NULL) or (4) does not reference an announced Buffer module retrieved through a call to DSAllocAndAnnounce or DSAnnounceBuffer.

If the GenTL Producer follows the standard document, then we have the four options above and if we want to make sure if Harvester can release the resources we should log the behavior.

To log the behavior of Harvester please go through the following steps:

1. Copy a configuration file from the package directory to an arbitrary directory so that you can edit it without any concern. The file is located at site-packages/harvesters/src/harvesters/logging/logging.report.ini.
2. Create an environment variable, name it HARVESTERS_LOGGING_CONFIG and set a path to the configuration file as its value.
3. Edit the configuration file to name the output file and its location. The original code is located at the line 49 and says args=('harvesters.log', 'a', 'utf-8', 'False'). By default, it creates harvesters.log at the execution directory. If you want to create an output file naming it foo.log at /Users/bar then edit the line as follows: args=('/Users/bar/foo.log', 'a', 'utf-8', 'False'). Note that the file path must be absolute.
4. Finally, run your script until it reproduces the phenomenon.
5. Once you get the log file, could you drop the file here, please?

Oh, perhaps it's also worth trying to run with another GenTL Producer. Maybe bringing our the GenTL Producer from Matrix Vision?

[ahirner]

Thank you for both suggestions! I will follow through asap. It will likely take a few days to get to the system.

[kazunarikudo]

Hi Alexander,

I have written an advice for you at the chat room. Please take a look when you can.

Best regards,
Kazunari

[ahirner]

The first method didn't leave any file unfortunately.

root@d77fb6adadc9:/src# env | grep HARVEST
HARVESTERS_LOGGING_CONFIG=/usr/local/lib/python3.6/site-packages/harvesters/logging/logging.report.ini
root@d77fb6adadc9:/src# cat /usr/local/lib/python3.6/site-packages/harvesters/logging/logging.report.ini | grep /logs/ -A 3 -B 3
class=FileHandler
level=DEBUG
formatter=formatter
args=('/logs/harvesters.log', 'a', 'utf-8', 'False')

[formatter_formatter]
format=%(asctime)s :: %(name)s :: %(levelname)s :: %(message)s
root@d77fb6adadc9:/src# ls -lh /logs
total 0

To have more verbose output, I passed our own logger to Harvester and enabled HARVESTERS_LOG_BUFFER_MANIPULATION with a single restart of the image acquisition:
last_run_manual_stripped.log

Even if details are missing, maybe the sequence of events provide insight. Exceptions reoccur every 5 to 20 frames after restarting the image acquisition. Also, is there another way to probe what's causing GC_ERR_INVALID_HANDLE?

[kazunarikudo]

Hi Alexander.

Thank you for the update. Regarding the logging based on the environment variable, perhaps it might work just adding the export keyword like this export HARVESTERS_LOGGING_CONFIG=/path/to/foo.ini. However, you may face an issue that is described in #72 so please upgrade harvesters typing pip install -U --no-cache-dir harvesters if you want to make a try but please not that it's not necessary at least at this moment due to the following reason. The approach you took was sufficient to get a log file that I had expected to have. Thank you!

Well, I found something interesting in the log. Please look at the following block first.

2018-12-17 20:21:58,627 [INFO] (MainThread) Basler daA2500-14uc (22841742) started image acquisition.
2018-12-17 20:21:58,779 [DEBUG] (Thread-2  ) Acquired Buffer module #0 containing frame #0 from DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:58,780 [DEBUG] (MainThread) Fetched Buffer module #0 containing frame #0 of DataStream module Stream0 of Device module Stream0.
2018-12-17 20:21:58,785 [DEBUG] (MainThread) Queued Buffer module #0 containing frame #0 to DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
frame=    8 fps=1.5 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   0x    
2018-12-17 20:21:58,872 [DEBUG] (Thread-2  ) Acquired Buffer module #1 containing frame #1 from DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:58,966 [DEBUG] (Thread-2  ) Acquired Buffer module #2 containing frame #2 from DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:58,966 [DEBUG] (Thread-2  ) Queued Buffer module #1 containing frame #1 to DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:58,978 [DEBUG] (MainThread) Fetched Buffer module #2 containing frame #2 of DataStream module Stream0 of Device module Stream0.
2018-12-17 20:21:58,982 [DEBUG] (MainThread) Queued Buffer module #2 containing frame #2 to DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,059 [DEBUG] (Thread-2  ) Acquired Buffer module #3 containing frame #3 from DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
frame=    9 fps=1.2 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A speed=   0x    
2018-12-17 20:21:59,152 [DEBUG] (Thread-2  ) Acquired Buffer module #4 containing frame #4 from DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,152 [DEBUG] (Thread-2  ) Queued Buffer module #3 containing frame #3 to DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,245 [DEBUG] (Thread-2  ) Acquired Buffer module #5 containing frame #5 from DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,245 [DEBUG] (Thread-2  ) Queued Buffer module #4 containing frame #4 to DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,274 [DEBUG] (MainThread) Fetched Buffer module #5 containing frame #5 of DataStream module Stream0 of Device module Stream0.
2018-12-17 20:21:59,279 [DEBUG] (MainThread) Queued Buffer module #5 containing frame #5 to DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,338 [DEBUG] (Thread-2  ) Acquired Buffer module #6 containing frame #6 from DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,431 [DEBUG] (Thread-2  ) Acquired Buffer module #7 containing frame #7 from DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,432 [DEBUG] (Thread-2  ) Queued Buffer module #6 containing frame #6 to DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,525 [DEBUG] (Thread-2  ) Acquired Buffer module #8 containing frame #8 from DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,525 [DEBUG] (Thread-2  ) Queued Buffer module #7 containing frame #7 to DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,618 [DEBUG] (Thread-2  ) Acquired Buffer module #9 containing frame #9 from DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,618 [DEBUG] (Thread-2  ) Queued Buffer module #8 containing frame #8 to DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,711 [DEBUG] (Thread-2  ) Acquired Buffer module #10 containing frame #10 from DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,711 [DEBUG] (Thread-2  ) Queued Buffer module #9 containing frame #9 to DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,805 [DEBUG] (Thread-2  ) Acquired Buffer module #11 containing frame #0 from DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,805 [DEBUG] (Thread-2  ) Queued Buffer module #10 containing frame #10 to DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,900 [DEBUG] (Thread-2  ) Acquired Buffer module #12 containing frame #0 from DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,900 [DEBUG] (Thread-2  ) Queued Buffer module #11 containing frame #0 to DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).
2018-12-17 20:21:59,936 [ERROR] (MainThread) Exception during image aquisition (InvalidHandleException('GenTL exception: Given handle does not support the operation. (Message from the source: Given handle does not support the operation) (ID: -1006)',), retry 2

And I would like to get your attention to the following line:

2018-12-17 20:21:59,805 [DEBUG] (Thread-2 ) Acquired Buffer module #11 containing frame #0 from DataStream module Stream0 of Device module Basler daA2500-14uc (22841742).

See, even though your Harvester object had already acquired frame #0 at 2018-12-17 20:21:58,779 but it told you that it acquired frame #0 AGAIN(!) at 2018-12-17 20:21:59,900. Once you let the camera start transmitting images, then the camera must increment the frame number until the number overflows its defined bit width.

Having that log, I assume one or more stream packets that compose (probably) frame #11 were corrupted. Of course, it should be proven from the transport layer level using Wireshark though. The reason I assume this possibility is that I had faced the similar phenomenon with one of their USB3 cameras. In my case, it delivered an image with a chunk data but the chunk data had been corrupted (it's revealed because the GenICam reference implementation refused the corrupted data throwing a defined exception). Interestingly, when I get such a corrupted chunk data, then the image was completely black and such images were occasionally delivered.

If I wanted to investigate this issue, I would check if the camera occasionally transmits a corrupted frame having help from Wireshark anyway because it will expose a corrupted packet if the camera really transmits a corrupted one; you'll be able to check the frame ID field (sorry, I've forgotten the correct name) in the Wireshark log. If it tells you the packet incorrectly reports you that's frame #0, then its GenTL Producer will have to report Harvester that it's frame #0 anyway. Neither Harvester or GenTL Producers are willing to intentionally alter the information in general excepting a case where they have a bug.

Anyway, please don't worry I think I can keep working with you on this issue for a while.

Best regards,
Kazunari

[kazunarikudo]

Besides the Wireshark way I introduced above, you may have the following trials:

• Try with another camera.
• Swap USB3 accessories such as cables, hubs, and/or interface cards with other spare ones to improve electrical integrity.

Please note that the Wireshark ways are the most important. Other ways are counted as workaround candidates.

[ahirner]

Acquiring frame #0 again is indeed always precededing the exceptions. I tried to eliminate some conflating following factors:

• Different USB3 camera
• Different port on same machine
• More robust cable and connector
• Different machine
• Running outside of docker on Ubuntu 16.04

Unfortunately, the issue reproduced under all circumstances. I noticed that the frequency of exceptions, at least most often, gradually increases over some hours until no more images come through (with a restart delay of 0.4 seconds). We didn't try with a dedicated USB hub yet.

Black images were not ever transferred. I think an exception is always raised before. Instead of restarting the image acquisition thread, would it be worthwhile to fetch buffers with is_raw, discard incomplete frames and convert them to components manually?

The wireshark logs were done by capturing 1944x1600 BGR8 frames. Frame No. 21050 seems to be an anomaly (PENDING_ACK 500 ms). They can be correlated with the application logs starting 17 secs after the wireshark logs.

• Application logs: run_test_0_stripped.log
• Wireshark logs collapsed: wireshark_test_0_collapsed.log
• Wireshark logs full: wireshark_test_0.log

If helpful, a raw dump from wireshark is also available. Any further guidance is highly appreciated.

[kazunarikudo]

Hi Alexander,

Thank you for taking your time. To be honest I wish I could test with you having the idetical setup. I'm sorry about that.

Unfortunately, the issue reproduced under all circumstances.

It's indeed unfortunate but I see a light in such a high reproducibility: We can think that you have caught the root cause in the environment!

If helpful, a raw dump from wireshark is also available

Yes, please. I would like to have the Wireshark file (*.pcap) and check the stream packets in detail. It's not only the PENDING_ACK issue, something remains in my mind. I feel it's a bit strange.

would it be worthwhile to fetch buffers with is_raw, discard incomplete frames and convert them to components manually?

Unfortunately, it doesn't help you. Technically speaking perhaps it could be possible to parse the content of a buffer by yourself but it means you will have to know everything happening in the transport layer level but in most cases required information would be encapsulated by the GenTL Producer in a place where you can't reach out.

Black images were not ever transferred.

Thank you for the confirmation but it's okay because that was just a case I had.

I noticed that the frequency of exceptions, at least most often, gradually increases over some hours until no more images come through (with a restart delay of 0.4 seconds).

Could you describe the test you had more precisely?

Regards,
Kazunari

[ahirner]

At his point, I think I can attribute the issue to docker. Apparently, two factors lead to the same result.

First, an exception occurred after ~400 frames in the Ubuntu 16.04 environment. This led to the conclusions before. Running a second time, an exception occurred only after 39996 frames.
At this point /sys/module/usbcore/parameters/usbfs_memory_mb was 16. I changed that to an unlimited amount after which the system is acquiring > 50000 frames without any exception. Thus, the issue seems to be the interface between host and docker.

Second, exceptions occur frequently within the docker container. I can hardly reproduce the aggravation mentioned above on my local setup, but the 2h video files saved on the remote system drop from 1.1GB to 148MB to 0. Accordingly, the application logs not a single image acquired image before or in between a retry (sleep(0.2); stop_image_acquisition(); sleep(0.2); start_image_acquisition()).
Note, that the new usbfs_memory_mb limit of the host propagates to the root process in docker, but to no effect. It behaves the same as logged above. On my local setup, frames drop at random every 100 to 300. It seems to be affected by compute load in the acquisition loop. E.g., if saving video to disk is disabled, it lasts a few hundred frames longer.

Now, we also know that many or in fact all drivers I know of (Basler, Point Grey, Ximea, MV) demand to be scheduled with high priority. For that, I made the docker container as close as I know to the host system's capabilities: docker run --privileged=true -lxc-conf="aa_profile=unconfined" --security-opt seccomp:unconfined --pid=host --cap-add=ALL --ulimit rtprio=99, again to no effect.

How is scheduling priority handled in the Core -> GenTL Binding -> GenTL Producer stack? Would it use or benefit from a realtime scheduler?

In addition to a *.pcapng dump from the first run, I did a new strace -f of the last 1000 lines before the fatal frame #0. I hope to get new hints, because from an application lifeycycle point of view, docker is ideal.

• Wireshark logs raw: wireshark_test_0.pcapng
• strace until close to exception: run_1_strace_privileged_frame0.log

If helpful, I'm also more than willing to get an image into the wild for direct reproduction on your side.

[kazunarikudo]

Hi Alexander,

Thank you for the update. I have found something strange in the Wireshark log that you provided at this time:

Please take a look at an image that was tagged Block ID: 45. You may note that the URB data length of the consecutive stream payload packets unexpectedly turn from 61504 to 68 in bytes even though an image consists of the 61504 bytes stream payload packets and the 43328 bytes stream payload packet for the final transfer. In addition, once a stream payload packet turns 68 bytes, the following stream payload packets never come back to 61054 bytes.

As long as I read the Wireshark log, a concern which is relevant to the driver software comes to the top of my head. Would you get any idea from this impression?

Best regards,
Kazunari

[ahirner]

Good catch. I think both anomalies 68 bytes and Payload Data: 00000000 (meaning an incomplete black image if no exception was raised?) are relevant.

Thus, I grepped the frame numbers for both.

grep -n "Frame Length: 68 bytes" \ -B 1 wireshark_test_0.log \| \grep "Frame Number" \| \cut -d ':' -f2 \| awk '{$1=$1};1' 
grep -n "Payload Data: 00000000" \-B 34 wireshark_test_0.log \| \grep "Frame Number" \| \grep "Frame Number" \| \cut -d ':' -f2 \| awk '{$1=$1};1'

It follows, that Payload Data: 00000000 strictly occurs only for an incomplete frame. But not always. Interestingly, the first incomplete frames and frame 20998 have non-zero payload data. Something must go wrong in the USB stack. Intuitively the driver times out waiting for data and gets into an invalid state, but I don't know where to look at next.

Left Frame Number : (Frame Length: 68 bytes)
Right Frame Number : (Payload Data: 00000000)

6 |  
10 |  
14 |  
18 |  
22 |  
26 |  
30 |  
34 |  
38 |  
20754 |  
20815 | 20815
20816 | 20816
20817 | 20817
20818 | 20818
20819 | 20819
20820 | 20820
20821 | 20821
20823 | 20823
20824 | 20824
20825 | 20825
20826 | 20826
20827 | 20827
20828 | 20828
20833 | 20833
20834 | 20834
20835 | 20835
20836 | 20836
20837 | 20837
20841 | 20841
20842 | 20842
20843 | 20843
20844 | 20844
20845 | 20845
20849 | 20849
20850 | 20850
20851 | 20851
20852 | 20852
20853 | 20853
20854 | 20854
20858 | 20858
20859 | 20859
20860 | 20860
20861 | 20861
20865 | 20865
20866 | 20866
20867 | 20867
20868 | 20868
20869 | 20869
20870 | 20870
20876 | 20876
20877 | 20877
20878 | 20878
20879 | 20879
20880 | 20880
20881 | 20881
20882 | 20882
20886 | 20886
20887 | 20887
20888 | 20888
20889 | 20889
20890 | 20890
20895 | 20895
20896 | 20896
20897 | 20897
20898 | 20898
20899 | 20899
20903 | 20903
20904 | 20904
20905 | 20905
20906 | 20906
20907 | 20907
20908 | 20908
20912 | 20912
20913 | 20913
20914 | 20914
20915 | 20915
20916 | 20916
20921 | 20921
20922 | 20922
20923 | 20923
20924 | 20924
20925 | 20925
20930 | 20930
20931 | 20931
20932 | 20932
20933 | 20933
20934 | 20934
20935 | 20935
20939 | 20939
20940 | 20940
20941 | 20941
20942 | 20942
20943 | 20943
20946 | 20946
20947 | 20947
20948 | 20948
20949 | 20949
20950 | 20950
20955 | 20955
20956 | 20956
20957 | 20957
20958 | 20958
20959 | 20959
20960 | 20960
20964 | 20964
20965 | 20965
20966 | 20966
20967 | 20967
20968 | 20968
20974 | 20974
20975 | 20975
20976 | 20976
20977 | 20977
20978 | 20978
20982 | 20982
20983 | 20983
20984 | 20984
20985 | 20985
20986 | 20986
20987 | 20987
20993 | 20993
20994 | 20994
20995 | 20995
20998 |  
21052 | 21052
21053 | 21053
21054 | 21054
21055 | 21055
21056 | 21056
21057 | 21057
21058 | 21058
21059 | 21059
21060 | 21060
21061 | 21061
21062 | 21062
21063 | 21063
21064 | 21064
21066 | 21066
21067 | 21067
21068 | 21068
21069 | 21069
21070 | 21070
21074 | 21074
21075 | 21075
21076 | 21076
21077 | 21077
21078 | 21078
21079 | 21079
21081 | 21081
21082 | 21082
21083 | 21083
21084 | 21084
21087 | 21087
21088 | 21088
21089 | 21089
21090 | 21090
21091 | 21091
21092 | 21092
21093 | 21093
21096 | 21096
21097 | 21097
21098 | 21098
21099 | 21099
21100 | 21100
21105 | 21105
21106 | 21106
21107 | 21107
21108 | 21108
21109 | 21109
21110 | 21110
21115 | 21115
21116 | 21116
21117 | 21117
21118 | 21118
21119 | 21119
21123 | 21123
21124 | 21124
21125 | 21125
21126 | 21126
21127 | 21127
21128 | 21128
21132 | 21132
21133 | 21133
21134 | 21134
21135 | 21135
21136 | 21136
21140 | 21140
21141 | 21141
21142 | 21142
21143 | 21143
21144 | 21144
21147 | 21147
21148 | 21148
21149 | 21149
21150 | 21150
21151 | 21151
21157 | 21157
21158 | 21158
21159 | 21159
21160 | 21160
21161 | 21161
21162 | 21162
21166 | 21166
21167 | 21167
21168 | 21168
21169 | 21169
21170 | 21170
21174 | 21174
21175 | 21175
21176 | 21176
21177 | 21177
21178 | 21178
21179 | 21179
21183 | 21183
21184 | 21184
21185 | 21185
21186 | 21186
21187 | 21187
21191 | 21191
21192 | 21192
21193 | 21193
21194 | 21194
21195 | 21195
21196 | 21196
21200 | 21200
21201 | 21201
21202 | 21202
21203 | 21203
21204 | 21204
21208 | 21208
21209 | 21209
21210 | 21210
21211 | 21211
21212 | 21212
21213 | 21213
21218 | 21218
21219 | 21219
21220 | 21220
21221 | 21221
21222 | 21222
21226 | 21226
21227 | 21227
21228 | 21228
21229 | 21229
21230 | 21230
21236 | 21236
21237 | 21237
21238 | 21238
21239 | 21239
21240 | 21240
21241 | 21241
21245 | 21245
21246 | 21246
21253 | 21253
21254 | 21254
21255 | 21255
21256 | 21256
21257 | 21257
21258 | 21258
21262 | 21262
21263 | 21263
21264 | 21264
21265 | 21265
21269 | 21269
21270 | 21270
21271 | 21271
21273 | 21273
21279 | 21279
21280 | 21280
21281 | 21281
21282 | 21282
21287 | 21287
21288 | 21288
21289 | 21289
21290 | 21290
21291 | 21291
21295 | 21295
21296 | 21296
21297 | 21297
21303 | 21303
21304 | 21304
21305 | 21305
21306 | 21306
21307 | 21307
21310 | 21310
21311 | 21311
21316 | 21316
21317 | 21317
21318 | 21318
21323 | 21323
21324 | 21324
21325 | 21325
21326 | 21326
21327 | 21327
21332 | 21332
21333 | 21333
21334 | 21334
21340 | 21340
21341 | 21341
21343 | 21343
21344 | 21344
21345 | 21345
21346 | 21346
21350 | 21350
21351 | 21351
21352 | 21352
21357 | 21357
21358 | 21358
21359 | 21359
21360 | 21360
21365 | 21365
21366 | 21366
21367 | 21367
21368 | 21368
21375 | 21375
21376 | 21376
21377 | 21377
21378 | 21378
21383 | 21383
21384 | 21384
21385 | 21385
21386 | 21386
21387 | 21387
21395 | 21395
21396 | 21396
21397 | 21397
21398 | 21398
21399 | 21399
21400 | 21400
21405 | 21405
21406 | 21406
21407 | 21407
21408 | 21408
21409 | 21409
21415 | 21415
21416 | 21416
21417 | 21417
21418 | 21418
21419 | 21419
21425 | 21425
21426 | 21426
21427 | 21427
21428 | 21428
21429 | 21429
21430 | 21430
21435 | 21435
21436 | 21436
21437 | 21437
21442 | 21442
21443 | 21443
21444 | 21444
21445 | 21445
21446 | 21446
21452 | 21452

[kazunarikudo]

Hi again,

As far as I know, this kind of issue can be triggered by a USB host controller even if it has the root cause or not. Do you have another USB3 interface card which has a USB3 host controller from another manufacturer?

If a USB3 host controller was buggy then it would be able to corrupt the behavior of both/either of the GenTL Producer or the U3V camera. They, a GenTL Producer or a camera, might be able to automatically recover from such a state but it'd obviously be a specialized implementation for the host controller.

Perhaps updating the driver software of the USB host controller could resolve this but not really sure if it works. It's just one of general troubleshooting.

Best regards,
Kazunari

[ahirner]

I'll have some updates by the end of the week which should close the issue.

[kazunarikudo]

Okay, I'll be waiting for the update. Thanks.

[ahirner]

A discussion with Basler's support team yielded two suggestions and one explanation for the non-recoverability of the error. The second suggestion is sufficient to have the camera run stable up to a few fps and thus resolves the issue.

Suggestion 1: Set MaxTransferSize as high as possible, 4.19 MB for the given model

camera.GetStreamGrabberParams().MaxTransferSize.SetValue(4194304);
int64_t i = camera.GetStreamGrabberParams().MaxTransferSize.GetValue();

MaxTransferSize ought to be a GeniCam node too, but _genapi said it's not. Is there another way to set StreamGrabberParams?

Suggestion 2: Set DeviceLinkThroughputLimit to the lowest possible value

camera.DeviceLinkSelector.SetValue(0);
camera.DeviceLinkThroughputLimit.SetValue(8000000);
int64_t i = camera.DeviceLinkThroughputLimit.GetValue();

For our case, 32000000 proved to be stable yet fast enough.

Explanation for non-recoverability of underruns
According to an in-depth article, Dart cameras need more than one buffer to recover. One for "pixel data discarded" but the next should be OK. I think that should be the case for harvesters on default. Maybe the rigorous treatment of buffer underruns can lead to improved error handling or a best-practice to handle such errors in harvesters. And I'll ask if I can share the document.

Before actually testing the long-term effect of these suggestions we switched to an acA1920-150uc. This one runs stable off the bat and is a pleasure to use through harvesters and our docker setup.

[kazunarikudo]

Hi Alexander,

Thank you for the update.

MaxTransferSize ought to be a GeniCam node too, but _genapi said it's not. Is there another way to set StreamGrabberParams?

I would like to know who is holding StreamGrabberParams (and even MaxTransferSize; however, I infer MaxTransferSize is a GenICam feature node but not sure for StreamGrabberParams because it looks like a kind of category which holds some parameters). If they are GenICam feature nodes that the camera holds, then you should be able to manipulate them. If the genicam module says they can't be found then it would just mean they are not in its XML file.

One concern that I have is they might be parameters that another entity holds such as a driver. I don't mind who holds them but if they are GenICam feature nodes of any of the modules that are defined by the GenTL standard.

Could you ask Basler people if those parameters can be manipulated through their GenTL Producer? I will take my time (yeah, maybe German people would say "gerne" but I don't know how to say in English) to make it with Harvester if they are avaialbe through the GenTL Producer.

Regards,
Kazunari

[Rahul828086]

ia = h.create_image_acquirer(0)
Traceback (most recent call last):
File "<pyshell#29>", line 1, in
ia = h.create_image_acquirer(0)
File "C:\Program Files (x86)\python64\lib\site-packages\harvesters\core.py", line 2376, in create_image_acquirer
device.open(_privilege)
File "C:\Program Files (x86)\python64\lib\site-packages\genicam2\gentl.py", line 3165, in open
return _gentl.Device_open(self, accessFlags)
_gentl.AccessDeniedException: GenTL exception: Requested operation is not allowed. (Message from the source: Open device with exclusive access failed.) (ID: -1005)

using baumer camera and gentlproducer from baumer "bgapi2_gige.cti ". I get above error

[kazunarikudo]

Hi Rahul828086. Are you sure that any other program is not holding the target camera? Please make a try terminating other image acquisition programs. -Kazunari

[Rahul828086]

@kazunarikudo , ya there is no other software using the camera.

[kazunarikudo]

Too bad. Hm, the error is telling an obvious fact that the device can't be controlled due to a reason. It typically happens if a device is controlled by another process. (1) Can you open the device using Bumer's viwer application? (2) Have you ever tried another GenTL Producer or (3) a camera so far?

[Rahul828086]

@kazunarikudo , Yes i m able to open camera using baumer GAPI application. I haven't tried with other GenTL Producer.. I m not able to get the .CTI file which is mentioned in document in harvesters. but with baumer .CTI file i got return value for h.device_info_list.. but when i go for ia =h.create_image_acquirer(0) it gives above mentioned error..

[kazunarikudo]

Okay, so you mean their software works with the camera but Harvester can't control the camera even though it uses their GenTL Producer, right? Are you really sure that no other applications is running beside Harvester?

[Rahul828086]

@kazunarikudo Yes.. I did close baumer application and there was no application open which uses the camera.

[kazunarikudo]

Which model are you trying to control? Is that from Baumer, too?

[Rahul828086]

@kazunarikudo , ya.. baumer model itself

[kazunarikudo]

Hmm... I can swear that I have ever tried USB3 version but never tried GEV version. To be honest I have no idea on your current case but... can Baumer software control the camera immediately after Harvetser failed to control the camera?

[Rahul828086]

yes

[dudeofea]

@kazunarikudo

I would like to know who is holding StreamGrabberParams (and even MaxTransferSize; however, I infer MaxTransferSize is a GenICam feature node but not sure for StreamGrabberParams because it looks like a kind of category which holds some parameters). If they are GenICam feature nodes that the camera holds, then you should be able to manipulate them. If the genicam module says they can't be found then it would just mean they are not in its XML file.

You can find this parameter under the Data Stream Module's nodemap (all modules have nodemaps). In most uses of the GenTL spec, we only need to use the remote device's nodemap, which we get by first getting a GenTL::PORT_HANDLE using DevGetPort(). But you can also pass in other handles to get the nodemaps of other modules (TL_HANDLE, DS_HANDLE, etc).

If you get a nodemap using the DS_HANDLE in the same manner as a remote device's nodemap (ie using GCGetNumPortURLs(), GCGetPortURLInfo(), GCGetPortInfo(), and GCReadPort()) you will see that MaxTransferSize is there.

This is mentioned throughout the documentation subtly but more specifically in GenTL spec v1.5, in sections 2.3, 4.1.1, and 7.1