Multi-threading fix, link IDs were reused in certain cases #73
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
/lx/XLink/src/shared/XLinkDispatcher.c: In function ‘eventSchedulerRun’:
/lx/XLink/src/shared/XLinkDispatcher.c:775:78: warning: ‘%.2d’ directive output may be truncated writing between 2 and 11 bytes into a region of size 7 [-Wformat-truncation=]
775 | snprintf(eventReaderThreadName, sizeof(eventReaderThreadName), "EventRead%.2dThr", schedulerId);
| ^~~~
/lx/XLink/src/shared/XLinkDispatcher.c:775:68: note: directive argument in the range [-2147483648, 63]
775 | snprintf(eventReaderThreadName, sizeof(eventReaderThreadName), "EventRead%.2dThr", schedulerId);
| ^~~~~~~~~~~~~~~~~~
/lx/XLink/src/shared/XLinkDispatcher.c:775:5: note: ‘snprintf’ output between 15 and 24 bytes into a destination of size 16
775 | snprintf(eventReaderThreadName, sizeof(eventReaderThreadName), "EventRead%.2dThr", schedulerId);
…threading failures due to reusing of previous linkId number (even if properly closed, some transfers were already queued)
alex-luxonis
added a commit
to luxonis/depthai-core
that referenced
this pull request
Nov 7, 2023
themarpe
approved these changes
Nov 7, 2023
This was referenced Dec 13, 2023
Serafadam
pushed a commit
to luxonis/depthai-core
that referenced
this pull request
Mar 6, 2024
* FW: Stereo: handle queue blocking settings * [FW] Updated for some devices and ToF * Added filter by device name * [FW] WIP for S2 PoE boards * [FW] WIP for new SoMs * Fixed build * [FW] Fixed camera orientation * Add example for read calibration in script node * Bump version to v2.22.0 * Add to cmake and fix errors * FW: update IMX296 tuning * [FW] Fixed OAK-D-SR and OV9782 issues * Change printf to node.info in script node read calibration example * ToF: Add median filter support * Update FW to latest develop * FW: Stereo: Apply alpha scaling parameter to RGB intrinsics when RGB alignment is enabled * FW: fix CAM_D enumeration on OAK-FFC-4P R7 * Camera: move alpha parameter as optional to be consistent with stereo * Update FW * BMI: Fix accumulating latency due to slow xlink read * Fix constructors overriding maxUsbSpeed * hasAutofocus fix based on device name/lensPosition * WIP: Device name improvements * Added improvements to device naming * Fixed device related logging * Moved ts, tsDevice, and sequenceNum from descendants to RawBuffer * Added getters and setters * z_map for tof * ts to tsDevice fix * Updated ColorCamera's setSensorCrop comment for documentation * Removed redundant getters and setters * FW: add support for OAK-D-SR-PoE R1M1E1 * Fixed mistake in ImgFrame * Ran ClangFormat * Added DEPTHAI_ENABLE_LIBUSB option, to enable/disable USB protocol in build time * Applied style * Updated XLink library with some fixes * FW: fix default fsync on OAK-D-SR-PoE. GPIO46 input by default * Added host side EncodedFrame with parsing and output handling * hasautofocusIC * Bugfixes * FW: fix 4 cams crash on PoE due to memory allocation * Update FW / shared * FW: fix OAK-D-SR camera enum. IMX296 updates: - support for RPi GS Cam paired with Luxonis SL6996 RPi camera adapter - support for external trigger on XTR/XTRIG pin: active low, pulse width determines exposure time. Limitation: needs an initial trigger shortly after start to avoid a crash, or `DEPTHAI_WATCHDOG=0` can be a workaround * Attempt to fix serialization error on windows compiler * Update FW * Implemented timesync optimization * getConnectivity feature * Fixed new timesync bugs * add corespoding SHA * Moved treceive back to XLink * Changes according to PR issue * Clangformat * Bump FW * Moved timestamps to buffer * WIP: Device name improvements * Added improvements to device naming * FIXME: does not work on rpi * Updated FW with changes * Bump shared * Fixed compiler error * Changed struct timespec to custom struct to fix RPI size issues * Bump shared * FW, ColorCamera: add new almost-full-FOV resolutions for IMX378/477, scaled: 1352x1012, 4lane-only, 1/3 scaling 2024x1520, 1/2 binning * EncFrm fixes * Added tests, bugfixes * clangformat * Bump device * Bump device again * Fixed incorrect warp in issue #882 * [FW] OAK-D SR PoE and ToF improvements * [FW] Improved OV9782 fps handling * Updated FW & XLink with backward compatible timestamping * FW: fixes for AR0234 fsync, AR0234 AE max exposure: 16->33 ms, more robust camera handling (especially in sync modes) * fixed the order of translation multiplication * clang update * [FW] deviceName fixes and updated for EepromData changes * Increased num XLink connections to 64 * Bumped num XLink connections to 64 * Implemented metadata serialization * Implemented raw data serialization in MG * FW: OAK-D-SR-PoE R1 fixes for fsync detect and status LED (green when running), StereoDepth fix for RGB-depth alignment with Left-Right-RGB layout * Added fields to EncodedFrame * Bump shared * Implemented sync node * CameraControl: add `setControlMode`, `setCaptureIntent` * Fixed msggrp serialization * Bump shared * Simplified msggrp * Moved frame type extraction into a getter * Bump shared * Fixed python bindings * Bump shared * FW: multiple tunings to fix image quality issues, OV9782 tuning, IMX462 fix for ISO * Implemented messagedemux * Added doc comments * Bump shared * Bump firmware * Bump shared * Bump device * Bump device * clangformat * Some changes WRT Android build * Removed unnecessary functions * Redid messageGroups * Fix multi stereo node race condition when depth output is enabled * Ir brightness control based on normalized intensity, instead of current. * Changed dynamic to static cast * Added option to only demux successful syncs * Added ability to set success on msggrp * Clangformat * Added IR intensity API * Applied suggestions * Fix XLink linkId reuse, caused multi-threading failures. See: luxonis/XLink#73 * Bump device * Implemented automatic crash dump retrieval * Fixed auto crash report for PoE * Bump device * Fix warning * Improved Monitor&WD handling in DeviceBootloader * Update README.md * Renamed getConnectivity * Update docstring. Point to correct device side commit. * Renamed getAvailableInterfaces * Use a new devicebase to connect to device * Clangformat * Deprecated ir brightness api, fixed spatial_calculator_multi_roi example. * Attempt to fix macos pipeline build * Bump shared * Added tests for encframe * Changed the way sync works + added test * Add CameraControl `setAutoExposureLimit`. FW: improve OV9282/OV9782 image quality in low-light / under high ISO (1300+) * Changed the way sync is configured * Clangformat * Added examples + bump fw * Added example, removed numPools from Sync * Minor modification to example * Bump fw * Bump fw * Bump fw * Throw error when indexing non-existing message, changed example to use disparity * Improved example * Clangformat * Bump fw * Bump fw * Bump shared * Changed syncThreshold setter to use chrono duration * Bump shared * Bump fw * Handle shutdown returning false, bump fw * Bump fw * Changed the way crash reports are retrieved * Bump fw * Clangformat * Bump fw * Bump fw * IR Intensity API (#924) * Ir brightness control based on normalized intensity, instead of current. * Added IR intensity API * Update docstring. Point to correct device side commit. * Deprecated ir brightness api, fixed spatial_calculator_multi_roi example. * Update depthai-shared, update FW. * Sync shared with device side * Update shared on device and host * Bump FW to latest develop * Bump FW - bumped shared to latest. * Rename connectivity enum * Update README * Minor encframe test fixes * Bump fw * Added DEPTHAI_CRASHDUMP_TIMEOUT to determine crashdump timeout * TODO fix windows test * Changed crashdump timing * Bump shared * Clangformat * Fix readme * Bump fw * Bump shared * Bump fw and XLink * Bump fw * Add an example with IMU * Add new line at the end of the example * [RVC2 FW] Update FW with updated shared * added docstring for fisheye * Check for crash dump before getting * Update FW with a stability fix for timesync * Revert the XLink profiling to uint64_t change * Update FW and shared to develop * Fix the xlink profiling * Bump version to 2.24.0 * Update changelog and package xml --------- Co-authored-by: SzabolcsGergely <szabi@luxonis.com> Co-authored-by: alex-luxonis <alex@luxonis.com> Co-authored-by: TheMarpe <martin.peterlin7@gmail.com> Co-authored-by: Andrej Susnik <andrej.susnik@luxonis.com> Co-authored-by: szabi-luxonis <60790666+szabi-luxonis@users.noreply.github.com> Co-authored-by: TheMarpe <martin@luxonis.com> Co-authored-by: Aniel Alexa <aniel.alexa@luxonis.com> Co-authored-by: AnielAlexa <139551403+AnielAlexa@users.noreply.github.com> Co-authored-by: asahtik <asahtik@gmail.com> Co-authored-by: Erol444 <erol123444@gmail.com> Co-authored-by: saching13 <sachin.guruswamy@gmail.com> Co-authored-by: zrezke <jeretina.filip@gmail.com> Co-authored-by: asahtik <38485424+asahtik@users.noreply.github.com> Co-authored-by: Filip Jeretina <59307111+zrezke@users.noreply.github.com> Co-authored-by: alex-luxonis <60824841+alex-luxonis@users.noreply.github.com> Co-authored-by: moratom <47612463+moratom@users.noreply.github.com> Co-authored-by: Matevz Morato <matevz.morato@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix
getNextAvailableLinkUniqueIdto always increment the next returned id. In certain cases it returned the previous link id, and even if the connection got closed (mainly by device after FW boot), there might have been queued transfers (like the blocking readback after bootMemory in DepthAI) interfering with the newly open connection on the same id.@diablodale's comment in 5d2d502 also relevant:
Fixes multi-threading failures (observed on PoE devices), and might fix some other bootup issues.
DepthAI code snippet repro:
std::vector<std::string> devices = { "192.168.8.129", "192.168.8.130", "192.168.8.131", "192.168.8.132", "192.168.8.133", }; std::vector<std::thread> threads; for (const auto& devName : devices) { threads.push_back(std::thread([devName](){ printf("===== %s: thread started\n", devName.c_str()); auto connected = dai::XLinkConnection::getAllConnectedDevices(); dai::DeviceInfo selected; for (const auto& devInfo : connected) { if (devInfo.name == devName) { selected = devInfo; break; } } // scoped, to close bootloader { dai::DeviceBootloader bl(selected); auto ver = bl.getVersion().toString(); printf("===== %s: bootloader version %s\n", devName.c_str(), ver.c_str()); } dai::Device dev(selected); printf("===== %s: device started\n", devName.c_str()); while(1) sleep(1); })); // usleep(100*1000); // some wait between starting threads } while(1) sleep(1);Some local logs (depthai-core, XLink), showing one of the failures. Note the reuse of
linkId 4for two devices, successfully booting192.168.8.130, then failing192.168.8.129:Logs after the fix: