docs: add guide for AHB zero-copy input on Android#6733
Merged
nihui merged 3 commits intoMay 25, 2026
Conversation
Address Tencent#5531 - the AHB import API (VkAndroidHardwareBufferImageAllocator, ImportAndroidHardwareBufferPipeline) has no user-facing docs today and existing users hit linker errors against the official prebuilt because build-android.cmd targets -DANDROID_PLATFORM=android-21 which strips the AHB symbols. The new page covers build prerequisites (__ANDROID_API__ >= 26), AImageReader_newWithUsage setup, the per-frame call sequence, per-AHB- pointer pipeline caching (~5000x setup-cost reduction), and trouble- shooting for Tencent#5531 / Tencent#5190.
Member
|
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds user-facing documentation for importing Android AHardwareBuffer camera/surface frames into ncnn’s Vulkan path as VkImageMat to enable zero-copy input on Android (addressing #5531).
Changes:
- Added a new how-to page explaining AHB import prerequisites (notably
__ANDROID_API__ >= 26) and the requiredAImageReader_newWithUsageflags. - Documented an end-to-end AHB →
VkImageMat→VkMatimport flow with troubleshooting and caveats. - Included guidance on caching
ImportAndroidHardwareBufferPipelineto reduce per-frame setup overhead.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ### troubleshooting | ||
|
|
||
| - `ld.lld: undefined symbol: ncnn::VkAndroidHardwareBufferImageAllocator::...` → ncnn was built with `__ANDROID_API__ < 26`. Rebuild ncnn yourself with `-DANDROID_PLATFORM=android-26` or higher; do not rely on the official prebuilt. Symbol-level diagnosis is in [#5531](https://github.com/Tencent/ncnn/issues/5531). | ||
| - `VkImageMat::from_android_hardware_buffer` returns an empty mat → check that `AImageReader_newWithUsage` was used (not `AImageReader_new`) and the AHB descriptor actually carries `GPU_SAMPLED_IMAGE` (`desc.usage & 0x100`). [#5190](https://github.com/Tencent/ncnn/issues/5190) is the prior `elemsize`-related bug — fixed in ncnn since. |
| opt); | ||
|
|
||
| ncnn::VkMat dst; | ||
| dst.create(width, height, /*c=*/3, /*elemsize=*/4u, /*elempack=*/1, opt.blob_vkallocator); |
Comment on lines
+79
to
+81
|
|
||
| **Per-frame `pipe.create` is expensive (~24 ms median on Adreno 830).** The pipeline depends only on the immutable sampler (= the `samplerYcbcrConversion` derived from the AHB's `externalFormat`) and the `(type_to, rotate_from, target_w, target_h)` specialisation constants, all of which are stable for a given camera session. `AImageReader` cycles through a fixed buffer pool (size = `maxImages`), so caching the allocator + pipeline by AHB pointer keeps the cache small (≤ `maxImages` entries) and brings the steady-state setup cost from 24 ms to a few microseconds: | ||
|
|
…bound Two findings from a follow-up spike on Adreno 830: - ex.input(VkMat) does NOT auto-convert format. The previous draft's hint "// ex.input(\"data\", dst, cmd);" actually crashes if the network was loaded with use_fp16_storage / use_fp16_packed (common on Vulkan path) - NetPrivate::forward_layer recurses on itself looking for an implicit converter that doesn't exist, observed as a ~180-frame stack overflow at net.cpp:251. Add a dedicated caveat explaining the failure mode and the vkdev->convert_packing workaround, and weaken the snippet hint accordingly. - AImageReader's AHB pool isn't strictly bounded by maxImages. Under steady state it stays at maxImages (3 -> 3 cache_n in 36 consecutive cycles), but surface/focus events can transiently push it higher (observed cache_n=7 with maxImages=3 in one run). Soften the "<= maxImages entries" claim.
Same import path verified end-to-end on: - Adreno 830, Vulkan 1.3.284 (Galaxy S25 Ultra) - Mali-G925 Immortalis MC12, Vulkan 1.3.278 (Galaxy Tab S11) Both report VK_ANDROID_external_memory_android_hardware_buffer spec version 5 and surface AHARDWAREBUFFER_FORMAT_Y8Cb8Cr8_420 AHBs from the camera HAL. Only auxiliary usage bits differ between vendors and are not consumed by this path (Adreno: 0x20103, Mali: 0x20133).
Member
|
@securekim I noticed you're still updating the documentation. Please ping me again once everything is finalized and ready for review :) |
Member
|
Thanks for your contribution ! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #5531.
The AHB import API (
VkAndroidHardwareBufferImageAllocator,VkImageMat::from_android_hardware_buffer,ImportAndroidHardwareBufferPipeline) has no user-facing documentation today and noexamples/entry. The diagnostic context and per-frame measurement data are in the linked issue — short version: users who try the headers against the official prebuilt archive getld.lld: undefined symbol: ...becausebuild-android.cmdbuilds with-DANDROID_PLATFORM=android-21, which compiles out every AHB symbol fromlibncnn.a.This PR adds a single how-to page (
docs/how-to-use-and-FAQ/use-ncnn-with-android-hardware-buffer.md) matching the existingFAQ-ncnn-vulkan.md/vulkan-notes.mdstyle. It covers:__ANDROID_API__ >= 26requirement that the official prebuilt currently doesn't meet)AImageReader_newWithUsagewithGPU_SAMPLED_IMAGE(AImageReader_newdoes not include that flag → AHB is not Vulkan-importable)AHardwareBuffertoVkMatAHardwareBuffer_acquirelifecycle, read-only sampling, the*255scale inconvert_ycbcr.compNo library or build-system changes. Tested code path on Galaxy S25 Ultra (Adreno 830, Vulkan 1.3.284, NDK r29, ncnn tag 20260113 rebuilt with
-DANDROID_PLATFORM=android-26).Happy to follow up with the
examples/cpp entry as a separate PR if there's interest — held back here becauseexamples/CMakeLists.txthas no precedent for Android-only sources and I didn't want to bundle build-system changes with a docs PR.