Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arm Release v1.3.0 #24

Merged
1 commit merged into from
Feb 20, 2020
Merged

Arm Release v1.3.0 #24

1 commit merged into from
Feb 20, 2020

Conversation

ZandroFargnoli
Copy link
Contributor

@ZandroFargnoli ZandroFargnoli commented Dec 16, 2019

Description

Read the full release article for a more complete description of the PR.

Please read the contribution guidelines

Checklist:

Do not submit your PR without all of the below being checked:

  • My code follows the coding style
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have tested my sample on at least one compliant Vulkan implementation
  • I have stated on what implementation the sample has been tested so that others can test on different implementations and platforms
  • If my sample is vendor-specific, I have tagged it appropriately
  • My changes do not add any new compiler warnings
  • Vulkan validation layer output is clean on at least one compliant implementation
  • Any dependent changes (e.g. assets) have been merged and published in downstream modules
  • I have used existing framework/helper functions where possible
  • I have reviewed file licenses
  • My changes build and run on Windows, Linux, macOS and Android. Otherwise I have documented any exceptions

docs/build.md Show resolved Hide resolved
docs/faq.md Outdated Show resolved Hide resolved
docs/memory_limits.md Outdated Show resolved Hide resolved
samples/performance/wait_idle/wait_idle_tutorial.md Outdated Show resolved Hide resolved
samples/performance/wait_idle/wait_idle_tutorial.md Outdated Show resolved Hide resolved
@MichaelParkinWhite
Copy link
Contributor

MichaelParkinWhite commented Jan 6, 2020

Getting a crash during shader compilation in the Specialization Constants sample due to lack of versioning information:

Revision: '26'
01-06 18:07:37.288 18528 18528 F DEBUG   : ABI: 'arm64'
01-06 18:07:37.288 18528 18528 F DEBUG   : pid: 18298, tid: 18490, name: .vulkan_samples  >>> com.khronos.vulkan_samples <<<
01-06 18:07:37.288 18528 18528 F DEBUG   : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
01-06 18:07:37.288 18528 18528 F DEBUG   : Abort message: 'terminating with uncaught exception of type vkb::VulkanException: Shader compilation failed:
01-06 18:07:37.288 18528 18528 F DEBUG   : ERROR: #version: ES shaders for Vulkan SPIR-V require version 310 or higher
01-06 18:07:37.288 18528 18528 F DEBUG   : ERROR: 1 compilation errors.  No code generated.
01-06 18:07:37.288 18528 18528 F DEBUG   :
01-06 18:07:37.288 18528 18528 F DEBUG   :
01-06 18:07:37.288 18528 18528 F DEBUG   :  : ERROR_INITIALIZATION_FAILED'
01-06 18:07:37.288 18528 18528 F DEBUG   :     x0  0000000000000000  x1  000000000000483a  x2  0000000000000006  x3  0000000000000008
01-06 18:07:37.288 18528 18528 F DEBUG   :     x4  fefeff797b54b66f  x5  fefeff797b54b66f  x6  fefeff797b54b66f  x7  7f7f7f7f7f7fff7f
01-06 18:07:37.288 18528 18528 F DEBUG   :     x8  0000000000000083  x9  0000007b1d5771d0  x10 0000000000000000  x11 fffffffc7ffffbdf
01-06 18:07:37.288 18528 18528 F DEBUG   :     x12 0000000000000001  x13 000000005e137769  x14 000d1cee5c4c4900  x15 00003c18c3ff4762
01-06 18:07:37.288 18528 18528 F DEBUG   :     x16 0000007b1d5af298  x17 0000007b1d4ee67c  x18 0000000000000000  x19 000000000000477a
01-06 18:07:37.288 18528 18528 F DEBUG   :     x20 000000000000483a  x21 0000007a7c55b9a8  x22 ffffff80ffffffc8  x23 0000007a7c55ba60
01-06 18:07:37.288 18528 18528 F DEBUG   :     x24 0000007a7c55b940  x25 0000007a7c55b980  x26 0000007b217e15f8  x27 0000000000000058
01-06 18:07:37.288 18528 18528 F DEBUG   :     x28 000000000000001c  x29 0000007a7c55b8b0
01-06 18:07:37.288 18528 18528 F DEBUG   :     sp  0000007a7c55b870  lr  0000007b1d4e1ed8  pc  0000007b1d4e1f00
01-06 18:07:37.294 18528 18528 F DEBUG   :
01-06 18:07:37.294 18528 18528 F DEBUG   : backtrace:
01-06 18:07:37.294 18528 18528 F DEBUG   :     #00 pc 0000000000021f00  /system/lib64/libc.so (abort+116)
01-06 18:07:37.294 18528 18528 F DEBUG   :     #01 pc 0000000000e6d29c  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so
01-06 18:07:37.294 18528 18528 F DEBUG   :     #02 pc 0000000000e6d3f4  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so
01-06 18:07:37.294 18528 18528 F DEBUG   :     #03 pc 0000000000e6a430  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so
01-06 18:07:37.294 18528 18528 F DEBUG   :     #04 pc 0000000000e69b34  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so
01-06 18:07:37.294 18528 18528 F DEBUG   :     #05 pc 0000000000e69ab4  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so (__cxa_throw+120)
01-06 18:07:37.294 18528 18528 F DEBUG   :     #06 pc 0000000000792c3c  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so (vkb::ShaderModule::ShaderModule(vkb::Device&, VkShaderStageFlagBits, vkb::ShaderSource const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&, vkb::ShaderVariant const&)+836)
01-06 18:07:37.294 18528 18528 F DEBUG   :     #07 pc 00000000006f3f48  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so (_ZN3vkb16request_resourceINS_12ShaderModuleEJ21VkShaderStageFlagBitsKNS_12ShaderSourceENSt6__ndk112basic_stringIcNS5_11char_traitsIcEENS5_9allocatorIcEEEEKNS_13ShaderVariantEEEERT_RNS_6DeviceEPNS_14ResourceRecordERNS5_13unordered_mapImSE_NS5_4hashImEENS5_8equal_toImEENS9_INS5_4pairIKmSE_EEEEEEDpRT0_+244)
01-06 18:07:37.294 18528 18528 F DEBUG   :     #08 pc 00000000006eb30c  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so
01-06 18:07:37.294 18528 18528 F DEBUG   :     #09 pc 00000000006eb248  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so (vkb::ResourceCache::request_shader_module(VkShaderStageFlagBits, vkb::ShaderSource const&, vkb::ShaderVariant const&)+108)
01-06 18:07:37.294 18528 18528 F DEBUG   :     #10 pc 0000000000632758  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so (SpecializationConstants::ForwardSubpassCustomLights::prepare()+600)
01-06 18:07:37.294 18528 18528 F DEBUG   :     #11 pc 0000000000800084  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so (vkb::RenderPipeline::RenderPipeline(std::__ndk1::vector<std::__ndk1::unique_ptr<vkb::Subpass, std::__ndk1::default_delete<vkb::Subpass>>, std::__ndk1::allocator<std::__ndk1::unique_ptr<vkb::Subpass, std::__ndk1::default_delete<vkb::Subpass>>>>&&)+276)
01-06 18:07:37.294 18528 18528 F DEBUG   :     #12 pc 0000000000614580  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so
01-06 18:07:37.294 18528 18528 F DEBUG   :     #13 pc 0000000000632014  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so (SpecializationConstants::create_specialization_renderpass()+384)
01-06 18:07:37.294 18528 18528 F DEBUG   :     #14 pc 0000000000631d10  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so (SpecializationConstants::prepare(vkb::Platform&)+400)
01-06 18:07:37.294 18528 18528 F DEBUG   :     #15 pc 000000000066d364  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so (vkb::VulkanSamples::prepare_active_app(std::__ndk1::function<std::__ndk1::unique_ptr<vkb::Application, std::__ndk1::default_delete<vkb::Application>> ()>, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&, bool, bool)+656)
01-06 18:07:37.294 18528 18528 F DEBUG   :     #16 pc 000000000066c46c  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so (vkb::VulkanSamples::prepare(vkb::Platform&)+1644)
01-06 18:07:37.294 18528 18528 F DEBUG   :     #17 pc 000000000067836c  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so (vkb::Platform::prepare()+76)
01-06 18:07:37.294 18528 18528 F DEBUG   :     #18 pc 0000000000879038  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so
01-06 18:07:37.294 18528 18528 F DEBUG   :     #19 pc 0000000000e23a5c  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so
01-06 18:07:37.295 18528 18528 F DEBUG   :     #20 pc 000000000087981c  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so (vkb::AndroidPlatform::main_loop()+108)
01-06 18:07:37.295 18528 18528 F DEBUG   :     #21 pc 0000000000663f70  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so (android_main+220)
01-06 18:07:37.295 18528 18528 F DEBUG   :     #22 pc 0000000000e239e0  /data/app/com.khronos.vulkan_samples-gsngiDxJof4B-G80hdKUUA==/lib/arm64/libvulkan_samples.so
01-06 18:07:37.295 18528 18528 F DEBUG   :     #23 pc 0000000000084108  /system/lib64/libc.so (__pthread_start(void*)+64)
01-06 18:07:37.295 18528 18528 F DEBUG   :     #24 pc 0000000000023ae8  /system/lib64/libc.so (__start_thread+68)

Adding version qualifier #version 320 es (as per other project shaders) to shaders\specialization_constants\specialization_constant.frag and shaders\specialization_constants\UBOs.frag
resolves the issue.

wasimabbas-arm
wasimabbas-arm previously approved these changes Jan 8, 2020
@gary-sweet
Copy link
Contributor

Broadcom are seeing issues with a number of the samples with this PR in place. I'll need some time to investigate.

These samples are just showing a black screen with the UI on top:
afbc, swapchain_images, surface_rotation, wait_idle

I get a std::bad_alloc from pipeline_cache

command_buffer_usage now somehow manages to provoke a GPU lock-up!

I also see lots of warnings about "Render target extent is not an optimal size"

@ZandroFargnoli
Copy link
Contributor Author

These samples are just showing a black screen with the UI on top:
afbc, swapchain_images, surface_rotation, wait_idle

Interesting. I have tested this branch on an S10, S9, S9 Plus, S8 Plus, and a Huawei Mate 20 Pro and all these samples are working. Are you receiving any errors with validation layers turned on? Does a clean reinstall fix the problem?

@gary-sweet
Copy link
Contributor

Interesting. I have tested this branch on an S10, S9, S9 Plus, S8 Plus, and a Huawei Mate 20 Pro and all these samples are working. Are you receiving any errors with validation layers turned on? Does a clean reinstall fix the problem?

No validation errors that I can see. Clean install has the same problem.
I'm still investigating. Visually it looks exactly like the problems we had previously relating to the defines for glm.

@MichaelParkinWhite
Copy link
Contributor

MichaelParkinWhite commented Jan 9, 2020

Also getting the Bad alloc error in the Pipeline management sample on S10 Adreno:
Sample log:

01-10 00:00:56.917 15451 15493 E VulkanSamples: [error] [eplay command not supported.:87] Replay command not supported.
01-10 00:00:56.926 15451 15493 E VulkanSamples: [error] [eplay command not supported.:87] Replay command not supported.

Stacktrace: PiplineManagement_BadAlloc_S10_Adreno.txt

EDIT: As per Gary's suggestions, this is resolved by deleting the cache.data and pipeline_cache.data files.

@MichaelParkinWhite
Copy link
Contributor

multiple_pipeline_management_entries

The pipeline management sample also appears to list itself multiple times.

@gary-sweet
Copy link
Contributor

Also getting the Bad alloc error in the Pipeline management sample on S10 Adreno:

This went away for me when I deleted the /tmp/cache.data and /tmp/pipeline_cache.data files that the sample reads.

@MichaelParkinWhite
Copy link
Contributor

This went away for me when I deleted the /tmp/cache.data and /tmp/pipeline_cache.data files that the sample reads.

So it does, thanks! This folder is located in /data/data/com.khronos.vulkan_samples/cache on Android. Deleting the files does indeed help. Perhaps there is a versioning issue? Wondering if this will become a common problem when infrastructure changes are made, and if a check is needed to ensure the cache matches the current version of the APK?

@gary-sweet
Copy link
Contributor

Wondering if this will become a common problem when infrastructure changes are made, and if a check is needed to ensure the cache matches the current version of the APK?

Indeed. I agree that some version checking is probably needed.

@ZandroFargnoli
Copy link
Contributor Author

multiple_pipeline_management_entries

The pipeline management sample also appears to list itself multiple times.

This is my fault, Git seemed to do some funky merging. pipeline_cache appeared twice in the CMake ORDER_LIST of samples, causing it to be inserted twice. The reason it was showing four times (twice at each order placement), is that the category sample list is compared with the filter sample list, which each had two entries thanks to our ORDER_LIST!

I have removed the duplicate in the ORDER_LIST, and CMake now ensures subsequent sample IDs are ignored, so the first instance decides where the sample is placed in the list (to avoid this happening in the future).

@MichaelParkinWhite
Copy link
Contributor

These samples are just showing a black screen with the UI on top:
afbc, swapchain_images, surface_rotation, wait_idle

This issue is also affecting the Samsung Galaxy S8 Adreno (G950U). I am currently getting a black screen with UI on top in the following samples:

  • Swapchain Images
  • Surface Rotation
  • Pipeline Management
  • Render Passes
  • Wait Idle
  • Command Buffer Usage;

Also getting a crash inside the core validation layer when switching options in samples which re-create the swapchain (Swapchain Images, Surface Rotation):

 backtrace:
01-09 16:28:08.993 17574 17574 F DEBUG   :     #00 pc 00000000000e6810  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libVkLayer_core_validation.so
01-09 16:28:08.993 17574 17574 F DEBUG   :     #01 pc 00000000007c7a6c  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb9Swapchain18acquire_next_imageERjP13VkSemaphore_TP9VkFence_T+124)
01-09 16:28:08.993 17574 17574 F DEBUG   :     #02 pc 00000000007f0aa0  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb13RenderContext11begin_frameEv+308)
01-09 16:28:08.993 17574 17574 F DEBUG   :     #03 pc 00000000007f08b0  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb13RenderContext5beginENS_13CommandBuffer9ResetModeE+148)
01-09 16:28:08.993 17574 17574 F DEBUG   :     #04 pc 0000000000735fc8  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb12VulkanSample6updateEf+112)
01-09 16:28:08.993 17574 17574 F DEBUG   :     #05 pc 00000000005fad08  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN15SurfaceRotation6updateEf+500)
01-09 16:28:08.993 17574 17574 F DEBUG   :     #06 pc 0000000000672618  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb11Application4stepEv+152)
01-09 16:28:08.993 17574 17574 F DEBUG   :     #07 pc 000000000066db08  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb13VulkanSamples6updateEf+88)
01-09 16:28:08.993 17574 17574 F DEBUG   :     #08 pc 0000000000672618  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb11Application4stepEv+152)
01-09 16:28:08.993 17574 17574 F DEBUG   :     #09 pc 0000000000678544  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb8Platform3runEv+240)
01-09 16:28:08.993 17574 17574 F DEBUG   :     #10 pc 000000000087987c  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb15AndroidPlatform9main_loopEv+204)
01-09 16:28:08.994 17574 17574 F DEBUG   :     #11 pc 0000000000663f70  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (android_main+220)
01-09 16:28:08.994 17574 17574 F DEBUG   :     #12 pc 0000000000e239e0  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so
01-09 16:28:08.994 17574 17574 F DEBUG   :     #13 pc 00000000000770f4  /system/lib64/libc.so (_ZL15__pthread_startPv+204)
01-09 16:28:08.994 17574 17574 F DEBUG   :     #14 pc 000000000001e7d0  /system/lib64/libc.so (__start_thread+16)

@MichaelParkinWhite
Copy link
Contributor

Getting another crash on the S8 Adreno (G950U) in the Render Subpasses,Layout Transitions and Pipeline Barriers samples relating to a VK_INCOMPLETE return flag.

Not sure why this is happening currently, it is apparently an invalid return flag from vkCreateGraphicsPipelines (which should not normally return that flag). This is likely a known driver bug: (https://developer.qualcomm.com/forum/qdn-forums/software/adreno-gpu-sdk/34709)

However, I would ideally like to understand why it is happening -- as these samples worked previously.

 *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
01-09 16:31:25.987 18990 18990 F DEBUG   : Revision: '12'
01-09 16:31:25.987 18990 18990 F DEBUG   : ABI: 'arm64'
01-09 16:31:25.987 18990 18990 F DEBUG   : pid: 17998, tid: 18979, name: .vulkan_samples  >>> com.khronos.vulkan_samples <<<
01-09 16:31:25.987 18990 18990 F DEBUG   : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
01-09 16:31:25.991 18990 18990 F DEBUG   : Abort message: 'terminating with uncaught exception of type vkb::VulkanException: Cannot create GraphicsPipelines : INCOMPLETE'
01-09 16:31:25.992 18990 18990 F DEBUG   :     x0   0000000000000000  x1   0000000000004a23  x2   0000000000000006  x3   0000000000000008
01-09 16:31:25.992 18990 18990 F DEBUG   :     x4   ffffffffffffffff  x5   8080000000000000  x6   0000000000008080  x7   feff4453444b4f4c
01-09 16:31:25.992 18990 18990 F DEBUG   :     x8   0000000000000083  x9   0000007f878f6450  x10  9145bc7c0995566b  x11  9145bc7c0995566b
01-09 16:31:25.992 18990 18990 F DEBUG   :     x12  0000000000000000  x13  0000000000000020  x14  ffffffffffffffdf  x15  0000000000000008
01-09 16:31:25.992 18990 18990 F DEBUG   :     x16  0000007fad04fee0  x17  0000007facff06bc  x18  0000000000000000  x19  0000007f878f64f8
01-09 16:31:25.992 18990 18990 F DEBUG   :     x20  0000000000000006  x21  0000007f878f6450  x22  0000000000000002  x23  0000007f878f4690
01-09 16:31:25.992 18990 18990 F DEBUG   :     x24  0000007f878f4570  x25  0000007f878f45b0  x26  0000007f8c013748  x27  9145bc7c0995566b
01-09 16:31:25.992 18990 18990 F DEBUG   :     x28  0000007f8c013540  x29  0000007f878f4480  x30  0000007facfed924
01-09 16:31:25.992 18990 18990 F DEBUG   :     sp   0000007f878f4460  pc   0000007facff06c4  pstate 0000000060000000
01-09 16:31:26.041 18990 18990 F DEBUG   :
01-09 16:31:26.041 18990 18990 F DEBUG   : backtrace:
01-09 16:31:26.042 18990 18990 F DEBUG   :     #00 pc 000000000007a6c4  /system/lib64/libc.so (tgkill+8)
01-09 16:31:26.042 18990 18990 F DEBUG   :     #01 pc 0000000000077920  /system/lib64/libc.so (pthread_kill+64)
01-09 16:31:26.042 18990 18990 F DEBUG   :     #02 pc 000000000002538c  /system/lib64/libc.so (raise+24)
01-09 16:31:26.042 18990 18990 F DEBUG   :     #03 pc 000000000001d24c  /system/lib64/libc.so (abort+52)
01-09 16:31:26.042 18990 18990 F DEBUG   :     #04 pc 0000000000e6d29c  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so
01-09 16:31:26.042 18990 18990 F DEBUG   :     #05 pc 0000000000e6d3f4  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so
01-09 16:31:26.042 18990 18990 F DEBUG   :     #06 pc 0000000000e6a430  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so
01-09 16:31:26.042 18990 18990 F DEBUG   :     #07 pc 0000000000e69b34  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so
01-09 16:31:26.042 18990 18990 F DEBUG   :     #08 pc 0000000000e69ab4  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (__cxa_throw+120)
01-09 16:31:26.042 18990 18990 F DEBUG   :     #09 pc 00000000007a3678  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb16GraphicsPipelineC1ERNS_6DeviceEP17VkPipelineCache_TRNS_13PipelineStateE+3312)
01-09 16:31:26.042 18990 18990 F DEBUG   :     #10 pc 00000000006fd0c4  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb16request_resourceINS_16GraphicsPipelineEJP17VkPipelineCache_TNS_13PipelineStateEEEERT_RNS_6DeviceEPNS_14ResourceRecordERNSt6__ndk113unordered_mapImS5_NSB_4hashImEENSB_8equal_toImEENSB_9allocatorINSB_4pairIKmS5_EEEEEEDpRT0_+220)
01-09 16:31:26.042 18990 18990 F DEBUG   :     #11 pc 00000000006eb6d4  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so
01-09 16:31:26.042 18990 18990 F DEBUG   :     #12 pc 00000000006eb674  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb13ResourceCache25request_graphics_pipelineERNS_13PipelineStateE+84)
01-09 16:31:26.042 18990 18990 F DEBUG   :     #13 pc 00000000007d0bc0  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb13CommandBuffer20flush_pipeline_stateE19VkPipelineBindPoint+180)
01-09 16:31:26.042 18990 18990 F DEBUG   :     #14 pc 00000000007d0abc  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb13CommandBuffer4drawEjjjj+64)
01-09 16:31:26.042 18990 18990 F DEBUG   :     #15 pc 000000000080fdb4  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb15LightingSubpass4drawERNS_13CommandBufferE+1240)
01-09 16:31:26.042 18990 18990 F DEBUG   :     #16 pc 0000000000800730  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb14RenderPipeline4drawERNS_13CommandBufferERNS_12RenderTargetE17VkSubpassContents+380)
01-09 16:31:26.042 18990 18990 F DEBUG   :     #17 pc 0000000000614788  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_Z13draw_pipelineRN3vkb13CommandBufferERNS_12RenderTargetERNS_14RenderPipelineEPNS_3GuiE+352)
01-09 16:31:26.042 18990 18990 F DEBUG   :     #18 pc 0000000000614890  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN15RenderSubpasses21draw_render_subpassesERN3vkb13CommandBufferERNS0_12RenderTargetE+120)
01-09 16:31:26.042 18990 18990 F DEBUG   :     #19 pc 0000000000614be0  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN15RenderSubpasses15draw_renderpassERN3vkb13CommandBufferERNS0_12RenderTargetE+80)
01-09 16:31:26.043 18990 18990 F DEBUG   :     #20 pc 0000000000736228  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb12VulkanSample4drawERNS_13CommandBufferERNS_12RenderTargetE+468)
01-09 16:31:26.043 18990 18990 F DEBUG   :     #21 pc 0000000000736020  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb12VulkanSample6updateEf+200)
01-09 16:31:26.043 18990 18990 F DEBUG   :     #22 pc 0000000000613a00  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN15RenderSubpasses6updateEf+800)
01-09 16:31:26.043 18990 18990 F DEBUG   :     #23 pc 0000000000672618  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb11Application4stepEv+152)
01-09 16:31:26.043 18990 18990 F DEBUG   :     #24 pc 000000000066db08  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb13VulkanSamples6updateEf+88)
01-09 16:31:26.043 18990 18990 F DEBUG   :     #25 pc 0000000000672618  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb11Application4stepEv+152)
01-09 16:31:26.043 18990 18990 F DEBUG   :     #26 pc 0000000000678544  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb8Platform3runEv+240)
01-09 16:31:26.043 18990 18990 F DEBUG   :     #27 pc 000000000087987c  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (_ZN3vkb15AndroidPlatform9main_loopEv+204)
01-09 16:31:26.043 18990 18990 F DEBUG   :     #28 pc 0000000000663f70  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so (android_main+220)
01-09 16:31:26.043 18990 18990 F DEBUG   :     #29 pc 0000000000e239e0  /data/app/com.khronos.vulkan_samples-1/lib/arm64/libvulkan_samples.so
01-09 16:31:26.043 18990 18990 F DEBUG   :     #30 pc 00000000000770f4  /system/lib64/libc.so (_ZL15__pthread_startPv+204)
01-09 16:31:26.043 18990 18990 F DEBUG   :     #31 pc 000000000001e7d0  /system/lib64/libc.so (__start_thread+16)

I'll need to investigate this further.
Relevant area in the code:

        auto result = vkCreateGraphicsPipelines(device.get_handle(), pipeline_cache, 1, &create_info, nullptr, &handle);
	if (result != VK_SUCCESS)
	{
		throw VulkanException{result, "Cannot create GraphicsPipelines"};
	}

By coincidence, there is also a validation error relating to the same previously discussed issue with valid format selection. This problem exists in all the samples which crash as above, and only in those samples.

[Validation] Code 165676888 :  [ VUID-VkImageCreateInfo-format-00940 ]  [ VUID-VkImageCreateInfo-format-00940 ] Object: VK_NULL_HANDLE (Type = 0) | vkCreateImage: The combination of format, type, tiling, usage and flags supplied in the VkImageCreateInfo struct is reported by vkGetPhysicalDeviceImageFormatProperties() as unsupported. The spec valid usage text states 'The combination of format, imageType, tiling, usage, and flags must be supported, as indicated by a VK_SUCCESS return value from vkGetPhysicalDeviceImageFormatProperties invoked with the same values passed to the corresponding parameters.' (https://www.khronos.org/registry/vulkan/specs/1.0/html/vkspec.html#VUID-VkImageCreateInfo-format-00940)
01-09 16:54:22.625 23800 23832 E vulkan  : [Validation] Code 180357104 :  [ VUID-VkImageViewCreateInfo-image-01016 ]  [ VUID-VkImageViewCreateInfo-image-01016 ] Object: 0x7f8a762d00 (Type = 10) | vkCreateImageView() pCreateInfo->format VK_FORMAT_D32_SFLOAT cannot be used with an image having the VK_IMAGE_TILING_OPTIMAL and VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT flags set. The spec valid usage text states 'If image was created with VK_IMAGE_TILING_OPTIMAL and usage contains VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT, format must be supported for depth/stencil attachments, as specified by the VK_FORMAT_FEATURE_DEPTH_STENCIL_ATTACHMENT_BIT flag in VkFormatProperties::optimalTilingFeatures returned by vkGetPhysicalDeviceFormatProperties with the same value of format' (https://www.khronos.org/registry/vulkan/specs/1.0/html/vkspec.html#VUID-VkImageViewCreateInfo-image-01016)
01-09 16:54:22.626 23800 23832 E vulkan  : [Validation] Code 165676888 :  [ VUID-VkImageCreateInfo-format-00940 ]  [ VUID-VkImageCreateInfo-format-00940 ] Object: VK_NULL_HANDLE (Type = 0) | vkCreateImage: The combination of format, type, tiling, usage and flags supplied in the VkImageCreateInfo struct is reported by vkGetPhysicalDeviceImageFormatProperties() as unsupported. The spec valid usage text states 'The combination of format, imageType, tiling, usage, and flags must be supported, as indicated by a VK_SUCCESS return value from vkGetPhysicalDeviceImageFormatProperties invoked with the same values passed to the corresponding parameters.' (https://www.khronos.org/registry/vulkan/specs/1.0/html/vkspec.html#VUID-VkImageCreateInfo-format-00940)
01-09 16:54:22.629 23800 23832 E vulkan  : [Validation] Code 180357104 :  [ VUID-VkImageViewCreateInfo-image-01016 ]  [ VUID-VkImageViewCreateInfo-image-01016 ] Object: 0x7f8740d600 (Type = 10) | vkCreateImageView() pCreateInfo->format VK_FORMAT_D32_SFLOAT cannot be used with an image having the VK_IMAGE_TILING_OPTIMAL and VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT flags set. The spec valid usage text states 'If image was created with VK_IMAGE_TILING_OPTIMAL and usage contains VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT, format must be supported for depth/stencil attachments, as specified by the VK_FORMAT_FEATURE_DEPTH_STENCIL_ATTACHMENT_BIT flag in VkFormatProperties::optimalTilingFeatures returned by vkGetPhysicalDeviceFormatProperties with the same value of format' (https://www.khronos.org/registry/vulkan/specs/1.0/html/vkspec.html#VUID-VkImageViewCreateInfo-image-01016)
01-09 16:54:22.630 23800 23832 E vulkan  : [Validation] Code 165676888 :  [ VUID-VkImageCreateInfo-format-00940 ]  [ VUID-VkImageCreateInfo-format-00940 ] Object: VK_NULL_HANDLE (Type = 0) | vkCreateImage: The combination of format, type, tiling, usage and flags supplied in the VkImageCreateInfo struct is reported by vkGetPhysicalDeviceImageFormatProperties() as unsupported. The spec valid usage text states 'The combination of format, imageType, tiling, usage, and flags must be supported, as indicated by a VK_SUCCESS return value from vkGetPhysicalDeviceImageFormatProperties invoked with the same values passed to the corresponding parameters.' (https://www.khronos.org/registry/vulkan/specs/1.0/html/vkspec.html#VUID-VkImageCreateInfo-format-00940)
01-09 16:54:22.630 23800 23832 E vulkan  : [Validation] Code 180357104 :  [ VUID-VkImageViewCreateInfo-image-01016 ]  [ VUID-VkImageViewCreateInfo-image-01016 ] Object: 0x7f8740e200 (Type = 10) | vkCreateImageView() pCreateInfo->format VK_FORMAT_D32_SFLOAT cannot be used with an image having the VK_IMAGE_TILING_OPTIMAL and VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT flags set. The spec valid usage text states 'If image was created with VK_IMAGE_TILING_OPTIMAL and usage contains VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT, format must be supported for depth/stencil attachments, as specified by the VK_FORMAT_FEATURE_DEPTH_STENCIL_ATTACHMENT_BIT flag in VkFormatProperties::optimalTilingFeatures returned by vkGetPhysicalDeviceFormatProperties with the same value of format' (https://www.khronos.org/registry/vulkan/specs/1.0/html/vkspec.html#VUID-VkImageViewCreateInfo-image-01016)
vkb::core::Image depth_image{device,
	                             extent,
	                             VK_FORMAT_D32_SFLOAT,
	                             VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT | VK_IMAGE_USAGE_INPUT_ATTACHMENT_BIT,
	                             VMA_MEMORY_USAGE_GPU_ONLY};

It would be good if the used depth format could have a fallback option, if the given format is not available, as is done with the API examples.

@ZandroFargnoli
Copy link
Contributor Author

@MichaelParkinWhite Myself and @gary-sweet have narrowed down the problem to being to do with how dynamic descriptor sets are handled now (when we force to using static UBOs the samples work). I am going to get my hands on an Adreno device to try and replicate this issue so I can fix it.

Copy link
Contributor

@sfricke-samsung sfricke-samsung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting

$ vulkan-samples/framework/stats.cpp:46:54: error: no member named 'TransactionEliminations' in 'hwcpipe::GpuCounter'
            {StatIndex::killed_tiles, {hwcpipe::GpuCounter::TransactionEliminations}},

Haven't had time to look into why yet

@ZandroFargnoli
Copy link
Contributor Author

Getting

$ vulkan-samples/framework/stats.cpp:46:54: error: no member named 'TransactionEliminations' in 'hwcpipe::GpuCounter'
            {StatIndex::killed_tiles, {hwcpipe::GpuCounter::TransactionEliminations}},

Haven't had time to look into why yet

Have you tried updating your submodules?

@ZandroFargnoli
Copy link
Contributor Author

ZandroFargnoli commented Jan 13, 2020

Wondering if this will become a common problem when infrastructure changes are made, and if a check is needed to ensure the cache matches the current version of the APK?

Indeed. I agree that some version checking is probably needed.

I have created an issue (#41) so we can solve this in the future.

@gary-sweet
Copy link
Contributor

All of Broadcom's black render issues have been fixed with the latest patches. Thanks

@ghost ghost self-requested a review January 13, 2020 17:46
@ghost ghost requested a review from gary-sweet January 16, 2020 12:57
aurolou
aurolou previously approved these changes Jan 21, 2020
Copy link

@aurolou aurolou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@MichaelParkinWhite
Copy link
Contributor

Hi, we're getting a new crash in the latest build in the Command Buffer Usage sample, relating to RenderFrameReset:

 Revision: '14'
01-22 10:43:41.508 15195 15195 F DEBUG   : ABI: 'arm64'
01-22 10:43:41.508 15195 15195 F DEBUG   : pid: 10131, tid: 15074, name: .vulkan_samples  >>> com.khronos.vulkan_samples <<<
01-22 10:43:41.508 15195 15195 F DEBUG   : signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
01-22 10:43:41.508 15195 15195 F DEBUG   :     x0   0000000000000000  x1   0000000000003ae2  x2   0000000000000006  x3   0000000000000008
01-22 10:43:41.508 15195 15195 F DEBUG   :     x4   8080000000000000  x5   8080000000000000  x6   8080000000000000  x7   0000000000008080
01-22 10:43:41.508 15195 15195 F DEBUG   :     x8   0000000000000083  x9   8fd26d1b1695ff3e  x10  0000000000000000  x11  0000000000000001
01-22 10:43:41.508 15195 15195 F DEBUG   :     x12  ffffffffffffffff  x13  0000000000000031  x14  ffffffffffffffff  x15  000077601969e715
01-22 10:43:41.508 15195 15195 F DEBUG   :     x16  000000752720b2e8  x17  00000075271ac878  x18  00000075085440a7  x19  0000000000002793
01-22 10:43:41.508 15195 15195 F DEBUG   :     x20  0000000000003ae2  x21  00000075057024f0  x22  0000000000002793  x23  00000075084c2df0
01-22 10:43:41.508 15195 15195 F DEBUG   :     x24  0000007505702570  x25  0000007505606000  x26  0000000000000058  x27  00000075084c1ffc
01-22 10:43:41.508 15195 15195 F DEBUG   :     x28  0000007ffd56a620  x29  0000007505701540  x30  000000752715f024
01-22 10:43:41.508 15195 15195 F DEBUG   :     sp   0000007505701500  pc   00000075271ac880  pstate 0000000060000000
01-22 10:43:41.544 15195 15195 F DEBUG   :
01-22 10:43:41.544 15195 15195 F DEBUG   : backtrace:
01-22 10:43:41.544 15195 15195 F DEBUG   :     #00 pc 000000000006b880  /system/lib64/libc.so (tgkill+8)
01-22 10:43:41.544 15195 15195 F DEBUG   :     #01 pc 000000000001e020  /system/lib64/libc.so (abort+88)
01-22 10:43:41.544 15195 15195 F DEBUG   :     #02 pc 0000000000731a88  /data/app/com.khronos.vulkan_samples-NIpL4Sr2-XrOhUVsv-bozg==/lib/arm64/libvulkan_samples.so (_ZN3vkb11RenderFrame5resetEv+180)
01-22 10:43:41.544 15195 15195 F DEBUG   :     #03 pc 000000000072d1a4  /data/app/com.khronos.vulkan_samples-NIpL4Sr2-XrOhUVsv-bozg==/lib/arm64/libvulkan_samples.so (_ZN3vkb13RenderContext10wait_frameEv+32)
01-22 10:43:41.544 15195 15195 F DEBUG   :     #04 pc 000000000072c3e8  /data/app/com.khronos.vulkan_samples-NIpL4Sr2-XrOhUVsv-bozg==/lib/arm64/libvulkan_samples.so (_ZN3vkb13RenderContext11begin_frameEv+620)
01-22 10:43:41.544 15195 15195 F DEBUG   :     #05 pc 000000000072c0c0  /data/app/com.khronos.vulkan_samples-NIpL4Sr2-XrOhUVsv-bozg==/lib/arm64/libvulkan_samples.so (_ZN3vkb13RenderContext5beginENS_13CommandBuffer9ResetModeE+148)
01-22 10:43:41.544 15195 15195 F DEBUG   :     #06 pc 000000000051d2b0  /data/app/com.khronos.vulkan_samples-NIpL4Sr2-XrOhUVsv-bozg==/lib/arm64/libvulkan_samples.so (_ZN18CommandBufferUsage6updateEf+404)
01-22 10:43:41.544 15195 15195 F DEBUG   :     #07 pc 000000000055782c  /data/app/com.khronos.vulkan_samples-NIpL4Sr2-XrOhUVsv-bozg==/lib/arm64/libvulkan_samples.so (_ZN3vkb11Application4stepEv+152)
01-22 10:43:41.544 15195 15195 F DEBUG   :     #08 pc 00000000005507e4  /data/app/com.khronos.vulkan_samples-NIpL4Sr2-XrOhUVsv-bozg==/lib/arm64/libvulkan_samples.so (_ZN3vkb13VulkanSamples6updateEf+132)
01-22 10:43:41.545 15195 15195 F DEBUG   :     #09 pc 000000000055782c  /data/app/com.khronos.vulkan_samples-NIpL4Sr2-XrOhUVsv-bozg==/lib/arm64/libvulkan_samples.so (_ZN3vkb11Application4stepEv+152)
01-22 10:43:41.545 15195 15195 F DEBUG   :     #10 pc 00000000005604a8  /data/app/com.khronos.vulkan_samples-NIpL4Sr2-XrOhUVsv-bozg==/lib/arm64/libvulkan_samples.so (_ZN3vkb8Platform3runEv+312)
01-22 10:43:41.545 15195 15195 F DEBUG   :     #11 pc 00000000007c76b8  /data/app/com.khronos.vulkan_samples-NIpL4Sr2-XrOhUVsv-bozg==/lib/arm64/libvulkan_samples.so (_ZN3vkb15AndroidPlatform9main_loopEv+232)
01-22 10:43:41.545 15195 15195 F DEBUG   :     #12 pc 000000000054801c  /data/app/com.khronos.vulkan_samples-NIpL4Sr2-XrOhUVsv-bozg==/lib/arm64/libvulkan_samples.so (android_main+360)
01-22 10:43:41.545 15195 15195 F DEBUG   :     #13 pc 0000000000f78f8c  /data/app/com.khronos.vulkan_samples-NIpL4Sr2-XrOhUVsv-bozg==/lib/arm64/libvulkan_samples.so
01-22 10:43:41.545 15195 15195 F DEBUG   :     #14 pc 000000000006820c  /system/lib64/libc.so (_ZL15__pthread_startPv+36)
01-22 10:43:41.545 15195 15195 F DEBUG   :     #15 pc 000000000001f774  /system/lib64/libc.so (__start_thread+68)

@MichaelParkinWhite
Copy link
Contributor

MichaelParkinWhite commented Jan 22, 2020

@ZandroFargnoli the previous format issue has now been resolved, cheers :)! I also believe the black screen issue has been resolved as well. Though only have access to a limited set of devices right now, so will update on that later.

However there appears to be a new validation issue (Samsung Galaxy S7 Adreno - G930V), which i'm assuming is associated with setting the STENCIL aspect flag when a format which does not have a stencil component is used?

[Validation] Code 0 :  [ UNASSIGNED-CoreValidation-DrawState-InvalidImageView ]  [ UNASSIGNED-CoreValidation-DrawState-InvalidImageView ] Object: 0x137 (Type = 23) | vkUpdateDescriptorSets() failed write update validation for VkDescriptorSet 0x137[] with error: Write update to VkDescriptorSet VkDescriptorSet 0x137[] allocated with VkDescriptorSetLayout VkDescriptorSetLayout 0x131[] binding #0 failed with error message: Attempted write update to image descriptor failed due to: ImageView (VkImageView 0x29[]) has both STENCIL and DEPTH aspects set.

This manifests as flickering and incorrect rendering in the 'Render Subpasses' and 'Pipeline Barriers' samples. The validation error also triggers in 'Layout Transitions' sample.

@claassistantio
Copy link

claassistantio commented Jan 23, 2020

CLA assistant check
All committers have signed the CLA.

@gary-sweet
Copy link
Contributor

I'm finding that I can only run the pipeline_cache example once after deleting /tmp/cache.data. Subsequent runs crash on startup until I delete /tmp/cache.data again.

Is anyone else seeing the same?

@SaschaWillems
Copy link
Collaborator

I'm finding that I can only run the pipeline_cache example once after deleting /tmp/cache.data. Subsequent runs crash on startup until I delete /tmp/cache.data again.

Same for me on an Intel IGP. Sample runs fine on first try and crashes on second run. Works again if I manually delete the cache data.

@ZandroFargnoli
Copy link
Contributor Author

@gary-sweet @SaschaWillems When fixing the validation error and crash Michael was experiencing I seemed to have fixed this problem. I am still trying to work out exactly why we are seeing this crash and will push the fixed code once I have done so

gary-sweet
gary-sweet previously approved these changes Feb 17, 2020
aurolou
aurolou previously approved these changes Feb 18, 2020
gary-sweet
gary-sweet previously approved these changes Feb 18, 2020
gary-sweet
gary-sweet previously approved these changes Feb 19, 2020
aurolou
aurolou previously approved these changes Feb 19, 2020
Updated liceneses and sample CMakeLists

Updated sample tutorials to be more clear about mali specific content

Ordering samples now removes duplicates from the order list

Render granularity only displays the first time, and it is now calculated correctly

Added missing headers to CMake, variant definitions are now OO, removed superfluous dynamic code

Shader program removed and reverted back inside pipeline layout

Removed FAQ to be added as a seperate MR

Added contents to memory limits document

Updated depth format function and integrated it into all existing depth buffer images
@ghost ghost merged commit 2d31f78 into master Feb 20, 2020
@JoseEmilio-ARM JoseEmilio-ARM deleted the arm-version-1-3 branch June 9, 2020 10:01
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants