Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate CUDA compilation problems #5799

Closed
hkaiser opened this issue Feb 23, 2022 · 3 comments · Fixed by #6452
Closed

Investigate CUDA compilation problems #5799

hkaiser opened this issue Feb 23, 2022 · 3 comments · Fixed by #6452
Labels
category: tests platform: CUDA type: compatibility issue type: defect
Milestone
1.10.0

Comments

@hkaiser
Copy link
Member

hkaiser commented Feb 23, 2022

Two of our CUDA tests fail compiling by NVCC unceremoniously stopping without error message. For now, these tests have been disabled (see: #5796). We should investigate whether this can be worked around.
@hkaiser
Copy link
Member Author

hkaiser commented Feb 23, 2022

@G-071 Gregor, this is the ticket I promised to create.

@G-071 G-071 mentioned this issue May 18, 2023
Merged
bors bot pushed a commit that referenced this issue Jun 1, 2023
@G-071
Merge #6248
3d95d66 
6248: Fix CUDA/HIP Jenkins pipelines  r=hkaiser a=G-071

This PR aims to fix the unstable CUDA/HIP tests.

Changes so far:
- Fixes CUDA architectures for Jenkins nodes
- Exclude faulty Jenkins node for now (bahram: listed as V100 GPU node but does not list any GPUs according to nvidia-smi)
- Allow Jenkins HIP builds with warnings (in order to run tests despite of the warnings, just like the CUDA tests already do)
- I further looked into #5799 again to see if the new compilers and cuda versions made any difference. Unfortunately not, using gcc/12 and cuda/12 compilation still fails with:
`'cudafe++' died due to signal 11 (Invalid memory reference)` 
I adapted the guards in the failing examples accordingly (to also skip the tests with newer cuda versions)

The PR is a bit of a work-in-progress as I imagine it will take a few iterations to get everything right. That being said, let's see if the current changes already help and trigger the Jenkins tests!

Co-authored-by: Gregor Daiss <Gregor.Daiss+git@gmail.com>
@G-071
Copy link
Member

G-071 commented Mar 4, 2024 •
edited
Loading

NVCC apparently got some improvement/fixes with the CUDA/12.3 release meaning we might be able to finally fix this old issue (which is still very much relevant as the same problem prevents us from using/compiling Kokkos >= 4.1 with gcc and the HPX and CUDA execution space both enabled)

With previous CUDA releases, it simply crashed when compiling transform_stream.cu:
nvcc error : 'cudafe++' died due to signal 11 (Invalid memory reference)

With CUDA 12.3 we get some actual compilation errors instead:

[100%] Building CUDA object libs/core/async_cuda/tests/unit/CMakeFiles/transform_stream_test.dir/transform_stream.cu.o
cd /work/gdaiss/hpx/build/libs/core/async_cuda/tests/unit && /usr/local/cuda-12.3/bin/nvcc -forward-unknown-to-host-compiler -DASIO_DISABLE_CONSTEXPR -DHPX_APPLICATION_EXPORTS -DHPX_APPLICATION_NAME=transform_stream_test -DHPX_APPLICATION_NAME_DEFAULT=transform_stream_test -DHPX_APPLICATION_STRING=\"transform_stream_test\" -DHPX_DEBUG -DHPX_PREFIX=\"/work/gdaiss/hpx/build\" -DHPX_PREFIX_DEFAULT=\"/usr/local\" -D_GNU_SOURCE --options-file CMakeFiles/transform_stream_test.dir/includes_CUDA.rsp -g -std=c++17 --generate-code=arch=compute_70,code=[compute_70,sm_70] --generate-code=arch=compute_80,code=[compute_80,sm_80] -Xcompiler=-fPIE
 -w --extended-lambda --default-stream per-thread --expt-relaxed-constexpr -MD -MT libs/core/async_cuda/tests/unit/CMakeFiles/transform_stream_test.dir/transform_stream.cu.o -MF CMakeFiles/transform_stream_test.dir/transform_stream.cu.o.d -x cu -c /work/gdaiss/hpx/libs/core/async_cuda/tests/unit/transform_stream.cu -o CMakeFiles/transform_stream_test.dir/transform_stream.cu.o                                                                                                                                                                                                                                                                    
/work/gdaiss/hpx/libs/core/execution/include/hpx/execution/algorithms/just.hpp(106): error: expected a ">"
                      all_of_v<std::is_nothrow_move_constructible<Ts>...>)
                                                                     ^
          detected during:
            instantiation of "hpx::execution::experimental::detail::tag_invoke" based on template argument <hpx::cuda::experimental::detail::transform_stream_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::tra
nsform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver, dummy>> at line 79 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/traits/is_invocable.hpp                                                                                                                                                                                                                                
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>> &&, hpx::cuda::experimental::detail::transform_stream_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::
thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver, dummy> &&>]" at line 167 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/tag_invoke.hpp                                   
            instantiation of "const __nv_bool hpx::functional::is_tag_invocable_v [with Tag=hpx::execution::experimental::connect_t, Args=<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>> &&, hpx::cuda::experimental::detail::transform_stream_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>::operatio
n_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver, dummy> &&>]" at line 309 of /work/gdaiss/hpx/libs/core/async_cuda/include/hpx/async_cuda/transform_stream.hpp                                                             
            instantiation of "auto hpx::cuda::experimental::detail::tag_invoke(hpx::execution::experimental::connect_t, hpx::cuda::experimental::detail::transform_stream_sender<std::remove_reference<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::make_index_pack_t<0UL>> &>::type, dummy> &&, R &&) [with R=hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimenta
l::thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver]" at line 79 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/traits/is_invocable.hpp                                    
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy> &&, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experime
ntal::thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver &&>]" at line 167 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/tag_invoke.hpp                                     
            [ 6 instantiation contexts not shown ]
            instantiation of class "hpx::util::detail::invoke_result_impl<F (Ts...), std::void_t<decltype((<expression>))>> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t &&, Ts=<hpx::execution::experimental::connect_t &&, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&, hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::
schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type &&>]" at line 38 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/invoke_result.hpp                                                                                                                                                                                                              
            instantiation of class "hpx::util::invoke_result<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&, hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::trans
form_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type &&>]" at line 443 of /work/gdaiss/hpx/libs/core/execution/include/hpx/execution/algorithms/sync_wait.hpp                                                                                                                                                                                                                                                                   
            instantiation of "auto hpx::this_thread::experimental::tag_fallback_invoke(hpx::this_thread::experimental::sync_wait_t, Sender &&) [with Sender=hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, _concept_requires_427=42, _concept_check_427=0]" at line 79 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/traits/is_invocable.hpp                               
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::detail::tag_fallback_invoke_t_ns::tag_fallback_invoke_t, Ts=<hpx::this_thread::experimental::sync_wait_t, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&>]" at line 171 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/detail/tag_fallback_invoke.hpp                      
            instantiation of "const __nv_bool hpx::functional::detail::is_tag_fallback_invocable_v [with Tag=hpx::this_thread::experimental::sync_wait_t, Args=<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&>]" at line 125 of /work/gdaiss/hpx/libs/core/async_cuda/tests/unit/transform_stream.cu                                                                                      

/work/gdaiss/hpx/libs/core/execution/include/hpx/execution/algorithms/just.hpp(106): error: pack expansion does not make use of any argument packs
                      all_of_v<std::is_nothrow_move_constructible<Ts>...>)
                                                                     ^
          detected during:
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>> &&, hpx::cuda::experimental::detail::transform_stream_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::
thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver, dummy> &&>]" at line 167 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/tag_invoke.hpp                                   
            instantiation of "const __nv_bool hpx::functional::is_tag_invocable_v [with Tag=hpx::execution::experimental::connect_t, Args=<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>> &&, hpx::cuda::experimental::detail::transform_stream_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>::operatio
n_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver, dummy> &&>]" at line 309 of /work/gdaiss/hpx/libs/core/async_cuda/include/hpx/async_cuda/transform_stream.hpp                                                             
            instantiation of "auto hpx::cuda::experimental::detail::tag_invoke(hpx::execution::experimental::connect_t, hpx::cuda::experimental::detail::transform_stream_sender<std::remove_reference<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::make_index_pack_t<0UL>> &>::type, dummy> &&, R &&) [with R=hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimenta
l::thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver]" at line 79 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/traits/is_invocable.hpp                                    
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy> &&, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experime
ntal::thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver &&>]" at line 167 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/tag_invoke.hpp                                     
            instantiation of "const __nv_bool hpx::functional::is_tag_invocable_v [with Tag=hpx::execution::experimental::connect_t, Args=<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy> &&, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>::op
eration_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver &&>]" at line 38 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/invoke_result.hpp                                                                   
            [ 5 instantiation contexts not shown ]
            instantiation of class "hpx::util::detail::invoke_result_impl<F (Ts...), std::void_t<decltype((<expression>))>> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t &&, Ts=<hpx::execution::experimental::connect_t &&, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&, hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::
schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type &&>]" at line 38 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/invoke_result.hpp                                                                                                                                                                                                              
            instantiation of class "hpx::util::invoke_result<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&, hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::trans
form_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type &&>]" at line 443 of /work/gdaiss/hpx/libs/core/execution/include/hpx/execution/algorithms/sync_wait.hpp                                                                                                                                                                                                                                                                   
            instantiation of "auto hpx::this_thread::experimental::tag_fallback_invoke(hpx::this_thread::experimental::sync_wait_t, Sender &&) [with Sender=hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, _concept_requires_427=42, _concept_check_427=0]" at line 79 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/traits/is_invocable.hpp                               
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::detail::tag_fallback_invoke_t_ns::tag_fallback_invoke_t, Ts=<hpx::this_thread::experimental::sync_wait_t, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&>]" at line 171 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/detail/tag_fallback_invoke.hpp                      
            instantiation of "const __nv_bool hpx::functional::detail::is_tag_fallback_invocable_v [with Tag=hpx::this_thread::experimental::sync_wait_t, Args=<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&>]" at line 125 of /work/gdaiss/hpx/libs/core/async_cuda/tests/unit/transform_stream.cu                                                                                      

/work/gdaiss/hpx/libs/core/type_support/include/hpx/type_support/pack.hpp(125): error: class "hpx::util::all_of<std::is_nothrow_move_constructible<<error-type>>>" has no member "value"
      inline constexpr bool all_of_v = all_of<Ts...>::value;
                                                      ^
          detected during:
            instantiation of "const __nv_bool hpx::util::all_of_v [with Ts=<std::is_nothrow_move_constructible<<error-type>>>]" at line 106 of /work/gdaiss/hpx/libs/core/execution/include/hpx/execution/algorithms/just.hpp
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>> &&, hpx::cuda::experimental::detail::transform_stream_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::
thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver, dummy> &&>]" at line 167 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/tag_invoke.hpp                                   
            instantiation of "const __nv_bool hpx::functional::is_tag_invocable_v [with Tag=hpx::execution::experimental::connect_t, Args=<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>> &&, hpx::cuda::experimental::detail::transform_stream_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>::operatio
n_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver, dummy> &&>]" at line 309 of /work/gdaiss/hpx/libs/core/async_cuda/include/hpx/async_cuda/transform_stream.hpp                                                             
            instantiation of "auto hpx::cuda::experimental::detail::tag_invoke(hpx::execution::experimental::connect_t, hpx::cuda::experimental::detail::transform_stream_sender<std::remove_reference<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::make_index_pack_t<0UL>> &>::type, dummy> &&, R &&) [with R=hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimenta
l::thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver]" at line 79 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/traits/is_invocable.hpp                                    
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy> &&, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experime
ntal::thread_pool_scheduler>::operation_state<hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type>::predecessor_sender_receiver &&>]" at line 167 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/tag_invoke.hpp                                     
            [ 6 instantiation contexts not shown ]
            instantiation of class "hpx::util::detail::invoke_result_impl<F (Ts...), std::void_t<decltype((<expression>))>> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t &&, Ts=<hpx::execution::experimental::connect_t &&, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&, hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::
schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type &&>]" at line 38 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/invoke_result.hpp                                                                                                                                                                                                              
            instantiation of class "hpx::util::invoke_result<F, Ts...> [with F=const hpx::functional::tag_invoke_t_ns::tag_invoke_t, Ts=<hpx::execution::experimental::connect_t, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&, hpx::execution::experimental::detail::sync_wait_receiver<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::trans
form_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, hpx::execution::experimental::detail::sync_wait_type::single>::type &&>]" at line 443 of /work/gdaiss/hpx/libs/core/execution/include/hpx/execution/algorithms/sync_wait.hpp                                                                                                                                                                                                                                                                   
            instantiation of "auto hpx::this_thread::experimental::tag_fallback_invoke(hpx::this_thread::experimental::sync_wait_t, Sender &&) [with Sender=hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler>, _concept_requires_427=42, _concept_check_427=0]" at line 79 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/traits/is_invocable.hpp                               
            instantiation of class "hpx::is_invocable<F, Ts...> [with F=const hpx::functional::detail::tag_fallback_invoke_t_ns::tag_fallback_invoke_t, Ts=<hpx::this_thread::experimental::sync_wait_t, hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&>]" at line 171 of /work/gdaiss/hpx/libs/core/tag_invoke/include/hpx/functional/detail/tag_fallback_invoke.hpp                      
            instantiation of "const __nv_bool hpx::functional::detail::is_tag_fallback_invocable_v [with Tag=hpx::this_thread::experimental::sync_wait_t, Args=<hpx::execution::experimental::detail::schedule_from_sender<hpx::cuda::experimental::detail::transform_stream_sender<hpx::execution::experimental::detail::just_sender<hpx::execution::experimental::set_value_t, hpx::util::pack_c<std::size_t>>, dummy>, hpx::execution::experimental::thread_pool_scheduler> &&>]" at line 125 of /work/gdaiss/hpx/libs/core/async_cuda/tests/unit/transform_stream.cu                                                                                      

3 errors detected in the compilation of "/work/gdaiss/hpx/libs/core/async_cuda/tests/unit/transform_stream.cu".
make[3]: *** [libs/core/async_cuda/tests/unit/CMakeFiles/transform_stream_test.dir/build.make:77: libs/core/async_cuda/tests/unit/CMakeFiles/transform_stream_test.dir/transform_stream.cu.o] Error 1
                                                                                                                                                                                                                                                                    ```

To reproduce:

  1. Log into one of the cuda-A100-intel nodes on rostam
  2. module load cuda/12.3 gcc/12.3
  3. cmake step:
cmake -DCMAKE_BUILD_TYPE=Debug -DHPX_WITH_PARALLEL_TESTS_BIND_NONE=ON  -DHPX_WITH_CHECK_MODULE_DEPENDENCIES=ON -DHPX_WITH_PARCELPORT_COUNTERS=ON -DHPX_WITH_CXX_STANDARD=17 -DHPX_WITH_MALLOC=system -DHPX_WITH_FETCH_ASIO=ON -DHPX_WITH_COMPILER_WARNI
NGS=ON -DHPX_WITH_COMPILER_WARNINGS_AS_ERRORS=OFF -DHPX_WITH_CUDA=ON -DHPX_WITH_NETWORKING=OFF -DHPX_WITH_DISTRIBUTED_RUNTIME=OFF  -DHPX_WITH_ASYNC_MPI=OFF -DCMAKE_CUDA_ARCHITECTURES='70;80'

(Note, changing the CXX standard to 20 does not seem change anything about this issue -- we still get the same error message)
4. make tests.unit.modules.async_cuda.transform_stream VERBOSE=1 -j32

I have tested this both with the current HPX master and with the commit I tried last year (when last checking this with CUDA/12.1), just to make sure this new behavior is not due to some changes in HPX (it is not -- without CUDA/12.3 I can still reproduce the old nvcc crash).

@hkaiser Given the new compilation errors: do you have an ideas how to fix this (or work around it)?

@hkaiser
Copy link
Member Author

hkaiser commented Mar 4, 2024

@G-071 we could do the same as we've done in similar cases before, i.e. wrap the whole noexcept() clause into a #if !defined(HPX_COMPUTE_DEVICE_CODE)

@G-071 G-071 mentioned this issue Mar 13, 2024
Merged
@hkaiser hkaiser added this to the 1.10.0 milestone May 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
category: tests platform: CUDA type: compatibility issue type: defect
Projects
None yet
Milestone
1.10.0
Development

Successfully merging a pull request may close this issue.

Fix nvcc crashes in transform_stream.cu and synchronize.cu G-071/hpx
2 participants
@hkaiser @G-071