Using add_halide_runtime when compiling with MSVC + GPU support?

[felipegb94]

Hi Halide Developers,

I am trying to better understand when I should try to use USE_RUNTIME

I have a library (mylib) where I have multiple Halide AOT-compiled functions. I was running into the following runtime error in Windows (compiling with MSVC) when I call two AOT generators in a row (on a GPU):

Error: halide_copy_to_device does not support switching interfaces

The error seems to come from here: https://github.com/halide/Halide/blob/main/src/runtime/device_interface.cpp#L170

What ended fixing this error was that in MSVC I had to make my library STATIC (See snippet below).

I tried fixing this by using add_halide_runtime(my_runtime TARGETS ${Halide_TARGET}) and then adding USE_RUNTIME to add_halide_library, but this did not work. I kept getting the same crash whenever I would call by two AOT-compiled functions in a row.

I am wondering if USE_RUNTIME should have fixed the above, and if I am just using it in the incorrect way.

CMakeLists Snippets

This worked:

# On Linux the dynamic linker deduplicates weak symbols across .so files,
# so the SHARED build works correctly there.
if(MSVC)
    add_library(mylib STATIC
        api.cpp
        api_impl.cpp
    )
else()
    add_library(mylib SHARED
        api.cpp
        api_impl.cpp
    )
endif()

# example of how I am linking a generator
add_executable(func1.generator func1.cpp)
add_halide_library(func1 FROM func1.generator
            GENERATOR func
            TARGETS ${Halide_TARGET}
            FEATURES cuda
        )
target_link_libraries(func1.generator PUBLIC Halide::Generator Halide::Tools)
target_include_directories(func1.generator PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include)
target_link_libraries(mylib PUBLIC func1)

This did not work

add_library(mylib SHARED
    api.cpp
    api_impl.cpp
)
add_halide_runtime(my_runtime TARGETS ${Halide_TARGET})
target_link_libraries(mylib PUBLIC my_runtime)


# example of how I am adding a generator
add_executable(func1.generator func1.cpp)
add_halide_library(func1 FROM func1.generator
            GENERATOR func
            TARGETS ${Halide_TARGET}
            FEATURES cuda
            USE_RUNTIME my_runtime
        )
target_link_libraries(func1.generator PUBLIC Halide::Generator Halide::Tools)
target_include_directories(func1.generator PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include)
target_link_libraries(mylib PUBLIC func1)

[gistrec]

USE_RUNTIME my_runtime needs to be applied to every add_halide_library() target that should share the same Halide runtime, not just one of them

If any of the AOT libraries is still built without USE_RUNTIME, Halide will generate a separate <target>.runtime for that library. That can leave you with multiple copies of the runtime/device-interface state in the same process, which is a common way to get errors like:

halide_copy_to_device does not support switching interfaces

So the important part is that all GPU-using AOT libraries share the same runtime target:

add_halide_runtime(my_runtime TARGETS ${Halide_TARGET})

add_halide_library(func1 FROM func1.generator
    GENERATOR func1
    TARGETS ${Halide_TARGET}
    FEATURES cuda
    USE_RUNTIME my_runtime
)

add_halide_library(func2 FROM func2.generator
    GENERATOR func2
    TARGETS ${Halide_TARGET}
    FEATURES cuda
    USE_RUNTIME my_runtime
)

target_link_libraries(mylib PUBLIC
    my_runtime
    func1
    func2
)

Your STATIC workaround makes sense: it likely causes the generated Halide code and runtime symbols to end up in one link unit, so the runtime state is effectively unified. With a SHARED library on MSVC/Windows, you should make that sharing explicit by using one add_halide_runtime() target and passing it via USE_RUNTIME to every generated Halide library that participates in the same GPU pipeline

Also, I would keep mylib linked directly to my_runtime, not only indirectly through the generated libraries, so the dependency is explicit

[alexreinking]

As the author of these helpers, this is the right approach.

[felipegb94]

Thanks for the response! This makes sense. Do you see anything wrong in the way I was using USE_RUNTIME?

The thing is that I was applying USE_RUNTIME my_runtime to all the add_halide_library calls and still got that error. This may just mean that I was using it correctly, but either I missed one halide library, or I had a different bug in my CMake that was not letting this work in Windows.

[alexreinking]

If this is indeed solvable in the build system, you might need to link the runtime library as a whole archive. In the USE_RUNTIME case with a shared library, try this:

add_library(mylib SHARED
    api.cpp
    api_impl.cpp
)
add_halide_runtime(my_runtime TARGETS ${Halide_TARGET})
target_link_libraries(mylib
    PRIVATE "$<LINK_LIBRARY:WHOLE_ARCHIVE,my_runtime>"
    INTERFACE Halide::Runtime  # only needed if your library's API uses Halide types.
)

If that doesn't work, there might be something wrong in the build helper logic for merging GPU features together. In that case, I'd appreciate a full MRE.