CUDA compute capability check may be incomplete

[ocaisa]

cuobjdump -all may be required to query the ptx code within a binary, see https://forums.developer.nvidia.com/t/keep-ptx-with-cuda-separable-compilation/337648 for details.

[ocaisa]

To give a specific example:

[ocaisa@login1 ~]$ cuobjdump /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/software/LightGBM/4.5.0-foss-2023a-CUDA-12.1.1/lib/python3.11/site-packages/lightgbm/lib/lib_lightgbm.so

Fatbin elf code:
================
arch = sm_80
code version = [1,7]
host = linux
compile_size = 64bit
compressed
identifier = CMakeFiles/lightgbm_capi_objs.dir/src/c_api.cpp.o CMakeFiles/lightgbm_objs.dir/src/boosting/boosting.cpp.o CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt.cpp.o CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt_model_text.cpp.o CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt_prediction.cpp.o CMakeFiles/lightgbm_objs.dir/src/boosting/prediction_early_stop.cpp.o CMakeFiles/lightgbm_objs.dir/src/boosting/sample_strategy.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/bin.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/config.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/config_auto.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/dataset.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/dataset_loader.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/file_io.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/json11.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/metadata.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/parser.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/train_share_states.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/tree.cpp.o CMakeFiles/lightgbm_objs.dir/src/metric/dcg_calculator.cpp.o CMakeFiles/lightgbm_objs.dir/src/metric/metric.cpp.o CMakeFiles/lightgbm_objs.dir/src/network/linker_topo.cpp.o CMakeFiles/lightgbm_objs.dir/src/network/linkers_mpi.cpp.o CMakeFiles/lightgbm_objs.dir/src/network/linkers_socket.cpp.o CMakeFiles/lightgbm_objs.dir/src/network/network.cpp.o CMakeFiles/lightgbm_objs.dir/src/objective/objective_function.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/data_parallel_tree_learner.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/feature_histogram.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/feature_parallel_tree_learner.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/gpu_tree_learner.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/gradient_discretizer.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/linear_tree_learner.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/serial_tree_learner.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/tree_learner.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/voting_parallel_tree_learner.cpp.o CMakeFiles/lightgbm_objs.dir/src/utils/openmp_wrapper.cpp.o CMakeFiles/lightgbm_objs.dir/src/boosting/cuda/cuda_score_updater.cpp.o CMakeFiles/lightgbm_objs.dir/src/boosting/cuda/cuda_score_updater.cu.o CMakeFiles/lightgbm_objs.dir/src/cuda/cuda_algorithms.cu.o CMakeFiles/lightgbm_objs.dir/src/cuda/cuda_utils.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_column_data.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_column_data.cu.o CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_metadata.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_row_data.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_tree.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_tree.cu.o CMakeFiles/lightgbm_objs.dir/src/metric/cuda/cuda_binary_metric.cpp.o CMakeFiles/lightgbm_objs.dir/src/metric/cuda/cuda_pointwise_metric.cpp.o CMakeFiles/lightgbm_objs.dir/src/metric/cuda/cuda_pointwise_metric.cu.o CMakeFiles/lightgbm_objs.dir/src/metric/cuda/cuda_regression_metric.cpp.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_binary_objective.cpp.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_binary_objective.cu.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_multiclass_objective.cpp.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_multiclass_objective.cu.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_rank_objective.cpp.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_rank_objective.cu.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_regression_objective.cpp.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_regression_objective.cu.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_best_split_finder.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_best_split_finder.cu.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_data_partition.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_data_partition.cu.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_gradient_discretizer.cu.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_histogram_constructor.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_histogram_constructor.cu.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_leaf_splits.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_leaf_splits.cu.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_single_gpu_tree_learner.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_single_gpu_tree_learner.cu.o CMakeFiles/histo_16_64_256_sp_const.dir/src/treelearner/kernels/histogram_16_64_256.cu.o CMakeFiles/histo_16_64_256_sp.dir/src/treelearner/kernels/histogram_16_64_256.cu.o CMakeFiles/histo_16_64_256-fulldata_sp_const.dir/src/treelearner/kernels/histogram_16_64_256.cu.o CMakeFiles/histo_16_64_256-fulldata_sp.dir/src/treelearner/kernels/histogram_16_64_256.cu.o

This looks like it has no ptx support, but if you dig a little deeper with cuobjdump -all:

Fatbin elf code:
================
arch = sm_80
code version = [1,7]
host = linux
compile_size = 64bit
compressed
identifier = CMakeFiles/lightgbm_capi_objs.dir/src/c_api.cpp.o CMakeFiles/lightgbm_objs.dir/src/boosting/boosting.cpp.o CMakeFiles/lightgbm_objs.dir/src/boost
ing/gbdt.cpp.o CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt_model_text.cpp.o CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt_prediction.cpp.o CMakeFiles/ligh
tgbm_objs.dir/src/boosting/prediction_early_stop.cpp.o CMakeFiles/lightgbm_objs.dir/src/boosting/sample_strategy.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/bin
.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/config.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/config_auto.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/dataset.cpp.
o CMakeFiles/lightgbm_objs.dir/src/io/dataset_loader.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/file_io.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/json11.cpp.o
CMakeFiles/lightgbm_objs.dir/src/io/metadata.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/parser.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/train_share_states.cpp
.o CMakeFiles/lightgbm_objs.dir/src/io/tree.cpp.o CMakeFiles/lightgbm_objs.dir/src/metric/dcg_calculator.cpp.o CMakeFiles/lightgbm_objs.dir/src/metric/metric.
cpp.o CMakeFiles/lightgbm_objs.dir/src/network/linker_topo.cpp.o CMakeFiles/lightgbm_objs.dir/src/network/linkers_mpi.cpp.o CMakeFiles/lightgbm_objs.dir/src/n
etwork/linkers_socket.cpp.o CMakeFiles/lightgbm_objs.dir/src/network/network.cpp.o CMakeFiles/lightgbm_objs.dir/src/objective/objective_function.cpp.o CMakeFi
les/lightgbm_objs.dir/src/treelearner/data_parallel_tree_learner.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/feature_histogram.cpp.o CMakeFiles/lightgb
m_objs.dir/src/treelearner/feature_parallel_tree_learner.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/gpu_tree_learner.cpp.o CMakeFiles/lightgbm_objs.di
r/src/treelearner/gradient_discretizer.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/linear_tree_learner.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearn
er/serial_tree_learner.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/tree_learner.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/voting_parallel_tree
_learner.cpp.o CMakeFiles/lightgbm_objs.dir/src/utils/openmp_wrapper.cpp.o CMakeFiles/lightgbm_objs.dir/src/boosting/cuda/cuda_score_updater.cpp.o CMakeFiles/
lightgbm_objs.dir/src/boosting/cuda/cuda_score_updater.cu.o CMakeFiles/lightgbm_objs.dir/src/cuda/cuda_algorithms.cu.o CMakeFiles/lightgbm_objs.dir/src/cuda/c
uda_utils.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_column_data.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_column_data.cu.o CMakeFiles/ligh
tgbm_objs.dir/src/io/cuda/cuda_metadata.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_row_data.cpp.o CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_tree.
cpp.o CMakeFiles/lightgbm_objs.dir/src/io/cuda/cuda_tree.cu.o CMakeFiles/lightgbm_objs.dir/src/metric/cuda/cuda_binary_metric.cpp.o CMakeFiles/lightgbm_objs.d
ir/src/metric/cuda/cuda_pointwise_metric.cpp.o CMakeFiles/lightgbm_objs.dir/src/metric/cuda/cuda_pointwise_metric.cu.o CMakeFiles/lightgbm_objs.dir/src/metric
/cuda/cuda_regression_metric.cpp.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_binary_objective.cpp.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda
/cuda_binary_objective.cu.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_multiclass_objective.cpp.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cu
da_multiclass_objective.cu.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_rank_objective.cpp.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_ra
nk_objective.cu.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_regression_objective.cpp.o CMakeFiles/lightgbm_objs.dir/src/objective/cuda/cuda_regress
ion_objective.cu.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_best_split_finder.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_best_
split_finder.cu.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_data_partition.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_data_part
ition.cu.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_gradient_discretizer.cu.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_histogram_c
onstructor.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_histogram_constructor.cu.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_leaf
_splits.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_leaf_splits.cu.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_single_gpu_tree_l
earner.cpp.o CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda/cuda_single_gpu_tree_learner.cu.o CMakeFiles/histo_16_64_256_sp_const.dir/src/treelearner/kerne
ls/histogram_16_64_256.cu.o CMakeFiles/histo_16_64_256_sp.dir/src/treelearner/kernels/histogram_16_64_256.cu.o CMakeFiles/histo_16_64_256-fulldata_sp_const.di
r/src/treelearner/kernels/histogram_16_64_256.cu.o CMakeFiles/histo_16_64_256-fulldata_sp.dir/src/treelearner/kernels/histogram_16_64_256.cu.o

Fatbin ptx code:
================
arch = sm_80
code version = [8,1]
host = linux
compile_size = 64bit
compressed
identifier = /tmp/vsc48506/easybuild/build/LightGBM/4.5.0/foss-2023a-CUDA-12.1.1/lightgbm/lightgbm-4.5.0/src/boosting/cuda/cuda_score_updater.cu
ptxasOptions = --compile-only  --generate-line-info

Fatbin elf code:
================
arch = sm_80
code version = [1,7]
host = linux
compile_size = 64bit
compressed
identifier = /tmp/vsc48506/easybuild/build/LightGBM/4.5.0/foss-2023a-CUDA-12.1.1/lightgbm/lightgbm-4.5.0/src/boosting/cuda/cuda_score_updater.cu

Fatbin ptx code:
================
arch = sm_80
code version = [8,1]
host = linux
compile_size = 64bit
compressed
identifier = /tmp/vsc48506/easybuild/build/LightGBM/4.5.0/foss-2023a-CUDA-12.1.1/lightgbm/lightgbm-4.5.0/src/cuda/cuda_algorithms.cu
ptxasOptions = --compile-only  --generate-line-info

Fatbin elf code:
================
arch = sm_80
code version = [1,7]
host = linux
compile_size = 64bit
compressed
identifier = /tmp/vsc48506/easybuild/build/LightGBM/4.5.0/foss-2023a-CUDA-12.1.1/lightgbm/lightgbm-4.5.0/src/cuda/cuda_algorithms.cu

Fatbin ptx code:
================
arch = sm_80
code version = [8,1]
host = linux
compile_size = 64bit
compressed
identifier = /tmp/vsc48506/easybuild/build/LightGBM/4.5.0/foss-2023a-CUDA-12.1.1/lightgbm/lightgbm-4.5.0/src/io/cuda/cuda_column_data.cu
ptxasOptions = --compile-only  --generate-line-info

Fatbin elf code:
================
arch = sm_80
code version = [1,7]
host = linux
compile_size = 64bit
compressed
identifier = /tmp/vsc48506/easybuild/build/LightGBM/4.5.0/foss-2023a-CUDA-12.1.1/lightgbm/lightgbm-4.5.0/src/io/cuda/cuda_column_data.cu

Fatbin ptx code:
================
arch = sm_80
...

so in fact the ptx is there for individual compiled objects (and one could infer that if it is there for one, then it was probably enabled in general).

[ocaisa]

@casparvl I think this is maybe an easy fix (add the -all option to cuobjdump), I'm just not sure if the regex can handle well the additional (pretty repetitive) output.