Suggestion: Enable opt-in L1 caching via nvcc-wrapper

[mcarilli]

Hi Kokkos Team,

During my compilation adventures I tried to opt-into L1 caching on Kepler using the compiler flag -Xptxas="-dlcm=ca" as described here.

However, simply adding -Xptxas="-dlcm=ca" to CXXFLAGS in the makefile failed, because apparently CXXFLAGS is passed to both the host compiler and nvcc, and the host compiler did not recognize the option.

After some poking around, I fixed the problem by appending -Xptxas="-dlcm=ca" to the list of "known nvcc args" at line 110 of nvcc_wrapper.

old:

  #Handle known nvcc args
  -gencode*|--dryrun|--verbose|--keep|--keep-dir*|-G|--relocatable-device-code*|-lineinfo|-expt-extended-lambda|--resource-usage)
    cuda_args="$cuda_args $1"

new:

  #Handle known nvcc args
  -gencode*|--dryrun|--verbose|--keep|--keep-dir*|-G|--relocatable-device-code*|-lineinfo|-expt-extended-lambda|--resource-usage|-Xptxas="-dlcm=ca")
    cuda_args="$cuda_args $1"

I think you should consider adding -Xptxas="-dlcm=ca" to nvcc_wrapper permanently, since it is a small change that adds functionality and is unlikely to break anything. You may also wish to add its sibling that disables L1 caching, -Xptxas="-dlcm=cg".

Hope this helps,
Michael

[crtrott]

I can look into adding something for this. Btw. nvcc_wrapper is now maintained in its own little repository (kokkos/nvcc_wrapper), so I will first add it there before pulling it into Kokkos.