[ITensors] [BUG] DMRG breaks when using cuTENSOR backend

[iagoleal]

Description of bug
Hello, I tried to run DMRG using the cuTENSOR.jl backend but stumbled into a method error.
There is no problem when running with just CUDA.jl.

Minimal code demonstrating the bug or unexpected behavior

Minimal runnable code

using ITensors, ITensorMPS

using CUDA, cuTENSOR

dim   = 2
T     = Float32
sites = siteinds("Qubit", dim)
os    = OpSum{T}()

for i in 1:dim
  os .+= (1, "Proj1", i)
end

H    = MPO(T, os, sites)
psi0 = ITensors.MPS(T, sites, "+")

energy, psi = ITensors.dmrg(cu(H), cu(psi0); nsweeps = 2)

Expected output or behavior

The expected behavior was for the code to run with no exception, as happens when using only CUDA.jl.

Actual output or behavior

The code throws an error with the output below.

Output of minimal runnable code

ERROR: LoadError: MethodError: no method matching CuTensor(::CuArray{Float32, 2, CUDA.Mem.DeviceBuffer}, ::Vector{Int64})
Closest candidates are:
  CuTensor(::CuArray{T, N, B} where B, !Matched::Vector{Char}) where {T<:Number, N} at /home/x-ilealdefreit/.julia/packages/cuTENSOR/F7PCr/src/tensor.jl:17
  CuTensor(::CuArray{T, N, B} where B, !Matched::Vector{var"#s14"} where var"#s14"<:AbstractChar) where {T<:Number, N} at /home/x-ilealdefreit/.julia/packages/cuTENSOR/F7PCr/src/tensor.jl:14
Stacktrace:
  [1] contract!(exposedR::NDTensors.Expose.Exposed{CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}}, labelsR::Tuple{Int64, Int64}, exposedT1::NDTensors.Expose.Exposed{CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}}, labelsT1::Tuple{Int64, Int64}, exposedT2::NDTensors.Expose.Exposed{CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}}, labelsT2::Tuple{Int64, Int64}, α::Bool, β::Bool)
    @ NDTensors.NDTensorscuTENSORExt ~/.julia/packages/NDTensors/gdQSG/ext/NDTensorscuTENSORExt/contract.jl:42
  [2] contract!(exposedR::NDTensors.Expose.Exposed{CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}}, labelsR::Tuple{Int64, Int64}, exposedT1::NDTensors.Expose.Exposed{CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}}, labelsT1::Tuple{Int64, Int64}, exposedT2::NDTensors.Expose.Exposed{CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}}, labelsT2::Tuple{Int64, Int64})
    @ NDTensors.NDTensorscuTENSORExt ~/.julia/packages/NDTensors/gdQSG/ext/NDTensorscuTENSORExt/contract.jl:25
  [3] _contract!!(output_tensor::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labelsoutput_tensor::Tuple{Int64, Int64}, tensor1::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labelstensor1::Tuple{Int64, Int64}, tensor2::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labelstensor2::Tuple{Int64, Int64}, α::Int64, β::Int64)
    @ NDTensors ~/.julia/packages/NDTensors/gdQSG/src/tensoroperations/generic_tensor_operations.jl:143
  [4] _contract!!(output_tensor::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labelsoutput_tensor::Tuple{Int64, Int64}, tensor1::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labelstensor1::Tuple{Int64, Int64}, tensor2::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labelstensor2::Tuple{Int64, Int64})
    @ NDTensors ~/.julia/packages/NDTensors/gdQSG/src/tensoroperations/generic_tensor_operations.jl:131
  [5] contract!!(output_tensor::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labelsoutput_tensor::Tuple{Int64, Int64}, tensor1::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labelstensor1::Tuple{Int64, Int64}, tensor2::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labelstensor2::Tuple{Int64, Int64}, α::Int64, β::Int64)
    @ NDTensors ~/.julia/packages/NDTensors/gdQSG/src/tensoroperations/generic_tensor_operations.jl:219
  [6] contract!!(output_tensor::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labelsoutput_tensor::Tuple{Int64, Int64}, tensor1::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labelstensor1::Tuple{Int64, Int64}, tensor2::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labelstensor2::Tuple{Int64, Int64})
    @ NDTensors ~/.julia/packages/NDTensors/gdQSG/src/tensoroperations/generic_tensor_operations.jl:188
  [7] contract(tensor1::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labelstensor1::Tuple{Int64, Int64}, tensor2::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labelstensor2::Tuple{Int64, Int64}, labelsoutput_tensor::Tuple{Int64, Int64})
    @ NDTensors ~/.julia/packages/NDTensors/gdQSG/src/tensoroperations/generic_tensor_operations.jl:113
  [8] contract(::Type{NDTensors.CanContract{NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}}}, tensor1::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labels_tensor1::Tuple{Int64, Int64}, tensor2::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, labels_tensor2::Tuple{Int64, Int64})
    @ NDTensors ~/.julia/packages/NDTensors/gdQSG/src/tensoroperations/generic_tensor_operations.jl:91
  [9] contract
    @ ~/.julia/packages/SimpleTraits/l1ZsK/src/SimpleTraits.jl:331 [inlined]
 [10] _contract(A::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}, B::NDTensors.DenseTensor{Float32, 2, Tuple{Index{Int64}, Index{Int64}}, NDTensors.Dense{Float32, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}})
    @ ITensors ~/.julia/packages/ITensors/23eRw/src/tensor_operations/tensor_algebra.jl:3
 [11] _contract(A::ITensor, B::ITensor)
    @ ITensors ~/.julia/packages/ITensors/23eRw/src/tensor_operations/tensor_algebra.jl:9
 [12] contract(A::ITensor, B::ITensor)
    @ ITensors ~/.julia/packages/ITensors/23eRw/src/tensor_operations/tensor_algebra.jl:74
 [13] *(A::ITensor, B::ITensor)
    @ ITensors ~/.julia/packages/ITensors/23eRw/src/tensor_operations/tensor_algebra.jl:61
 [14] orthogonalize!(M::MPS, j::Int64; maxdim::Nothing, normalize::Nothing)
    @ ITensors.ITensorMPS ~/.julia/packages/ITensors/23eRw/src/lib/ITensorMPS/src/abstractmps.jl:1642
 [15] orthogonalize!
    @ ~/.julia/packages/ITensors/23eRw/src/lib/ITensorMPS/src/abstractmps.jl:1604 [inlined]
 [16] #orthogonalize!#337
    @ ~/.julia/packages/ITensors/23eRw/src/lib/ITensorMPS/src/abstractmps.jl:1655 [inlined]
 [17] orthogonalize!
    @ ~/.julia/packages/ITensors/23eRw/src/lib/ITensorMPS/src/abstractmps.jl:1655 [inlined]
 [18] dmrg(PH::ProjMPO, psi0::MPS, sweeps::Sweeps; which_decomp::Nothing, svd_alg::Nothing, observer::NoObserver, outputlevel::Int64, write_when_maxdim_exceeds::Nothing, write_path::String, eigsolve_tol::Float64, eigsolve_krylovdim::Int64, eigsolve_maxiter::Int64, eigsolve_verbosity::Int64, eigsolve_which_eigenvalue::Symbol, ishermitian::Bool)
    @ ITensors.ITensorMPS ~/.julia/packages/ITensors/23eRw/src/lib/ITensorMPS/src/dmrg.jl:192
 [19] dmrg
    @ ~/.julia/packages/ITensors/23eRw/src/lib/ITensorMPS/src/dmrg.jl:176 [inlined]
 [20] #dmrg#499
    @ ~/.julia/packages/ITensors/23eRw/src/lib/ITensorMPS/src/dmrg.jl:28 [inlined]
 [21] dmrg
    @ ~/.julia/packages/ITensors/23eRw/src/lib/ITensorMPS/src/dmrg.jl:22 [inlined]
 [22] #dmrg#505
    @ ~/.julia/packages/ITensors/23eRw/src/lib/ITensorMPS/src/dmrg.jl:388 [inlined]
 [23] top-level scope
    @ ~/projects/cutensor-debug/debug.jl:17
in expression starting at /home/x-ilealdefreit/projects/cutensor-debug/debug.jl:17

Version information

• Output from versioninfo():

julia> versioninfo()
Julia Version 1.6.2
Commit 1b93d53fc4* (2021-07-14 15:36 UTC)
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  CPU: AMD EPYC 7543 32-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, generic)
Environment:
  JULIA_HOME = /apps/spack/anvilgpu/apps/julia/1.6.2-gcc-8.4.1-gdxcaek
  RCAC_JULIA_VERSION = 1.6.2
  RCAC_JULIA_ROOT = /apps/spack/anvilgpu/apps/julia/1.6.2-gcc-8.4.1-gdxcaek

• Output from using Pkg; Pkg.status("ITensors"):

julia> using Pkg; Pkg.status("ITensors")
      Status `~/projects/cutensor-debug/Project.toml`
  [9136182c] ITensors v0.6.18

Additionally, I am running it with CUDA 11.2.2.

[mtfishman]

Thanks for the report, it looks like it shouldn't be difficult to fix. It looks like we are just converting to the cuTENSOR.CuTensor type using integers instead of characters as dimension labels. @kmp5VT can you take a look?

[kmp5VT]

Hello, @mtfishman yes I can look into this more. From a first glance, I would guess the issue is related to the Julia 1.6. Using an Julia version below 1.10 will grab outdated versions of CUDA.jl and cuTENSOR.jl and in since these earlier versions there has been a syntax change in the CuTensor constructor.
@iagoleal Would it be possible for you to use a more up to date version of Julia?

[iagoleal]

Hi @kmp5VT. Unfortunately, I am stuck with Julia 1.6. The cluster where I am running this code only has support for the LTS version on its GPU nodes.

[emstoudenmire]

Would it work for you to install Julia locally within your cluster home folder ? We have a tutorial on how to do that (basically it's the same as installing Julia from binary on your own personal machine). It might work just as well as using the cluster-wide Julia install except that then you could use a newer version of Julia.

Note that whichever install you use, Julia just installs your packages into your home folder anyway (underneath the ~/.julia/ folder).

[mtfishman]

I see, thanks for pointing that out @kmp5VT, good point about the Julia version.

@iagoleal, this isn't really up to us unfortunately, the latest package versions of the entire GPU ecosystem in Julia only works on more recent versions of Julia (it depends heavily on developments in the Julia compiler), so it is best to use the latest stable release (Julia 1.10) if possible, for example by installing Julia yourself as Miles suggested. We also plan to only support Julia 1.10 and above once it becomes LTS (I'm not sure that's official yet but I've heard that will happen when Julia 1.11 is released, which should be pretty soon).

I'm closing this since your original code should work if you use a more recent Julia version.

[mtfishman]

@iagoleal if you need more help using a newer version of Julia on your cluster, please start a post either in the ITensor forum (https://itensor.discourse.group) or the Julia forum (https://discourse.julialang.org).

[iagoleal]

Hello, thanks for all your help! I just followed the tutorial sent by Miles and it works with Julia 1.10.