scvi integration not plotting in interactive session on HPC

[JarneBelien]

panpipes integration keeps terminating for me at the plotting stage when I run it on an interactive session on a cluster.
The error is always the same indexerror:

The issue is not dataset-specific as @deevdevil88 was able to run integration fine without any issues in her environment.
The issue also does not seem scvi-tools version related as my error is reproducible with both v1.1.2 and v1.1.1.

The rna_scvi.log file shows the model does seem to converge (progress bar goes up to 100%) but no figures are outputted to the 'figures' folder.

[bio-la]

Hi @JarneBelien and thanks for using panpipes!
could you please share the scvi log file? also, we should have a checkpoint to save the model just before that command, could you please share that as well?

from the screenshot it seems you're running an older version of panpipes.

[JarneBelien]

Hi @bio-la!

I now cloned the github repo and installed the nightly version - tried to run again in an interactive session and now the model failed to converge at the very last moment. I've attached the scvi log file here.
rna_scvi.log
I will send the model through email as its format is not supported on this website.

[bio-la]

thanks for sharing the log file!
it seems that the model performance is not improving so the early_stopping = True and the default patience are stopping the computation. this is not a panpipes issue per se so i will open an issue on the scvi repo to help with troubleshooting.

Lastly, if you have GPUs available, you can activate panpipes environment and install a CUDA-enabled jaxlib manually while on an interactive session on said GPU (see this

I'm testing a few things to ensure that the CUDA-enabled jax is correctly installed without having to manually fix the environment afterwards!

[JarneBelien]

Thanks!
I did the CUDA-enabled jaxlib fix that was discussed in another issue here and was able to run some of the trials on an interactive session with GPU and however faster, it did not make the model converge.
I find it peculiar that with the same dataset, the model does converge when @deevdevil88 ran it in her env. This was however not in an interactive session.