Spacy 3.5 works on Colab with no hardware acceleration but not on Standard GPU

[pipkin40]

!pip install -U pip setuptools wheel
!pip install -U spacy==3.4.1
!pip install -U spacy[cuda113]
!pip install benepar
!pip install shap
!pip install docx2txt
!python -m spacy download en_core_web_trf
from google.colab import files
from google.colab import drive
drive.mount('/content/drive')
import nltk
import nltk.data
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
nltk.download('omw-1.4')
sent_detector = nltk.data.load('tokenizers/punkt/english.pickle')
from nltk.corpus import wordnet
import spacy as sp
nlp = sp.load('en_core_web_trf')
!python -m spacy info --markdown
#2-13-2023

Works on unaccelerated runtime. Was working on standard GPU until few days ago. Now does not work on standard GPU (many attempts) or premium GPU (one attempt). Also works on TPU. We really need standard GPU for some tasks. Error message is that Spacy cannot be found. Sometimes error message re UTF-8!--

What web browser you are using
(Chrome, Firefox, Safari, etc.) Chrome

Additional context
Link to a minimal, public, self-contained notebook that reproduces this issue.

• Share the file using your GitHub account using File > Save a copy as a GitHub Gist.
• or Share Drive notebooks using the Share button then 'Get Shareable Link'.

[metrizable]

@pipkin40 Thanks for your report and thanks for using Colab.

I've reproduced your error and isolated the cause of the issue. There is an upstream bug report for this already. Reducing your code to a couple relevant lines reveals the issue.

Importing spacy on GPU:

Importing spacy on CPU:

On GPU, importing spacy actually invokes cupy as it's available, which gets closer to the culprit.

Invoking cupy on GPU:

Issue NVIDIA/cuda-python#29 has already been filed, and in that issue, there is reference to an upstream bug with Nvidia that you can follow: https://developer.nvidia.com/nvidia_bug/3833924. In the meantime, in explosion/spaCy#11909, there is reference to possible work-arounds that may work for you, although YMMV.