Torch not compiled with CUDA enabled #91
Comments
|
Python Launcher( But torch supports Python 3.8, pip should install CUDA enabled torch. Does your PC have CUDA and CUDA supported GPU? "worddumb-libs-*" folders won't be deleted by plugin, you may want to delete the unused one to save space or let plugin reinstall dependencies. You may also need to run "py -m pip cache purge" to clear the pip cache, just in case pip has a CUDA disabled torch cache. |
|
"please uninstall all Python versions and reinstall Python 3.10 and select the default option("Install Now" button) in the Python installer." Done. "Does your PC have CUDA" Yes: "and CUDA supported GPU?" "You may also need to run "py -m pip cache purge" to clear the pip cache, just in case pip has a CUDA disabled torch cache." Done. Still, though Python 3.10 is invoked now, the error persists: |
|
How about running the this command |
|
The latest CUDA release is 11.8 but yours is 12.0. I find a relevant discussion at here and they said |
|
"How about running the this command manually" Done (without understanding much): "the correct command to find CUDA version" Right, 11.8 as you said: Alas the problem still persists: BTW Thanks a lot for helping but since that seems to be a CUDA installation problem on my system (more than a WordDumb one) feel free to close if you get tired. |
|
Well... The torch package for Windows on pypi.org is the CPU build version, indeed "not compiled with CUDA enabled". In order to install the CUDA build version, you'd have to run this command: |
|
I add a commit e7ed60d to install CUDA enabled torch. But you'll have to delete torch package( |
|
Reporting after doing fresh install of WordDumb... "you'd have to run this command" Done: Create X-Ray selecting "run spaCy with GPU" now gives a new "charmap' codec can't decode" error: In addition, Create X-Ray deselecting "run spaCy with GPU" is now also broken, giving a "object has no attribute 'format'" error: "not sure if the 11.7 version would work with 11.8." ...hmm, perhaps something else. |
|
a6f6cc5 should fix the UnicodeDecodeError error. That I add another commit: 3c797e8, maybe this could fix the second error. |
|
^ Updated parse_job.py, Sorry, made no difference: |
|
At least the "UnicodeDecodeError" error was fixed and X-Ray was created without error when GPU is used? The "AttributeError" is a known error because calibre sets the optimize level to 2 and that removes all doc strings. I need to figure out why the "transformers" library is called when the "use GPU" option is not selected. |
|
"At least the "UnicodeDecodeError" error was fixed and X-Ray was created without error when GPU is used?" I wasn't clear: No X-Ray got created with or without using GPU. The above was run without enabling GPU usage. I shouldn't have snipped: When enabling GPU I still get the same "UnicodeDecodeError": |
|
The code line numbers in the error message indicate you're running outdated code, please install the zip file from the latest GitHub Actions Artifacts. I suspect both errors could be solved. |
|
Mybad, instead of patching manually, I freshly reinstalled the plugin from the latest artifact worddumb-117d064104bb761e30442d83c7f15fe2fbf35993.zip With the GPU model: now getting 'lxml' ModuleNotFoundError With the CPU model: still getting object has no attribute 'format' Sorry for my incompetence with GitHub, still learning. |
|
3ad53d5 would fix the Still don't have a clue about the And you can use |
|
Qapla', lxml is now found and X-Ray has been created with GPU! "What about delete those "spacy_transformers" and "transformers" packages" Sure, I will try that and report (but later). |
|
^ OK, time to show my pip ignorance now: Let's see what is in there: Why would it be as simple as: |
|
I just manually deleted transformers, transformers-4.21.3.dist-info, spacy_transformers, and spacy_transformers-1.1.8.dist-info directories and the X-Ray creation (via CPU) now completes successfully. No more AttributeError error. "...this would cause error when "run with GPU" is enabled since these dependencies are missing" indeed. |
|
En passant I had this "conversation" with this ChatGPT OpenAI bot. My questions are in bold. I am still amazed: how do i uninstall a python package? To uninstall a python package, you can use the pip uninstall command. For example, to uninstall the numpy package, you can use the following command: pip uninstall numpy This will remove the package from your system. Keep in mind that you may need to use sudo in front of the pip command if you are using a version of Python that is installed globally on your system, rather than in a virtual environment. sudo pip uninstall numpy You can also use the conda command if you are using the Anaconda distribution of Python. To uninstall a package using conda, you can use the conda remove command. For example, to uninstall the numpy package, you can use the following command: conda remove numpy This will remove the package from your system. Note that you may need to use the --force option if you are removing a package that is required by other packages on your system. For example: conda remove numpy --force I hope this helps! Let me know if you have any other questions. how do I do it with pip? To uninstall a python package using pip, you can use the pip uninstall command. For example, to uninstall the numpy package, you can use the following command: pip uninstall numpy This will remove the package from your system. Keep in mind that you may need to use sudo in front of the pip command if you are using a version of Python that is installed globally on your system, rather than in a virtual environment. sudo pip uninstall numpy You can also use the --yes or -y option to automatically answer yes to any prompts that the pip command may ask you. For example: pip uninstall numpy --yes Or: pip uninstall numpy -y I hope this helps! Let me know if you have any other questions. why does pip list a package but says it is not installed when I run pip uninstall? If pip lists a package but says it is not installed when you run pip uninstall, it is possible that the package is installed in a different location or in a different Python environment. When you run pip list, it shows you the packages that are installed in the current environment. If you are using a virtual environment, then pip list will only show you the packages that are installed in that environment. On the other hand, when you run pip uninstall, it will try to uninstall the package from the current environment. If the package is not installed in the current environment, then pip will return an error saying that the package is not installed. To fix this, you can try activating the correct Python environment using the activate command (if you are using virtual environments) and then running pip uninstall again. For example, if you are using the virtualenv package to manage your virtual environments, you can activate the correct environment using the following command: source /path/to/venv/bin/activate Replace /path/to/venv with the actual path to your virtual environment. Once the environment is activated, you can run pip uninstall to uninstall the package. Alternatively, you can use the -P or --prefix option with pip uninstall to specify the path to the environment where the package is installed. For example: pip uninstall numpy -P /path/to/venv This will uninstall the numpy package from the environment located at /path/to/venv. I hope this helps! Let me know if you have any other questions. [EDIT: 2 hours later I added] Python transformer and spacy-transformer are two different packages that are not related to each other. The transformer package is a library for natural language processing tasks, specifically for training and evaluating transformer-based models for language understanding tasks. The spacy-transformer package is a plugin for the popular natural language processing library spaCy, which allows users to use transformer-based models trained with the transformer package in their spaCy pipelines. As for the "decode error" you mentioned, it is not clear what you are referring to or what the error message is. However, it is worth noting that the transformer and spacy-transformer packages are designed to work with GPUs (Graphics Processing Units) for efficient and fast training and inference of transformer-based models. If you are using a non-CUDA version of Python, it is possible that you may encounter errors or performance issues when using these packages. It is recommended to use a CUDA-enabled version of Python for optimal performance with these packages. UnicodeDecodeError: 'charmap' codec can't decode byte was the precise error A UnicodeDecodeError with the message 'charmap' codec can't decode byte indicates that there was a problem with trying to decode some text using the character mapping (charmap) codec. This error typically occurs when there is a mismatch between the character encoding of the text and the character map being used to decode it. For example, if the text is encoded using the UTF-8 character encoding, but the character map being used to decode it is for a different encoding, such as ISO-8859-1, this error can occur. To fix this error, you need to make sure that the character map being used to decode the text is compatible with the character encoding of the text. You can do this by specifying the correct character encoding when you load or read the text into your program. For example, if your text is encoded using UTF-8, you can specify the utf-8 encoding when you read the text using the open() function in Python, like this: with open("my_text_file.txt", "r", encoding="utf-8") as f: This will ensure that the text is properly decoded using the correct character map (in this case, UTF-8), and the UnicodeDecodeError should no longer occur. |
|
I don't even know the
I don't have any CUDA supported device, but I thought add support of spaCy transformer model would just need to add a few lines of code. I was completely wrong... Thanks for testing my buggy code! |
|
11dd530 should fix |
|
Fixed indeed. Thanks again. |
|
Once transformers releases a new version and spacy-transformers also supports the new transformers version, I can remove the hacks to avoid the |
|
Hi @miiPoP, the latest transformers package fixed the doc string error and I have removed the monkey patch in the master branch so creating X-Ray file with GPU no longer need to run in the system Python interpreter. Could you install the test plugin from GitHub Actions to test the "Run spaCy with GPU" feature? |
|
Sorry, it fails both using the GPU or not `With "Run spaCy with GPU" feature, after a very long time (minutes) it fails with: calibre, version 6.11.0 (win32, embedded-python: True) Starting job: Générant X-Ray pour The Mythical Man-Month The above exception was the direct cause of the following exception: Traceback (most recent call last): Called with args: ((6291, 'AZW3', 'H:\NB\Bibliothèque Calibre\Frederick P. Brooks Jr_\The Mythical Man-Month (6291)\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x000002104F241FC0>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x000002104D82E470>, 'abort': <threading.Event object at 0x00000210505D3040>, 'log': <calibre.utils.logging.GUILog object at 0x00000210505D2F80>} Without "Run spaCy with GPU" feature, it almost immediately gives the error: calibre, version 6.11.0 (win32, embedded-python: True) Starting job: Générant X-Ray pour The Mythical Man-Month Called with args: ((6291, 'AZW3', 'H:\NB\Bibliothèque Calibre\Frederick P. Brooks Jr_\The Mythical Man-Month (6291)\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x0000021011317280>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x00000210113170A0>, 'abort': <threading.Event object at 0x00000210113176D0>, 'log': <calibre.utils.logging.GUILog object at 0x0000021011317790>} ` |
|
Dang, transformers still has some code using the doc string. I restore the monkey patch in the master branch, the first error should be fixed. But I can't make of the second error, it says the one of the Lines 43 to 58 in f1b0d01 I add a please install the new test zip file and start calibre with command |
|
As requested, terminal log: ` C:\Windows\System32>calibre-debug -g Called with args: ((6291, 'AZW3', 'H:\NB\Bibliothèque Calibre\Frederick P. Brooks Jr_\The Mythical Man-Month (6291)\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x00000194F8697D90>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x00000194F8697460>, 'abort': <threading.Event object at 0x00000194F844F760>, 'log': <calibre.utils.logging.GUILog object at 0x00000194F844F670>} ` And Calibre log, with CUDA processing enabled: ` Starting job: Générant X-Ray pour The Mythical Man-Month Called with args: ((6291, 'AZW3', 'H:\NB\Bibliothèque Calibre\Frederick P. Brooks Jr_\The Mythical Man-Month (6291)\The Mythical Man-Month - Frederick P. Brooks Jr_.azw3', <calibre.ebooks.metadata.book.base.Metadata object at 0x00000194F8697D90>, {'spacy': 'en_core_web_', 'wiki': 'en', 'kaikki': 'English', 'gloss': True, 'has_trf': True}), False, True) {'notifications': <queue.Queue object at 0x00000194F8697460>, 'abort': <threading.Event object at 0x00000194F844F760>, 'log': <calibre.utils.logging.GUILog object at 0x00000194F844F670>} During handling of the above exception, another exception occurred: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): ` |
|
The error message says And what's the output when "run with GPU" is disabled? I also notice there are some errors caused by other plugins, but those are probably irrelevant. |
|
My bad (Visual Studio had been de-installed in the last month), after re-installing it both WordDumb flavors work perfectly: Without CUDA: With CUDA: |
Checkboxes
Describe the bug
WordDumb Fail to use GPU to create X-Ray file (when CUDA is disabled everything works fine).
A fresh installation of the plugin installs
worddumb-libs-py3.8, worddumb-libs-py3.10, worddumb-wikimedia
in Calibre plugin directory.
But Python 3.8 is invoked instead of Python 3.10 (which I think is required, per your documentation).
If that is relevant, a separate Python 3.10 directory is in my OS PATH.
System Information
OS: Windows 10 Pro 64bit
Calibre: version 6.9.0 64bit
Python: version 3.10.8 64bit
Plugin: version 3.27.1 (+"space in user name" patch)
Error message
Reproduce steps
Screenshots or videos
No response
The text was updated successfully, but these errors were encountered: