Why can't I use my CPU?

[Papill0n]

I downloaded deepfacelab 1.0 opencl 01-11-2020 and until a few days ago I was able to use my cpu (i5-9600k)

For some days now, I don't understand why, when I do the training dfl automatically selects the GPU 9800gt 1 gb.

Obviously 1 gb of memory are very few, so the software notifies me of an error message. <<python-3.6.8\lib\site-packages\plaidml\library.py", line 131, in raise_last_status raise self.last_status() plaidml.exceptions.Unknown: Unable to map memory: CL_MEM_OBJECT_ALLOCATION_FAILURE)>> <<(ERROR:plaidml:Unable to map memory:CL_MEM_OBJECT_ALLOCATION_FAILURE Error: Unable to map memory: CL_MEM_OBJECT_ALLOCATION_FAILURE Traceback (most recent call last)>>

How can I reselect my CPU? When I start the PC in safe mode and start training, the software automatically selects the CPU and everything works well.

Windows 10 64bit
Nvidia 9800GT 1GB
I5-9600k
RAM 16 GB
Python 3.6.8
All versions visual c++
numpy==1.17.0
h5py==2.9.0
Keras==2.2.4
opencv-python==4.1.0.25
tensorflow==1.12.0
plaidml==0.6.0
plaidml-keras==0.5.0
scikit-image
tqdm
ffmpeg-python==0.1.17
torch===1.3.1+cpu -f https://download.pytorch.org/whl/torch_stable.html
torchvision===0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

[Flova]

In most cases the CPU is much slower than a GPU, so the GPU is preferred.
In Linux you set the environment variable CUDA_VISIBLE_DEVICES.

Example (Sets training on CPU):

CUDA_VISIBLE_DEVICES="" <your training command>

Example (Sets training on GPU X, where X is the GPU ID, starting at 0):

CUDA_VISIBLE_DEVICES="X" <your training command>

For Windows you need to do some research, how env variables work.

[Flova]

A quick google search resulted in the following command for Windows systems:

set CUDA_VISIBLE_DEVICES="" & <your training command>

[Papill0n]

Thank you for your help @Flova . I tried using the command (in windows 10 prompt set CUDA_VISIBLE_DEVICES="0" OR set CUDA_VISIBLE_DEVICES="1" OR set CUDA_VISIBLE_DEVICES="2" OR set CUDA_VISIBLE_DEVICES="3">) but it didn't work. None of the versions of deepfacelab work on my pc (cuda, OpenCL, nvidia).
Do you have any other tips to suggest to me?
Is it possible to use my video card by bypassing the error message regarding the low memory of the video card?

Running trainer.
Loading model...
Press enter in 2 seconds to override model settings.Using plaidml.keras.backend backend.
INFO:plaidml:Opening device "opencl_nvidia_geforce_9800_gt.0"
INFO:plaidml:Analyzing Ops: 63 of 158 operations complete
INFO:plaidml:Analyzing Ops: 137 of 158 operations complete
ERROR:plaidml:unable to run OpenCL kernel: CL_OUT_OF_RESOURCES
INFO:plaidml:Analyzing Ops: 73 of 158 operations complete
Running On
Device index: 0
Name: NVIDIA Corporation GeForce 9800 GT (OpenCL)
VRAM: 1GB
ERROR:plaidml:Unable to map memory: CL_MEM_OBJECT_ALLOCATION_FAILURE
Error: Unable to map memory: CL_MEM_OBJECT_ALLOCATION_FAILURE
Traceback (most recent call last):
\DeepFaceLab_OpenCL_internal\DeepFaceLab\mainscripts\Trainer.py", line 111, in trainerThread
iter, iter_time = model.train_one_iter()
\DeepFaceLab_OpenCL_internal\DeepFaceLab\models\ModelBase.py", line 507, in train_one_iter
losses = self.onTrainOneIter(sample, self.generator_list)
\DeepFaceLab_OpenCL_internal\DeepFaceLab\models\Model_H64\Model.py", line 88, in onTrainOneIter
total, loss_src_bgr, loss_src_mask, loss_dst_bgr, loss_dst_mask = self.ae.train_on_batch( [warped_src, target_src_full_mask, warped_dst, target_dst_full_mask], [target_src, target_src_full_mask, target_dst, target_dst_full_mask] )
\DeepFaceLab_OpenCL_internal\python-3.6.8\lib\site-packages\keras\engine\training.py", line 1216, in train_on_batch
self._make_train_function()
\DeepFaceLab_OpenCL_internal\python-3.6.8\lib\site-packages\keras\engine\training.py", line 509, in _make_train_function
loss=self.total_loss)
\DeepFaceLab_OpenCL_internal\DeepFaceLab\nnlib\nnlib.py", line 1075, in get_updates
ms = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]
\DeepFaceLab_OpenCL_internal\DeepFaceLab\nnlib\nnlib.py", line 1075, in
ms = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]
\DeepFaceLab_OpenCL_internal\python-3.6.8\lib\site-packages\plaidml\keras\backend.py", line 1676, in zeros
return constant(0.0, shape=shape, dtype=dtype, name=_prepend_name_scope(name, 'zeros'))
\DeepFaceLab_OpenCL_internal\python-3.6.8\lib\site-packages\plaidml\keras\backend.py", line 432, in constant
return variable(np_value, dtype=dtype, name=_prepend_name_scope(name, 'constant'))
\DeepFaceLab_OpenCL_internal\python-3.6.8\lib\site-packages\plaidml\keras\backend.py", line 1668, in variable
with tensor.mmap_discard(ctx) as view:
File "contextlib.py", line 81, in enter
\DeepFaceLab_OpenCL_internal\python-3.6.8\lib\site-packages\plaidml_init.py", line 1239, in mmap_discard
mapping = lib().plaidml_map_buffer_discard(ctx, self.buffer)
\DeepFaceLab_OpenCL_internal\python-3.6.8\lib\site-packages\plaidml_init.py", line 764, in _check_err
self.raise_last_status()
\DeepFaceLab_OpenCL_internal\python-3.6.8\lib\site-packages\plaidml\library.py", line 131, in raise_last_status
raise self.last_status()
plaidml.exceptions.Unknown: Unable to map memory: CL_MEM_OBJECT_ALLOCATION_FAILURE

[Flova]

No problem.
Did you run the environment variable command and the training command both in the same line of the command prompt? If not, do so.
Use the CUDA Version of the software. By hiding the CUDA devices, the training should move to the CPU.
In your case, you want all CUDA devices to be hidden, so you should set CUDA_VISIBLE_DEVICES="" instead of CUDA_VISIBLE_DEVICES="0" or some other GPU ID.
It is probably not so useful/realistic to use the GPU if the memory requirements are over its capabilities. Even if it would be possible, accessing the system ram from the GPU would be a huge bottleneck.

[Papill0n]

Hi, I started the Command Prompt with admin privilege and I wrote set cuda_device_"" OR set cuda_device_"" 6) train H64.bat , in the c: \ DeepFaceLab_OpenCL_01-11-2020 folder,in the c: \ DeepFaceLab_1.0_CUDA_build_01-11-2020 folder, and in C:\ DeepFaceLab_NVIDIA_build_02_03_2020 folder, but nothing happens after writing these commands. Maybe I should edit the plaidml.jeras.backend file, but I don't understand where in the code. When I try to start "6) train H64.bat" in safe mode it works correctly with the cpu.

[Flova]

I am not so familiar with the windows. But in your case it sounds like you want to run:
set CUDA_VISIBLE_DEVICES="" & 6) train H64.bat in the CUDA version. The & and the case sensitivity are important!

[Papill0n]

Thanks again for your help @Flova . It finally works!
If I open the Command Prompt with admin privilege and I write set CUDA_VISIBLE_DEVICES="" & 6) train H64.bat , in the c:\DeepFaceLab_OpenCL_01-11-2020 folder,in the c:\DeepFaceLab_1.0_CUDA_build_01-11-2020 folder, and in C:\DeepFaceLab_NVIDIA_build_02_03_2020 folder, I'm getting the following error:

"6)" is not recognized as an internal or external command, operable program or batch file.

To fix it I had to rename the file "6) train H64.bat" to "trainH64.bat". I add that only the OpenCL version works with my CPU. Cuda and Nvidia don't work.

[Papill0n]

DeepFaceLab OpenCL now works perfectly with the CPU.
DeepFaceLab NVIDIA and DeepFaceLab CUDA still don't work and I don't understand why.
I report the errors I get when I try to use NVIDIA and CUDA.

NVIDIA ERRORS
[n] Write debug images to aligned_debug? ( y/n ) : y
Extracting faces...
Exception: Traceback (most recent call last):
File "C:\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('pywrap_tensorflow_internal', fp, pathname, description)
File "imp.py", line 243, in load_module
File "imp.py", line 343, in load_dynamic
ImportError: DLL load failed: Impossibile trovare la procedura specificata.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\joblib\SubprocessorBase.py", line 62, in subprocess_run
self.on_initialize(client_dict)
File "C:\DeepFaceLab_NVIDIA_internal\DeepFaceLab\mainscripts\Extractor.py", line 66, in on_initialize
nn.initialize (device_config)
File "C:\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\leras\nn.py", line 123, in initialize
import tensorflow as tf
File "C:\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow_init.py", line 24, in
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
File "C:\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python_init.py", line 49, in
from tensorflow.python import pywrap_tensorflow
File "C:\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 74, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "C:\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\DeepFaceLab_NVIDIA_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "imp.py", line 243, in load_module
File "imp.py", line 343, in load_dynamic
ImportError: DLL load failed: Impossibile trovare la procedura specificata.
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
Traceback (most recent call last):
File "C:\DeepFaceLab_NVIDIA_internal\DeepFaceLab\main.py", line 350, in
arguments.func(arguments)
File "C:\DeepFaceLab_NVIDIA_internal\DeepFaceLab\main.py", line 41, in process_extract
force_gpu_idxs = [ int(x) for x in arguments.force_gpu_idxs.split(',') ] if arguments.force_gpu_idxs is not None else None,
File "C:\DeepFaceLab_NVIDIA_internal\DeepFaceLab\mainscripts\Extractor.py", line 758, in main
device_config=device_config).run()
File "C:\DeepFaceLab_NVIDIA_internal\DeepFaceLab\core\joblib\SubprocessorBase.py", line 217, in run
raise Exception ( "Unable to start subprocesses." )
Exception: Unable to start subprocesses.

==============================================================
CUDA ERRORS
Performing 1st pass...
Running on CPU0.
Exception: Traceback (most recent call last):
File "C:\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "imp.py", line 243, in load_module
File "imp.py", line 343, in load_dynamic
ImportError: DLL load failed: Impossibile trovare la procedura specificata.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\DeepFaceLab_CUDA_internal\DeepFaceLab\joblib\SubprocessorBase.py", line 59, in _subprocess_run
self.on_initialize(client_dict)
File "C:\DeepFaceLab_CUDA_internal\DeepFaceLab\mainscripts\Extractor.py", line 78, in on_initialize
nnlib.import_all (device_config)
File "C:\DeepFaceLab_CUDA_internal\DeepFaceLab\nnlib\nnlib.py", line 1613, in import_all
nnlib.import_keras(device_config)
File "C:\DeepFaceLab_CUDA_internal\DeepFaceLab\nnlib\nnlib.py", line 221, in import_keras
nnlib.import_tf(device_config)
File "C:\DeepFaceLab_CUDA_internal\DeepFaceLab\nnlib\nnlib.py", line 192, in import_tf
import tensorflow as tf
File "C:\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow_init.py", line 24, in
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
File "C:\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python_init.py", line 49, in
from tensorflow.python import pywrap_tensorflow
File "C:\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 74, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "C:\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\DeepFaceLab_CUDA_internal\python-3.6.8\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "imp.py", line 243, in load_module
File "imp.py", line 343, in load_dynamic
ImportError: DLL load failed: Impossibile trovare la procedura specificata.
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.