Replies: 2 comments
-
>>> baconator |
Beta Was this translation helpful? Give feedback.
0 replies
-
>>> buxbaum |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
>>> buxbaum
[September 11, 2020, 7:10pm]
Hello, slash
I encountered some issues while finetuning on gpu (Cuda 10.1, CuDNN
7.6). slash
I want to finetune the model using checkpoints from version 0.8.1, and
my custom data. The finetuning is working on cpu but takes many days, so
I decided to use GPU. After typing: slash
python3 DeepSpeech.py slash --n_hidden 2048 slash --checkpoint_dir
german_checkpoints/ slash --epochs 3 slash --train_files
finetuning_data/synthetic_data/my-train.csv slash --dev_files
finetuning_data/synthetic_data/my-dev.csv slash --test_files
finetuning_data/synthetic_data/my_dev.csv slash --learning_rate 0.0001
slash --use_allow_growth true slash --train_cudnn true slash --test_batch_size=128
slash --train_batch_size=128 slash --dev_batch_size=128
I'm getting the following error:
Traceback (most recent call last):
'/home/ubuntu/Desktop/DeepSpeech/training/deepspeech_training/train.py',
line 961, in run_script slash *
'/home/ubuntu/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/absl/app.py',
line 300, in run slash *
'/home/ubuntu/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/absl/app.py',
line 251, in slash _run_main slash *
'/home/ubuntu/Desktop/DeepSpeech/training/deepspeech_training/train.py',
line 933, in main slash *
'/home/ubuntu/Desktop/DeepSpeech/training/deepspeech_training/train.py',
line 523, in train slash *
'/home/ubuntu/Desktop/DeepSpeech/training/deepspeech_training/util/checkpoints.py',
line 132, in load_or_init_graph_for_training slash *
'/home/ubuntu/Desktop/DeepSpeech/training/deepspeech_training/util/checkpoints.py',
line 97, in slash _load_or_init_impl slash *
'/home/ubuntu/Desktop/DeepSpeech/training/deepspeech_training/util/checkpoints.py',
line 70, in slash _load_checkpoint slash *
'/home/ubuntu/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py',
line 324, in new_func slash *
'/home/ubuntu/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/ops/variables.py',
line 1006, in load slash *
value}) slash *
'/home/ubuntu/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py',
line 950, in run slash *
'/home/ubuntu/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py',
line 1173, in slash _run slash *
'/home/ubuntu/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py',
line 1350, in slash _do_run slash *
'/home/ubuntu/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py',
line 1370, in slash _do_call slash *
tensorflow.python.framework.errors_impl.InvalidArgumentError: No
OpKernel was registered to support Op 'CudnnRNNCanonicalToParams'
used by node tower_0/cudnn_lstm/cudnn_lstm/CudnnRNNCanonicalToParams
(defined at
/home/ubuntu/finetuning1/home//Desktop/DeepSpeech/training/deepspeech_training/train.py:128)
with these attrs: slash [input_mode='linear_input', T=DT_FLOAT,
direction='unidirectional', rnn_mode='lstm', seed2=247, seed=4568,
dropout=0, num_params=8 slash ] slash
Registered devices: slash [CPU, XLA_CPU, XLA_GPU slash ] slash
Registered kernels:
Here the pip list in my env:
absl-py 0.10.0 slash
alabaster 0.7.12 slash
alembic 1.4.3 slash
anaconda-client 1.7.2 slash
anaconda-project 0.8.3 slash
appdirs 1.4.4 slash
argh 0.26.2 slash
asn1crypto 1.3.0 slash
astor 0.8.1 slash
astroid 2.4.2 slash
astropy 4.0.1.post1 slash
astunparse 1.6.3 slash
atomicwrites 1.3.0 slash
attrdict 2.0.1 slash
attrs 20.2.0 slash
audioread 2.1.8 slash
autopep8 1.4.4 slash
autovizwidget 0.15.0 slash
Babel 2.8.0 slash
backcall 0.1.0 slash
backports.shutil-get-terminal-size 1.0.0 slash
beautifulsoup4 4.9.1 slash
bitarray 1.2.1 slash
bkcharts 0.2 slash
bleach 3.1.4 slash
bokeh 2.0.1 slash
boto 2.49.0 slash
boto3 1.14.37 slash
botocore 1.17.37 slash
Bottleneck 1.3.2 slash
bs4 0.0.1 slash
cachetools 4.1.1 slash
certifi 2020.6.20 slash
cffi 1.14.2 slash
chardet 3.0.4 slash
click 7.1.1 slash
cliff 3.4.0 slash
cloudpickle 1.3.0 slash
clyent 1.2.2 slash
cmaes 0.6.1 slash
cmd2 1.3.9 slash
colorama 0.4.3 slash
colorlog 4.2.1 slash
contextlib2 0.6.0.post1 slash
cryptography 2.8 slash
cycler 0.10.0 slash
Cython 0.29.15 slash
cytoolz 0.10.1 slash
dask 2.14.0 slash
decorator 4.4.2 slash
deepspeech-gpu 0.8.2 slash
slash *deepspeech-training 0.9.0a3 slash *
defusedxml 0.6.0 slash
diff-match-patch 20181111 slash
distributed 2.14.0 slash
docutils 0.15.2 slash
ds-ctcdecoder 0.9.0a3 slash
entrypoints 0.3 slash
environment-kernels 1.1.1 slash
et-xmlfile 1.0.1 slash
fastcache 1.1.0 slash
filelock 3.0.12 slash
flake8 3.7.9 slash
Flask 1.1.1 slash
fsspec 0.7.1 slash
future 0.18.2 slash
gast 0.3.3 slash
gevent 1.4.0 slash
glob2 0.7 slash
gmpy2 2.0.8 slash
google-auth 1.20.1 slash
google-auth-oauthlib 0.4.1 slash
google-pasta 0.2.0 slash
greenlet 0.4.15 slash
grpcio 1.32.0 slash
h5py 2.10.0 slash
hdijupyterutils 0.15.0 slash
HeapDict 1.0.1 slash
horovod 0.19.5 slash
html5lib 1.0.1 slash
hypothesis 5.8.3 slash
idna 2.10 slash
imageio 2.8.0 slash
imagesize 1.2.0 slash
importlib-metadata 1.7.0 slash
intervaltree 3.0.2 slash
ipykernel 5.1.4 slash
ipyparallel 6.2.4 slash
ipython 7.13.0 slash
ipython-genutils 0.2.0 slash
ipywidgets 7.5.1 slash
isort 4.3.21 slash
itsdangerous 1.1.0 slash
jdcal 1.4.1 slash
jedi 0.15.2 slash
jeepney 0.4.3 slash
Jinja2 2.11.1 slash
jmespath 0.9.4 slash
joblib 0.16.0 slash
json5 0.9.4 slash
jsonschema 3.2.0 slash
jupyter 1.0.0 slash
jupyter-client 6.1.2 slash
jupyter-console 6.1.0 slash
jupyter-core 4.6.3 slash
jupyterlab 1.2.6 slash
jupyterlab-server 1.1.0 slash
Keras 2.3.0 slash
Keras-Applications 1.0.8 slash
Keras-Preprocessing 1.1.2 slash
keyring 21.1.1 slash
kiwisolver 1.1.0 slash
lazy-object-proxy 1.4.3 slash
libarchive-c 2.8 slash
librosa 0.8.0 slash
lief 0.9.0 slash
llvmlite 0.31.0 slash
locket 0.2.0 slash
lxml 4.5.0 slash
Mako 1.1.3 slash
Markdown 3.2.2 slash
MarkupSafe 1.1.1 slash
matplotlib 3.1.3 slash
mccabe 0.6.1 slash
mistune 0.8.4 slash
mkl-fft 1.0.15 slash
mkl-random 1.1.0 slash
mkl-service 2.3.0 slash
mock 4.0.1 slash
more-itertools 8.2.0 slash
mpmath 1.1.0 slash
msgpack 1.0.0 slash
multipledispatch 0.6.0 slash
nb-conda 2.2.1 slash
nb-conda-kernels 2.2.3 slash
nbconvert 5.6.1 slash
nbformat 5.0.4 slash
networkx 2.4 slash
nltk 3.4.5 slash
nose 1.3.7 slash
notebook 6.0.3 slash
numba 0.47.0 slash
numexpr 2.7.1 slash
numpy 1.19.2 slash
numpydoc 0.9.2 slash
oauthlib 3.1.0 slash
olefile 0.46 slash
opencv-python 4.2.0.32 slash
openpyxl 3.0.3 slash
opt-einsum 3.3.0 slash
optuna 2.1.0 slash
opuslib 2.0.0 slash
packaging 20.4 slash
pandas 1.1.2 slash
pandocfilters 1.4.2 slash
parso 0.5.2 slash
partd 1.1.0 slash
path 13.1.0 slash
pathlib2 2.3.5 slash
pathtools 0.1.2 slash
patsy 0.5.1 slash
pbr 5.5.0 slash
pep8 1.7.1 slash
pexpect 4.8.0 slash
pickleshare 0.7.5 slash
Pillow 7.1.2 slash
pip 20.0.2 slash
pkginfo 1.5.0.1 slash
plotly 4.9.0 slash
pluggy 0.13.1 slash
ply 3.11 slash
pooch 1.2.0 slash
prettytable 0.7.2 slash
progressbar2 3.53.1 slash
prometheus-client 0.7.1 slash
prompt-toolkit 3.0.4 slash
protobuf 3.13.0 slash
protobuf3-to-dict 0.1.5 slash
psutil 5.7.0 slash
psycopg2 2.7.5 slash
ptyprocess 0.6.0 slash
py 1.8.1 slash
pyasn1 0.4.8 slash
pyasn1-modules 0.2.8 slash
pycodestyle 2.5.0 slash
pycosat 0.6.3 slash
pycparser 2.20 slash
pycrypto 2.6.1 slash
pycurl 7.43.0.5 slash
pydocstyle 4.0.1 slash
pyflakes 2.1.1 slash
pygal 2.4.0 slash
Pygments 2.6.1 slash
pykerberos 1.2.1 slash
pylint 2.5.3 slash
pyodbc 4.0.0-unsupported slash
pyOpenSSL 19.1.0 slash
pyparsing 2.4.7 slash
pyperclip 1.8.0 slash
pyrsistent 0.16.0 slash
PySocks 1.7.1 slash
pytest 5.4.1 slash
pytest-arraydiff 0.3 slash
pytest-astropy 0.8.0 slash
pytest-astropy-header 0.1.2 slash
pytest-doctestplus 0.5.0 slash
pytest-openfiles 0.4.0 slash
pytest-remotedata 0.3.2 slash
python-dateutil 2.8.1 slash
python-editor 1.0.4 slash
python-jsonrpc-server 0.3.4 slash
python-language-server 0.31.9 slash
python-utils 2.4.0 slash
pytz 2020.1 slash
PyWavelets 1.1.1 slash
pyxdg 0.26 slash
PyYAML 5.3.1 slash
pyzmq 18.1.1 slash
QDarkStyle 2.8 slash
QtAwesome 0.7.0 slash
qtconsole 4.7.2 slash
QtPy 1.9.0 slash
requests 2.24.0 slash
requests-kerberos 0.12.0 slash
requests-oauthlib 1.3.0 slash
resampy 0.2.2 slash
retrying 1.3.3 slash
rope 0.16.0 slash
rsa 4.6 slash
Rtree 0.9.4 slash
ruamel-yaml 0.15.87 slash
s3fs 0.4.0 slash
s3transfer 0.3.3 slash
sagemaker 1.72.0 slash
scikit-image 0.16.2 slash
scikit-learn 0.23.2 slash
scipy 1.4.1 slash
seaborn 0.10.0 slash
SecretStorage 3.1.2 slash
semver 2.10.2 slash
Send2Trash 1.5.0 slash
setuptools 50.3.0 slash
simplegeneric 0.8.1 slash
singledispatch 3.4.0.3 slash
six 1.15.0 slash
smdebug-rulesconfig 0.1.4 slash
snowballstemmer 2.0.0 slash
sortedcollections 1.1.2 slash
sortedcontainers 2.1.0 slash
SoundFile 0.10.3.post1 slash
soupsieve 2.0.1 slash
sox 1.4.0 slash
sparkmagic 0.15.0 slash
Sphinx 3.0.4 slash
sphinxcontrib-applehelp 1.0.2 slash
sphinxcontrib-devhelp 1.0.2 slash
help 1.0.3 slash
sphinxcontrib-jsmath 1.0.1 slash
sphinxcontrib-qthelp 1.0.3 slash
1.1.4 slash
sphinxcontrib-websupport 1.2.1 slash
spyder 4.1.2 slash
spyder-kernels 1.9.0 slash
SQLAlchemy 1.3.19 slash
statsmodels 0.11.0 slash
stevedore 3.2.1 slash
sympy 1.5.1 slash
tables 3.6.1 slash
tblib 1.6.0 slash
tensorboard 1.14.0 slash
tensorboard-plugin-wit 1.7.0 slash
tensorflow-estimator 1.14.0 slash
tensorflow-gpu 1.14.0 slash
tensorflow-serving-api 2.1.0 slash
termcolor 1.1.0 slash
terminado 0.8.3 slash
testpath 0.4.4 slash
threadpoolctl 2.1.0 slash
toml 0.10.1 slash
toolz 0.10.0 slash
tornado 6.0.4 slash
tqdm 4.48.2 slash
traitlets 4.3.3 slash
typed-ast 1.4.1 slash
typing-extensions 3.7.4.1 slash
ujson 1.35 slash
unicodecsv 0.14.1 slash
urllib3 1.25.10 slash
watchdog 0.10.2 slash
wcwidth 0.2.5 slash
webencodings 0.5.1 slash
Werkzeug 1.0.1 slash
wheel 0.35.1 slash
widgetsnbextension 3.5.1 slash
wrapt 1.12.1 slash
wurlitzer 2.0.0 slash
xlrd 1.2.0 slash
XlsxWriter 1.2.8 slash
xlwt 1.3.0 slash
yapf 0.28.0 slash
zict 2.0.0 slash
zipp 3.1.0
I tried also with tensorflow-gpu==1.15.2, but got the same error.
Could someone give me some hint ? slash
Thanks in advance
[This is an archived TTS discussion thread from discourse.mozilla.org/t/finetuning-the-model-on-gpu-machine]
Beta Was this translation helpful? Give feedback.
All reactions