Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError: can't pickle _thread.lock objects #11157

Closed
matteo-brucato opened this issue Jun 30, 2017 · 64 comments
Closed

TypeError: can't pickle _thread.lock objects #11157

matteo-brucato opened this issue Jun 30, 2017 · 64 comments
Assignees
Labels
stat:awaiting tensorflower Status - Awaiting response from tensorflower

Comments

@matteo-brucato
Copy link

matteo-brucato commented Jun 30, 2017

Please go to Stack Overflow for help and support:

http://stackoverflow.com/questions/tagged/tensorflow

If you open a GitHub issue, here is our policy:

  1. It must be a bug or a feature request.
  2. The form below must be filled out.
  3. It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.


System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No, using stock examples
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
  • TensorFlow installed from (source or binary): pip
  • TensorFlow version (use command below): 1.2.1
  • Python version: 3.6.1 (Anaconda 4.4.0 64-bit)
  • Bazel version (if compiling from source):
  • CUDA/cuDNN version:
  • GPU model and memory:
  • Exact command to reproduce: I'm running the seq2seq example in models/tutorials/rnn/translate, verbatim.

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

Collecting system information...
2017-06-29 18:35:16.672194: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-29 18:35:16.672242: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-29 18:35:16.672250: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
Wrote environment to tf_env.txt. You can review the contents of that file.
and use it to populate the fields in the github issue template.

cat tf_env.txt

== cat /etc/issue ===============================================
Linux GCRGDL171 4.8.0-58-generic #63~16.04.1-Ubuntu SMP Mon Jun 26 18:08:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
VERSION="16.04.2 LTS (Xenial Xerus)"
VERSION_ID="16.04"
VERSION_CODENAME=xenial

== are we in docker =============================================
No

== compiler =====================================================
c++ (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

== uname -a =====================================================
Linux GCRGDL171 4.8.0-58-generic #63~16.04.1-Ubuntu SMP Mon Jun 26 18:08:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

== check pips ===================================================
numpy (1.12.1)
numpydoc (0.6.0)
protobuf (3.3.0)
tensorflow (1.2.1)

== check for virtualenv =========================================
False

== tensorflow import ============================================
tf.VERSION = 1.2.1
tf.GIT_VERSION = v1.2.0-5-g435cdfc
tf.COMPILER_VERSION = v1.2.0-5-g435cdfc
Sanity check: array([1], dtype=int32)

== env ==========================================================
LD_LIBRARY_PATH /usr/local/cuda:/usr/local/cuda/lib64:
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
Thu Jun 29 18:35:19 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66 Driver Version: 375.66 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K40m Off | 0000:27:00.0 Off | 0 |
| N/A 27C P8 21W / 235W | 0MiB / 11439MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

== cuda libs ===================================================
/usr/local/cuda-8.0/lib64/libcudart.so.8.0.61
/usr/local/cuda-8.0/lib64/libcudart_static.a
/usr/local/cuda-8.0/doc/man/man7/libcudart.7
/usr/local/cuda-8.0/doc/man/man7/libcudart.so.7

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.

I get exception: TypeError: can't pickle _thread.lock objects. It happens on different machines with the same python version. Just running your example code verbatim.

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

Traceback (most recent call last):
File "translate.py", line 322, in
tf.app.run()
File "/home/t-mabruc/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "translate.py", line 319, in main
train()
File "translate.py", line 178, in train
model = create_model(sess, False)
File "translate.py", line 136, in create_model
dtype=dtype)
File "/home/t-mabruc/models/tutorials/rnn/translate/seq2seq_model.py", line 179, in init
softmax_loss_function=softmax_loss_function)
File "/home/t-mabruc/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py", line 1206, in model_with_buckets
decoder_inputs[:bucket[1]])
File "/home/t-mabruc/models/tutorials/rnn/translate/seq2seq_model.py", line 178, in
lambda x, y: seq2seq_f(x, y, False),
File "/home/t-mabruc/models/tutorials/rnn/translate/seq2seq_model.py", line 142, in seq2seq_f
dtype=dtype)
File "/home/t-mabruc/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py", line 848, in embedding_attention_seq2seq
encoder_cell = copy.deepcopy(cell)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 161, in deepcopy
y = copier(memo)
File "/home/t-mabruc/anaconda3/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 476, in deepcopy
setattr(result, k, copy.deepcopy(v, memo))
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 215, in _deepcopy_list
append(deepcopy(a, memo))
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/t-mabruc/anaconda3/lib/python3.6/copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: can't pickle _thread.lock objects

@printdhruv
Copy link
Contributor

Hi,
As website sates "NOTE: The conda package is community supported, not officially supported. That is, the TensorFlow team neither tests nor maintains this conda package. Use that package at your own risk."
I think that you should try it with native Python environment. I tested and seems running successfully.

@matteo-brucato
Copy link
Author

I also tried it with native Python, and the error persists. The error disappears only when downgrading to tensorflow version 1.0.0

@printdhruv
Copy link
Contributor

Can you list the exact steps,source code which you are executing?

@matteo-brucato
Copy link
Author

matteo-brucato commented Jun 30, 2017

I'm running the code found here https://github.com/tensorflow/models/tree/master/tutorials/rnn/translate

I run it with:
python translate.py --data_dir <path-to-data-dir> --train_dir <path-to-train-dir> --from_train_data <path-to-from-train-data> --to_train_data <path-to-to-train-data>

I'm running this code as-is. Python is native (no Anacoda, no virtualenv, etc.)

Also, the same error happens on both Linux and Windows. And it is fixed on both systems only by using tensorflow 1.0.0.

@skye
Copy link
Member

skye commented Jul 5, 2017

@nealwu

@skye skye added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Jul 5, 2017
@nealwu
Copy link
Contributor

nealwu commented Jul 6, 2017

That's very odd that the error disappears with TensorFlow 1.0. @lukaszkaiser do you know what might be happening here?

@lukaszkaiser
Copy link
Contributor

There were a lot of changes to RNNCell since 1.0, must ask @ebrevdo to take a look.

@ebrevdo
Copy link
Contributor

ebrevdo commented Jul 6, 2017

Do you get this error in earlier versions of python (i.e., python 3.4)? I'm trying to replicate locally with python2.7.

@ebrevdo
Copy link
Contributor

ebrevdo commented Jul 6, 2017

Scratch that; i'll test it with python3.4.

@SeekPoint
Copy link

I totally reproduced this error on py3.6.0 and py2.7.10 on Mac, but the error are different:

for python3.6.0:
seq2seq.py", line 910, in embedding_attention_seq2seq
encoder_cell = copy.deepcopy(cell)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 161, in deepcopy
y = copier(memo)
File "/usr/local/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 476, in deepcopy
setattr(result, k, copy.deepcopy(v, memo))
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 215, in _deepcopy_list
append(deepcopy(a, memo))
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/local/Cellar/python3/3.6.0_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: can't pickle _thread.lock objects

for python2.7.10:
encoder_cell = copy.deepcopy(cell)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 174, in deepcopy
y = copier(memo)
File "/Library/Python/2.7/site-packages/tensorflow/python/layers/base.py", line 476, in deepcopy
setattr(result, k, copy.deepcopy(v, memo))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 230, in _deepcopy_list
y.append(deepcopy(a, memo))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 230, in _deepcopy_list
y.append(deepcopy(a, memo))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 230, in _deepcopy_list
y.append(deepcopy(a, memo))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
y.append(deepcopy(a, memo))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 329, in _reconstruct
y = callable(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 93, in newobj
return cls.new(cls, *args)
TypeError: object.new(NotImplementedType) is not safe, use NotImplementedType.new()

@printdhruv
Copy link
Contributor

@mattfeel Can you tell the log when you execute python translate.py? It will download WMT data from internet thus make sure that you have at least 20GB of disk space.

@fabriciorsf
Copy link

fabriciorsf commented Jul 29, 2017

I get this same error with Keras+TensorFlow on fit_generator.
And the same code with Keras+Theano works fine.

Follow the command gets the error:

model.fit_generator(self.train_inputs, steps_per_epoch=self.train_inputs.steps_per_epoch(),
                    validation_data=test_input_sequence, validation_steps=steps_test,
                    max_queue_size=self.train_inputs.workers, epochs=i+1, initial_epoch=i,
                    workers=self.train_inputs.workers, use_multiprocessing=True,
                    callbacks = callbacks)

The error:

Epoch 1/1
Traceback (most recent call last):
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/site-packages/keras/utils/data_utils.py", line 497, in get
    inputs = self.queue.get(block=True).get()
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/multiprocessing/pool.py", line 608, in get
    raise self._value
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/multiprocessing/pool.py", line 385, in _handle_tasks
    put(task)
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./myfolder/mycode.py", line 473, in <module>
    main()
  File "./myfolder/mycode.py", line 459, in main
    autonem.train_autonem(args.embedding_file, args.tune_embedding)
  File "./myfolder/mycode.py", line 182, in train_autonem
    callbacks = callbacks)
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/site-packages/keras/engine/training.py", line 1809, in fit_generator
    generator_output = next(output_generator)
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/site-packages/keras/utils/data_utils.py", line 502, in get
    raise StopIteration(e)
StopIteration: can't pickle _thread.lock objects

System information:

Have I written custom code: Yes
OS Platform and Distribution: Linux GnomeUbuntu 16.04, but with new kernel
_TensorFlow installed from: pip
TensorFlow version: 1.2.1
Python version: 3.6.1 (Miniconda3 4.3.11-64bit)
Bazel version (if compiling from source): I don't know.
CUDA/cuDNN version: I don't use because my graphic card is AMD-Radeon
GPU model and memory: AMD Radeon R7 M260/M265
CPU model: Intel® Core™ i7-4510U CPU @ 2.00GHz × 4
RAM Memory: 16GiB (2x8Gib dual-channel)
Exact command to reproduce:

history = CumulativeHistory()
callbacks = [history]
from keras import backend as K
if K.backend() == 'tensorflow':
  board = keras.callbacks.TensorBoard(log_dir=f"{self.prefix_folder_logs}{time()}",
                                    histogram_freq=1, write_graph=True, write_images=True)
  callbacks.append(board)
metric_to_compare = 'val_euclidean_distance'
print("Begin of training model...")
for i in range(MAX_NUM_EPOCHS):
  model.fit_generator(self.train_inputs, steps_per_epoch=self.train_inputs.steps_per_epoch(),
                      validation_data=test_input_sequence, validation_steps=steps_test,
                      max_queue_size=self.train_inputs.workers, epochs=i+1, initial_epoch=i,
                      workers=self.train_inputs.workers, use_multiprocessing=True,
                      callbacks = callbacks)
  try:
    metrics_diff = history.history[metric_to_compare][i] - min(history.history[metric_to_compare][:i])
  except:
    metrics_diff = -1
  if metrics_diff < 0:
    self._save_models(i)
    self.data_processor = None  # Empty memory
    best_epoch = i
    num_worse_epochs = 0
  elif metrics_diff > 0:
    num_worse_epochs += 1
    if num_worse_epochs >= PATIENCE:
      print("Ran out of patience. Stopping training.")
      break
print("End of training model.")

Collected information:

(myenv) myuser@mymachine:~$ ./tf_env_collect.sh 
Collecting system information...
2017-07-28 21:05:00.140602: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-28 21:05:00.140632: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-28 21:05:00.140645: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-07-28 21:05:00.140650: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-28 21:05:00.140656: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
Wrote environment to tf_env.txt. You can review the contents of that file.
and use it to populate the fields in the github issue template.

cat tf_env.txt

(myenv) myuser@mymachine:~$ cat tf_env.txt

== cat /etc/issue ===============================================
Linux mymachine 4.4.0-87-generic #110~14.04.1-Ubuntu SMP Tue Jul 18 14:51:32 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
VERSION="14.04.5 LTS, Trusty Tahr"
VERSION_ID="14.04"

== are we in docker =============================================
No

== compiler =====================================================
c++ (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


== uname -a =====================================================
Linux mymachine 4.4.0-87-generic #110~14.04.1-Ubuntu SMP Tue Jul 18 14:51:32 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

== check pips ===================================================
numpy (1.13.1)
protobuf (3.3.0)
tensorflow (1.2.1)

== check for virtualenv =========================================
False

== tensorflow import ============================================
tf.VERSION = 1.2.1
tf.GIT_VERSION = v1.2.0-5-g435cdfc
tf.COMPILER_VERSION = v1.2.0-5-g435cdfc
Sanity check: array([1], dtype=int32)

== env ==========================================================
LD_LIBRARY_PATH /opt/programs/miniconda3/envs/myenv/lib:/opt/intel/compilers_and_libraries_2017.4.196/linux/tbb/lib/intel64_lin/gcc4.7:/opt/intel/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64_lin::/opt/programs/acml/gfortran64/lib
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
./tf_env_collect.sh: linha 105: nvidia-smi: comando não encontrado

== cuda libs  ===================================================

@ebrevdo
Copy link
Contributor

ebrevdo commented Jul 29, 2017 via email

@SeekPoint
Copy link

@ebigelow
it doesn't work!

@SeekPoint
Copy link

It is a real disaster!

any workaround?

@pwfff
Copy link

pwfff commented Aug 1, 2017

@lovejasmine I was able to work around it by replacing the .deepcopy call with .copy. This probably breaks other things in subtle ways, but for my purpose it allowed me to get on with training.

The line I changed was

packages/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py", line 848, in embedding_attention_seq2seq
encoder_cell = copy.deepcopy(cell)

@ebrevdo this was after trying the RC and compiling master myself to no avail.

@ebrevdo
Copy link
Contributor

ebrevdo commented Aug 1, 2017 via email

@SeekPoint
Copy link

@pwfff
in my project,change from deepcopy to copy will move on, but it doesn't make any sense on the logic

@Chesao
Copy link

Chesao commented Aug 4, 2017

@mattfeel ,

I suffered exactly the same problem.
I just passed the error by modifying 2 lines of the following seq2seq.py file from Tensorflow.

file: Anaconda3\Lib\site-packages\tensorflow\contrib\legacy_seq2seq\python\ops\seq2seq.py
848 #encoder_cell = copy.deepcopy(cell)
849 encoder_cell = core_rnn_cell.EmbeddingWrapper(
850 cell, #encoder_cell,

Now, my seq2seq model is being trained.
Good luck!

@SeekPoint
Copy link

@Chesao

what is core_rnn_cell

@Chesao
Copy link

Chesao commented Aug 4, 2017

@lovejasmine

it is tf.nn.rnn_cell.GRUCell()

@dshahrokhian
Copy link

Same issue here, when using a Keras + Tensorflow and creating an sklearn wrapper. Still couldn't find a workaround.

@Chesao
Copy link

Chesao commented Aug 16, 2017

@dshahrokhian

Have you tried my workaround? Why don't you dump up your full error log here?

@SeekPoint
Copy link

@dshahrokhian
your workaround works on my case, thanks

@dshahrokhian
Copy link

@Chesao AFAIK I cannot use your method in my case. Here's my error log, but I already decided to put some extra code so I don't need to use the sklearn wrapper that was giving me the error.

Traceback (most recent call last):
  File "experiments/openface_ck+.py", line 77, in <module>
    main()
  File "experiments/openface_ck+.py", line 74, in main
    io_utils.kfold_report_metrics(get_temporal_model, optimal['max_params'], features, labels)
  File "/home/dani/Git/EmotionRecognition/experiments/io_utils.py", line 140, in kfold_report_metrics
    print("Test loss and Confidence Interval: %.2f (+/- %.2f)" % (np.mean(losses), np.std(losses)))
  File "/home/dani/Git/EmotionRecognition/experiments/io_utils.py", line 225, in plot_learning_curve
    
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_validation.py", line 772, in learning_curve
    for train, test in cv_iter
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py", line 758, in __call__
    while self.dispatch_one_batch(iterator):
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py", line 603, in dispatch_one_batch
    tasks = BatchedCalls(itertools.islice(iterator, batch_size))
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py", line 127, in __init__
    self.items = list(iterator_slice)
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/sklearn/model_selection/_validation.py", line 773, in <genexpr>
    for n_train_samples in train_sizes_abs)
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/sklearn/base.py", line 69, in clone
    new_object_params[name] = clone(param, safe=False)
  File "/home/dani/Software/anaconda3/lib/python3.6/site-packages/sklearn/base.py", line 60, in clone
    return copy.deepcopy(estimator)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/dani/Software/anaconda3/lib/python3.6/copy.py", line 169, in deepcopy
    rv = reductor(4)
TypeError: can't pickle _thread.lock objects

Thanks,
Dani

@ebrevdo
Copy link
Contributor

ebrevdo commented Aug 16, 2017

Can someone provide a minimal reproducible code snippet? Ideally ~ 10 lines with tf.constant() inputs?

@JankiMehta
Copy link

JankiMehta commented Aug 28, 2017

@ebrevdo

import pickle
from keras.layers import Dense
from keras.layers import LSTM
from keras.models import Sequential
from keras.metrics import categorical_accuracy

model = Sequential()
model.add(LSTM(20, return_sequences=True, stateful=False, batch_input_shape=(10, 20, 4)))
model.add(Dense(3, activation='softmax'))

model.compile(loss="categorical_crossentropy",
              optimizer='adam',
              metrics=[categorical_accuracy],
              sample_weight_mode='temporal')

data_path = '/home/ubuntu/invoice/data/'     #any path to store pickle dump
output_file_path = data_path + 'model.dat'
with open(output_file_path, 'wb') as f:
    pickle.dump(model, f)

Error Msg:

Traceback (most recent call last):
  File "<input>", line 19, in <module>
TypeError: can't pickle _thread.lock objects

@zzks
Copy link

zzks commented Sep 5, 2017

same problem... keras1.2.0, tf0.12.1

@josh-marsh
Copy link

Does anyone know of any other solutions? I am having the same problem, and none of the above solutions seem to work.

@grayfall
Copy link

grayfall commented May 1, 2018

@ebrevdo unfortunately, keras.save doesn't not seem to be a universal solution, because that's what has brought me here in the first place. I've checked all major tensorflow releases from 1.4 to 1.7 – the issue is persistent.

@thormacy
Copy link

It seems to be an issue caused when using embedding_attention_seq2seq several times while training with buckets.

@thormacy
Copy link

One alternative might be only fit one bucket at a time

@thormacy
Copy link

@maxim5
your solution does not work for me.
@Chesao 's solution works.
But I am not sure if there's any other influence.

@suffic
Copy link

suffic commented Jun 22, 2018

Seeing same issue with tf 1.5.0:
pk.dumps(obj)
TypeError: can't pickle _thread.lock objects

@zdx3578
Copy link

zdx3578 commented Jul 3, 2018

Saved final_models/thor_model.pkl
Traceback (most recent call last):
File "train_expert/train_thor.py", line 46, in
main()
File "train_expert/train_thor.py", line 42, in main
act.save("final_models/thor_model.pkl")
File "/home/sdc/github/baselines-rudder/baselines/deepq/simple.py", line 62, in save
cloudpickle.dump((model_data, self._act_params), f)
File "/home/sdc/github/baselines-rudder/baseline/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 879, in dump
CloudPickler(file, protocol=protocol).dump(obj)
File "/home/sdc/github/baselines-rudder/baseline/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 268, in dump
return Pickler.dump(self, obj)
File "/usr/lib/python3.5/pickle.py", line 408, in dump
self.save(obj)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python3.5/pickle.py", line 725, in save_tuple
save(element)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
save(v)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/sdc/github/baselines-rudder/baseline/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 413, in save_function
self.save_function_tuple(obj)
File "/home/sdc/github/baselines-rudder/baseline/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 559, in save_function_tuple
save(state)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
save(v)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python3.5/pickle.py", line 770, in save_list
self._batch_appends(obj)
File "/usr/lib/python3.5/pickle.py", line 797, in _batch_appends
save(tmp[0])
File "/usr/lib/python3.5/pickle.py", line 520, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.5/pickle.py", line 623, in save_reduce
save(state)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
save(v)
File "/usr/lib/python3.5/pickle.py", line 520, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.5/pickle.py", line 623, in save_reduce
save(state)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
save(v)
File "/usr/lib/python3.5/pickle.py", line 520, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.5/pickle.py", line 623, in save_reduce
save(state)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
save(v)
File "/usr/lib/python3.5/pickle.py", line 520, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.5/pickle.py", line 623, in save_reduce
save(state)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python3.5/pickle.py", line 810, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.5/pickle.py", line 836, in _batch_setitems
save(v)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/home/sdc/github/baselines-rudder/baseline/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 630, in save_builtin_function
return self.save_function(obj)
File "/home/sdc/github/baselines-rudder/baseline/lib/python3.5/site-packages/cloudpickle/cloudpickle.py", line 400, in save_function
return self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.5/pickle.py", line 599, in save_reduce
save(args)
File "/usr/lib/python3.5/pickle.py", line 475, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python3.5/pickle.py", line 725, in save_tuple
save(element)
File "/usr/lib/python3.5/pickle.py", line 495, in save
rv = reduce(self.proto)
TypeError: can't pickle _thread.lock objects

tf 1.8

@alikureishy
Copy link

alikureishy commented Jul 25, 2018

I'm getting a similar error with keras 2.0.9 and tensorflow 1.3.0. Note, however, that this happens only when I wrap a model using the keras.utils.training_utils.multi_gpu_model class, as below:

model = multi_gpu_model(model, gpus=8) # I was trying to utilize 8 GPUs
..
..
checkpointer = ModelCheckpoint(filepath='results/model.hd5', verbose=0)
..
hist = model.fit_generator(generator=audio_gen.next_train(), steps_per_epoch=steps_per_epoch,
        epochs=epochs, validation_data=audio_gen.next_valid(), validation_steps=validation_steps,
        callbacks=[checkpointer], verbose=verbose)  <<<<<<<<<<< # ON THIS LINE #

In the absence of multi_gpu_model, this code above runs without a problem.

===================== OUTPUT ======================
Epoch 1/20
24/106 [================>.] - ETA: 35 - loss: 194.7247

TypeError                                 Traceback (most recent call last)
     74     hist = model.fit_generator(generator=audio_gen.next_train(), steps_per_epoch=steps_per_epoch,
     75         epochs=epochs, validation_data=audio_gen.next_valid(), validation_steps=validation_steps,
---> 76         callbacks=[checkpointer], verbose=verbose)
     77 
     78     # save model loss

/usr/local/lib/python3.5/dist-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs)
     85                 warnings.warn('Update your `' + object_name +
     86                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 87             return func(*args, **kwargs)
     88         wrapper._original_function = func
     89         return wrapper

/usr/local/lib/python3.5/dist-packages/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
   2115                         break
   2116 
-> 2117                 callbacks.on_epoch_end(epoch, epoch_logs)
   2118                 epoch += 1
   2119                 if callback_model.stop_training:

/usr/local/lib/python3.5/dist-packages/keras/callbacks.py in on_epoch_end(self, epoch, logs)
     71         logs = logs or {}
     72         for callback in self.callbacks:
---> 73             callback.on_epoch_end(epoch, logs)
     74 
     75     def on_batch_begin(self, batch, logs=None):

/usr/local/lib/python3.5/dist-packages/keras/callbacks.py in on_epoch_end(self, epoch, logs)
    423                     self.model.save_weights(filepath, overwrite=True)
    424                 else:
--> 425                     self.model.save(filepath, overwrite=True)
    426 
    427 

/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py in save(self, filepath, overwrite, include_optimizer)
   2554         """
   2555         from ..models import save_model
-> 2556         save_model(self, filepath, overwrite, include_optimizer)
   2557 
   2558     def save_weights(self, filepath, overwrite=True):

/usr/local/lib/python3.5/dist-packages/keras/models.py in save_model(model, filepath, overwrite, include_optimizer)
    105         f.attrs['model_config'] = json.dumps({
    106             'class_name': model.__class__.__name__,
--> 107             'config': model.get_config()
    108         }, default=get_json_type).encode('utf8')
    109 

/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py in get_config(self)
   2395             model_outputs.append([layer.name, new_node_index, tensor_index])
   2396         config['output_layers'] = model_outputs
-> 2397         return copy.deepcopy(config)
   2398 
   2399     @classmethod

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_dict(x, memo)
    241     memo[id(x)] = y
    242     for key, value in x.items():
--> 243         y[deepcopy(key, memo)] = deepcopy(value, memo)
    244     return y
    245 d[dict] = _deepcopy_dict

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_list(x, memo)
    216     memo[id(x)] = y
    217     for a in x:
--> 218         y.append(deepcopy(a, memo))
    219     return y
    220 d[list] = _deepcopy_list

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_dict(x, memo)
    241     memo[id(x)] = y
    242     for key, value in x.items():
--> 243         y[deepcopy(key, memo)] = deepcopy(value, memo)
    244     return y
    245 d[dict] = _deepcopy_dict

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_dict(x, memo)
    241     memo[id(x)] = y
    242     for key, value in x.items():
--> 243         y[deepcopy(key, memo)] = deepcopy(value, memo)
    244     return y
    245 d[dict] = _deepcopy_dict

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_tuple(x, memo)
    221 
    222 def _deepcopy_tuple(x, memo):
--> 223     y = [deepcopy(a, memo) for a in x]
    224     # We're not going to put the tuple in the memo, but it's still important we
    225     # check for it, in case the tuple contains recursive mutable structures.

/usr/lib/python3.5/copy.py in <listcomp>(.0)
    221 
    222 def _deepcopy_tuple(x, memo):
--> 223     y = [deepcopy(a, memo) for a in x]
    224     # We're not going to put the tuple in the memo, but it's still important we
    225     # check for it, in case the tuple contains recursive mutable structures.

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_tuple(x, memo)
    221 
    222 def _deepcopy_tuple(x, memo):
--> 223     y = [deepcopy(a, memo) for a in x]
    224     # We're not going to put the tuple in the memo, but it's still important we
    225     # check for it, in case the tuple contains recursive mutable structures.

/usr/lib/python3.5/copy.py in <listcomp>(.0)
    221 
    222 def _deepcopy_tuple(x, memo):
--> 223     y = [deepcopy(a, memo) for a in x]
    224     # We're not going to put the tuple in the memo, but it's still important we
    225     # check for it, in case the tuple contains recursive mutable structures.

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    180                             raise Error(
    181                                 "un(deep)copyable object of type %s" % cls)
--> 182                 y = _reconstruct(x, rv, 1, memo)
    183 
    184     # If is its own copy, don't memoize.

/usr/lib/python3.5/copy.py in _reconstruct(x, info, deep, memo)
    295     if state is not None:
    296         if deep:
--> 297             state = deepcopy(state, memo)
    298         if hasattr(y, '__setstate__'):
    299             y.__setstate__(state)

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_dict(x, memo)
    241     memo[id(x)] = y
    242     for key, value in x.items():
--> 243         y[deepcopy(key, memo)] = deepcopy(value, memo)
    244     return y
    245 d[dict] = _deepcopy_dict

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    180                             raise Error(
    181                                 "un(deep)copyable object of type %s" % cls)
--> 182                 y = _reconstruct(x, rv, 1, memo)
    183 
    184     # If is its own copy, don't memoize.

/usr/lib/python3.5/copy.py in _reconstruct(x, info, deep, memo)
    295     if state is not None:
    296         if deep:
--> 297             state = deepcopy(state, memo)
    298         if hasattr(y, '__setstate__'):
    299             y.__setstate__(state)

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_dict(x, memo)
    241     memo[id(x)] = y
    242     for key, value in x.items():
--> 243         y[deepcopy(key, memo)] = deepcopy(value, memo)
    244     return y
    245 d[dict] = _deepcopy_dict

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_dict(x, memo)
    241     memo[id(x)] = y
    242     for key, value in x.items():
--> 243         y[deepcopy(key, memo)] = deepcopy(value, memo)
    244     return y
    245 d[dict] = _deepcopy_dict

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_method(x, memo)
    248 
    249 def _deepcopy_method(x, memo): # Copy instance methods
--> 250     return type(x)(x.__func__, deepcopy(x.__self__, memo))
    251 _deepcopy_dispatch[types.MethodType] = _deepcopy_method
    252 

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    180                             raise Error(
    181                                 "un(deep)copyable object of type %s" % cls)
--> 182                 y = _reconstruct(x, rv, 1, memo)
    183 
    184     # If is its own copy, don't memoize.

/usr/lib/python3.5/copy.py in _reconstruct(x, info, deep, memo)
    295     if state is not None:
    296         if deep:
--> 297             state = deepcopy(state, memo)
    298         if hasattr(y, '__setstate__'):
    299             y.__setstate__(state)

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_dict(x, memo)
    241     memo[id(x)] = y
    242     for key, value in x.items():
--> 243         y[deepcopy(key, memo)] = deepcopy(value, memo)
    244     return y
    245 d[dict] = _deepcopy_dict

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_dict(x, memo)
    241     memo[id(x)] = y
    242     for key, value in x.items():
--> 243         y[deepcopy(key, memo)] = deepcopy(value, memo)
    244     return y
    245 d[dict] = _deepcopy_dict

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    180                             raise Error(
    181                                 "un(deep)copyable object of type %s" % cls)
--> 182                 y = _reconstruct(x, rv, 1, memo)
    183 
    184     # If is its own copy, don't memoize.

/usr/lib/python3.5/copy.py in _reconstruct(x, info, deep, memo)
    295     if state is not None:
    296         if deep:
--> 297             state = deepcopy(state, memo)
    298         if hasattr(y, '__setstate__'):
    299             y.__setstate__(state)

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_dict(x, memo)
    241     memo[id(x)] = y
    242     for key, value in x.items():
--> 243         y[deepcopy(key, memo)] = deepcopy(value, memo)
    244     return y
    245 d[dict] = _deepcopy_dict

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_dict(x, memo)
    241     memo[id(x)] = y
    242     for key, value in x.items():
--> 243         y[deepcopy(key, memo)] = deepcopy(value, memo)
    244     return y
    245 d[dict] = _deepcopy_dict

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    180                             raise Error(
    181                                 "un(deep)copyable object of type %s" % cls)
--> 182                 y = _reconstruct(x, rv, 1, memo)
    183 
    184     # If is its own copy, don't memoize.

/usr/lib/python3.5/copy.py in _reconstruct(x, info, deep, memo)
    295     if state is not None:
    296         if deep:
--> 297             state = deepcopy(state, memo)
    298         if hasattr(y, '__setstate__'):
    299             y.__setstate__(state)

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_dict(x, memo)
    241     memo[id(x)] = y
    242     for key, value in x.items():
--> 243         y[deepcopy(key, memo)] = deepcopy(value, memo)
    244     return y
    245 d[dict] = _deepcopy_dict

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    180                             raise Error(
    181                                 "un(deep)copyable object of type %s" % cls)
--> 182                 y = _reconstruct(x, rv, 1, memo)
    183 
    184     # If is its own copy, don't memoize.

/usr/lib/python3.5/copy.py in _reconstruct(x, info, deep, memo)
    295     if state is not None:
    296         if deep:
--> 297             state = deepcopy(state, memo)
    298         if hasattr(y, '__setstate__'):
    299             y.__setstate__(state)

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_dict(x, memo)
    241     memo[id(x)] = y
    242     for key, value in x.items():
--> 243         y[deepcopy(key, memo)] = deepcopy(value, memo)
    244     return y
    245 d[dict] = _deepcopy_dict

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    180                             raise Error(
    181                                 "un(deep)copyable object of type %s" % cls)
--> 182                 y = _reconstruct(x, rv, 1, memo)
    183 
    184     # If is its own copy, don't memoize.

/usr/lib/python3.5/copy.py in _reconstruct(x, info, deep, memo)
    295     if state is not None:
    296         if deep:
--> 297             state = deepcopy(state, memo)
    298         if hasattr(y, '__setstate__'):
    299             y.__setstate__(state)

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    153     copier = _deepcopy_dispatch.get(cls)
    154     if copier:
--> 155         y = copier(x, memo)
    156     else:
    157         try:

/usr/lib/python3.5/copy.py in _deepcopy_dict(x, memo)
    241     memo[id(x)] = y
    242     for key, value in x.items():
--> 243         y[deepcopy(key, memo)] = deepcopy(value, memo)
    244     return y
    245 d[dict] = _deepcopy_dict

/usr/lib/python3.5/copy.py in deepcopy(x, memo, _nil)
    172                     reductor = getattr(x, "__reduce_ex__", None)
    173                     if reductor:
--> 174                         rv = reductor(4)
    175                     else:
    176                         reductor = getattr(x, "__reduce__", None)

TypeError: can't pickle _thread.lock objects

@ebrevdo
Copy link
Contributor

ebrevdo commented Jul 25, 2018 via email

@PallawiSinghal
Copy link

848

did not work for me.maybe I did not type it correct,can u please write it properly

@Zhizh1
Copy link

Zhizh1 commented Feb 21, 2019

I get this same error with Keras+TensorFlow on fit_generator.
And the same code with Keras+Theano works fine.

Follow the command gets the error:

model.fit_generator(self.train_inputs, steps_per_epoch=self.train_inputs.steps_per_epoch(),
                    validation_data=test_input_sequence, validation_steps=steps_test,
                    max_queue_size=self.train_inputs.workers, epochs=i+1, initial_epoch=i,
                    workers=self.train_inputs.workers, use_multiprocessing=True,
                    callbacks = callbacks)

The error:

Epoch 1/1
Traceback (most recent call last):
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/site-packages/keras/utils/data_utils.py", line 497, in get
    inputs = self.queue.get(block=True).get()
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/multiprocessing/pool.py", line 608, in get
    raise self._value
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/multiprocessing/pool.py", line 385, in _handle_tasks
    put(task)
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./myfolder/mycode.py", line 473, in <module>
    main()
  File "./myfolder/mycode.py", line 459, in main
    autonem.train_autonem(args.embedding_file, args.tune_embedding)
  File "./myfolder/mycode.py", line 182, in train_autonem
    callbacks = callbacks)
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/site-packages/keras/engine/training.py", line 1809, in fit_generator
    generator_output = next(output_generator)
  File "/opt/programs/miniconda3/envs/myenv/lib/python3.6/site-packages/keras/utils/data_utils.py", line 502, in get
    raise StopIteration(e)
StopIteration: can't pickle _thread.lock objects

System information:

Have I written custom code: Yes
OS Platform and Distribution: Linux GnomeUbuntu 16.04, but with new kernel
_TensorFlow installed from: pip
TensorFlow version: 1.2.1
Python version: 3.6.1 (Miniconda3 4.3.11-64bit)
Bazel version (if compiling from source): I don't know.
CUDA/cuDNN version: I don't use because my graphic card is AMD-Radeon
GPU model and memory: AMD Radeon R7 M260/M265
CPU model: Intel® Core™ i7-4510U CPU @ 2.00GHz × 4
RAM Memory: 16GiB (2x8Gib dual-channel)
Exact command to reproduce:

history = CumulativeHistory()
callbacks = [history]
from keras import backend as K
if K.backend() == 'tensorflow':
  board = keras.callbacks.TensorBoard(log_dir=f"{self.prefix_folder_logs}{time()}",
                                    histogram_freq=1, write_graph=True, write_images=True)
  callbacks.append(board)
metric_to_compare = 'val_euclidean_distance'
print("Begin of training model...")
for i in range(MAX_NUM_EPOCHS):
  model.fit_generator(self.train_inputs, steps_per_epoch=self.train_inputs.steps_per_epoch(),
                      validation_data=test_input_sequence, validation_steps=steps_test,
                      max_queue_size=self.train_inputs.workers, epochs=i+1, initial_epoch=i,
                      workers=self.train_inputs.workers, use_multiprocessing=True,
                      callbacks = callbacks)
  try:
    metrics_diff = history.history[metric_to_compare][i] - min(history.history[metric_to_compare][:i])
  except:
    metrics_diff = -1
  if metrics_diff < 0:
    self._save_models(i)
    self.data_processor = None  # Empty memory
    best_epoch = i
    num_worse_epochs = 0
  elif metrics_diff > 0:
    num_worse_epochs += 1
    if num_worse_epochs >= PATIENCE:
      print("Ran out of patience. Stopping training.")
      break
print("End of training model.")

Collected information:

(myenv) myuser@mymachine:~$ ./tf_env_collect.sh 
Collecting system information...
2017-07-28 21:05:00.140602: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-28 21:05:00.140632: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-28 21:05:00.140645: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-07-28 21:05:00.140650: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-28 21:05:00.140656: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
Wrote environment to tf_env.txt. You can review the contents of that file.
and use it to populate the fields in the github issue template.

cat tf_env.txt

(myenv) myuser@mymachine:~$ cat tf_env.txt

== cat /etc/issue ===============================================
Linux mymachine 4.4.0-87-generic #110~14.04.1-Ubuntu SMP Tue Jul 18 14:51:32 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
VERSION="14.04.5 LTS, Trusty Tahr"
VERSION_ID="14.04"

== are we in docker =============================================
No

== compiler =====================================================
c++ (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


== uname -a =====================================================
Linux mymachine 4.4.0-87-generic #110~14.04.1-Ubuntu SMP Tue Jul 18 14:51:32 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

== check pips ===================================================
numpy (1.13.1)
protobuf (3.3.0)
tensorflow (1.2.1)

== check for virtualenv =========================================
False

== tensorflow import ============================================
tf.VERSION = 1.2.1
tf.GIT_VERSION = v1.2.0-5-g435cdfc
tf.COMPILER_VERSION = v1.2.0-5-g435cdfc
Sanity check: array([1], dtype=int32)

== env ==========================================================
LD_LIBRARY_PATH /opt/programs/miniconda3/envs/myenv/lib:/opt/intel/compilers_and_libraries_2017.4.196/linux/tbb/lib/intel64_lin/gcc4.7:/opt/intel/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64_lin:/opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64_lin::/opt/programs/acml/gfortran64/lib
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
./tf_env_collect.sh: linha 105: nvidia-smi: comando não encontrado

== cuda libs  ===================================================

when I use fit_generator() meet the same error with tf 1.8.0 and keras 2.2.4.

@minda163
Copy link

minda163 commented Mar 7, 2019

Hi,
As website sates "NOTE: The conda package is community supported, not officially supported. That is, the TensorFlow team neither tests nor maintains this conda package. Use that package at your own risk."
I think that you should try it with native Python environment. I tested and seems running successfully.

I think there is no connection with "Python environment", because I try it in Colaboratory, same error.

@minda163
Copy link

minda163 commented Mar 7, 2019

my problem was a lambda layer in my training loop. Please check if you have any and try to substitute it.

But I need this lambda layer!! Do you know another solution?

@grayfall
Copy link

grayfall commented Mar 7, 2019

@minda163 I've resorted to saving weigths alone and using a constructor function to build the model itself. It seems like there is no other way around this, because pickle can't handle closures. The devs might want to consider dill as a more robust alternative to pickle.

@minda163
Copy link

minda163 commented Mar 8, 2019

172                     reductor = getattr(x, "__reduce_ex__", None)
173                     if reductor:

--> 174 rv = reductor(4)
175 else:
176 reductor = getattr(x, "reduce", None)

TypeError: can't pickle _thread.lock objects

Did you fix it. I have the same error?

@alar0330
Copy link

alar0330 commented Mar 19, 2019

Although that the issues discussed in this thread are somewhat diverse in their origin, this answer on Stackoverflow might help some in looking for a solution to their problem.

@SumitNikam
Copy link

@ebrevdo

import pickle
from keras.layers import Dense
from keras.layers import LSTM
from keras.models import Sequential
from keras.metrics import categorical_accuracy

model = Sequential()
model.add(LSTM(20, return_sequences=True, stateful=False, batch_input_shape=(10, 20, 4)))
model.add(Dense(3, activation='softmax'))

model.compile(loss="categorical_crossentropy",
              optimizer='adam',
              metrics=[categorical_accuracy],
              sample_weight_mode='temporal')

data_path = '/home/ubuntu/invoice/data/'     #any path to store pickle dump
output_file_path = data_path + 'model.dat'
with open(output_file_path, 'wb') as f:
    pickle.dump(model, f)

Error Msg:

Traceback (most recent call last):
  File "<input>", line 19, in <module>
TypeError: can't pickle _thread.lock objects

Do you solve this issue ? If you solve please suggest me to modifications please

@ebrevdo
Copy link
Contributor

ebrevdo commented Oct 7, 2019 via email

@1vecera
Copy link

1vecera commented Oct 10, 2019

@ebrevdo Well, still huge amount of non-tf packages I want to use with my tf model depend on a pickle. I encountered the issue 3 times:
1: Trying to save my model with pickle after saving and loading it with custom objects did not work
2: Wrapping it in Keras Wrapper so it is usable in Scikit-Like APIs for Sampling, Interpretation and more
3: Trying to train my model in parallel as I can only use around 30% of the 40 CPUs the server has. (

@GGDRriedel
Copy link

GGDRriedel commented Feb 24, 2021

What happened to this issue?
I am running into it as well with this code

def nn_model3(in_shape,out_shape): 
    model=tf.keras.models.Sequential()
    model.add(tf.keras.layers.Conv1D(filters=256,kernel_size=3,activation='relu',input_shape=[256,1,]))
    model.add( tf.keras.layers.Conv1D(filters=64, kernel_size=6, activation='relu'))
    model.add(tf.keras.layers.Dropout(0.2))
    model.add(tf.keras.layers.Flatten())
    model.add(tf.keras.layers.Dense(256, activation = "relu"))
    model.add(tf.keras.layers.Lambda(lambda x: tf.expand_dims(model.output, axis=-1)))
    model.add(CuDNNLSTM(100,  return_sequences=True))
    model.add(tf.keras.layers.Flatten())
    model.add( tf.keras.layers.Dense(189, kernel_initializer='normal'))#, activation='sigmoid'))
    model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])
    return model
    ```

nn_model3.save('filename.h5') now throws the error

Thing is I need that  
```    model.add(tf.keras.layers.Lambda(lambda x: tf.expand_dims(model.output, axis=-1)))```

For the LSTM to work. 

The workarounds of declaring statics does not work for me since I need that model.output to be evaluated dynamically

@ebrevdo
Copy link
Contributor

ebrevdo commented Feb 24, 2021

@GGDRriedel this error can happen for many reasons; so your instance may differ a bit from the previous instances in this thread. I suggest opening a new issue on github and answering some of the questions in the template (version of TF, etc) and someone from the keras team can likely help you.

@GGDRriedel
Copy link

It's still the inability to pickle dynamic model structures from

my problem was a lambda layer in my training loop. Please check if you have any and try to substitute it.

But I need this lambda layer!! Do you know another solution?

so it's basically the same problem.
Anyway I resorted to saving just the weights as a workaround but obviously this won't work in deployment/production and so on

@ebrevdo
Copy link
Contributor

ebrevdo commented Feb 25, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stat:awaiting tensorflower Status - Awaiting response from tensorflower
Projects
None yet
Development

No branches or pull requests