MNIST works, but word2vec_basic does not #514

Closed
dinedal opened this Issue Dec 15, 2015 · 3 comments

Comments

Projects
None yet
4 participants
@dinedal

dinedal commented Dec 15, 2015

Ubuntu 15.10, tensorflow 0.6.0, Nvida Titan X

All of my MNIST example runs work fine. Other examples work great too.

Can confirm they use the GPU fine with log output.

Here's the output of word2vec_basic.py

~/tensorflow/tensorflow/examples/tutorials/word2vec$ python word2vec_basic.py
I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcublas.so.7.0 locally
I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcudnn.so.6.5 locally
I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcufft.so.7.0 locally
I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcuda.so locally
I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcurand.so.7.0 locally
Found and verified text8.zip
Data size 17005207
Most common words (+UNK) [['UNK', 418391], ('the', 1061396), ('of', 593677), ('and', 416629), ('one', 411764)]
Sample data [5239, 3084, 12, 6, 195, 2, 3137, 46, 59, 156]
3084 -> 5239
originated -> anarchism
3084 -> 12
originated -> as
12 -> 6
as -> a
12 -> 3084
as -> originated
6 -> 195
a -> term
6 -> 12
a -> as
195 -> 2
term -> of
195 -> 6
term -> a
I tensorflow/core/common_runtime/local_device.cc:40] Local device intra op parallelism threads: 8
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:909] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:103] Found device 0 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.2405
pciBusID 0000:03:00.0
Total memory: 12.00GiB
Free memory: 11.21GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:127] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:137] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:702] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:03:00.0)
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Allocating 10.65GiB bytes.
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:52] GPU 0 memory begins at 0xb06c80000 extends to 0xdb07d7e67
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 4.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 8.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 16.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 32.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 64.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 128.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 256.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 512.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 4.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 8.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 16.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 32.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 64.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 128.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 256.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 512.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.00GiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.00GiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 4.00GiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 8.00GiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 16.00GiB
I tensorflow/core/common_runtime/direct_session.cc:58] Direct session inter op parallelism threads: 8
Initialized
Traceback (most recent call last):
  File "word2vec_basic.py", line 189, in <module>
    _, loss_val = session.run([optimizer, loss], feed_dict=feed_dict)
  File "/home/paul/.virtualenvs/mypy/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 368, in run
    results = self._do_run(target_list, unique_fetch_targets, feed_dict_string)
  File "/home/paul/.virtualenvs/mypy/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 444, in _do_run
    e.code)
tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'GradientDescent/update_Variable_2/ScatterSub': Could not satisfy explicit device specification '' because the node was colocated with a group of nodes that required incompatible device '/job:localhost/replica:0/task:0/GPU:0'
     [[Node: GradientDescent/update_Variable_2/ScatterSub = ScatterSub[T=DT_FLOAT, Tindices=DT_INT64, use_locking=false](Variable_2, gradients/concat_1, GradientDescent/update_Variable_2/mul)]]
Caused by op u'GradientDescent/update_Variable_2/ScatterSub', defined at:
  File "word2vec_basic.py", line 163, in <module>
    optimizer = tf.train.GradientDescentOptimizer(1.0).minimize(loss)
  File "/home/paul/.virtualenvs/mypy/local/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 188, in minimize
    name=name)
  File "/home/paul/.virtualenvs/mypy/local/lib/python2.7/site-packages/tensorflow/python/training/optimizer.py", line 289, in apply_gradients
    update_ops.append(self._apply_sparse(grad, var))
  File "/home/paul/.virtualenvs/mypy/local/lib/python2.7/site-packages/tensorflow/python/training/gradient_descent.py", line 59, in _apply_sparse
    return var.scatter_sub(delta, use_locking=self._use_locking)
  File "/home/paul/.virtualenvs/mypy/local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 392, in scatter_sub
    use_locking=use_locking)
  File "/home/paul/.virtualenvs/mypy/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_state_ops.py", line 227, in scatter_sub
    name=name)
  File "/home/paul/.virtualenvs/mypy/local/lib/python2.7/site-packages/tensorflow/python/ops/op_def_library.py", line 664, in apply_op
    op_def=op_def)
  File "/home/paul/.virtualenvs/mypy/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1834, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/paul/.virtualenvs/mypy/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1043, in __init__
    self._traceback = _extract_stack()
@almostimplemented

This comment has been minimized.

Show comment
Hide comment
@almostimplemented

almostimplemented Dec 15, 2015

Just had the same error on Ubuntu 14.04 + NVIDIA GeForce GTX 780 Ti.

Just had the same error on Ubuntu 14.04 + NVIDIA GeForce GTX 780 Ti.

@zielmicha

This comment has been minimized.

Show comment
Hide comment
@zielmicha

zielmicha Dec 15, 2015

I have the same problem (TensorFlow 0.6.0, Ubuntu 15.10, GeForce GTX 750 Ti). word2vec works correctly when using CPU-only version of TensorFlow.

I have the same problem (TensorFlow 0.6.0, Ubuntu 15.10, GeForce GTX 750 Ti). word2vec works correctly when using CPU-only version of TensorFlow.

@dennybritz

This comment has been minimized.

Show comment
Hide comment
@dennybritz

dennybritz Dec 18, 2015

Contributor

I think that's related to #305. You currently need to force the embedding op onto the CPU. Should be an easy fix, I will submit a PR.

Contributor

dennybritz commented Dec 18, 2015

I think that's related to #305. You currently need to force the embedding op onto the CPU. Should be an easy fix, I will submit a PR.

dennybritz added a commit to dennybritz/tensorflow that referenced this issue Dec 18, 2015

dennybritz added a commit to dennybritz/tensorflow that referenced this issue Dec 21, 2015

dennybritz added a commit to dennybritz/tensorflow that referenced this issue Dec 21, 2015

@vrv vrv closed this in #544 Dec 21, 2015

vrv added a commit that referenced this issue Dec 21, 2015

Merge pull request #544 from dennybritz/fix/word2vec-basic-cpu
Force embedding_lookup onto CPU in basic word2vec example - Fixes #514
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment