Contrib support on Windows

[alarmz]

step to reproduce:

attached file (https://www.dropbox.com/s/7yglwzjx4tyoguz/save_restore_model.zip?dl=0)
is my source code
I am testing saver.save and saver.restore function under windows.
working envirment : Windows R2 Server 2012 x64,
python 3.5 x64, TF 1.0 and master daily build, working with virtualenv
I also tested TF 0.12

1. download source code save into C:\work\tensorflow001 ,
2. create folder mnist
3. download files
   (t10k-images-idx3-ubyte.gz,
   t10k-labels-idx1-ubyte.gz,
   train-images-idx3-ubyte.gz,
   train-labels-idx1-ubyte.gz) from http://yann.lecun.com/exdb/mnist/
   and save into mist folder(why not using input_data from examples? because that is another bug...I think.)
4. create folder "model" for saving values
5. python save_restore_model.py
6. output as fallow

(venv643) C:\work\tensorflow001>python save_restore_model.py
Extracting mnist\train-images-idx3-ubyte.gz
Extracting mnist\train-labels-idx1-ubyte.gz
Extracting mnist\t10k-images-idx3-ubyte.gz
Extracting mnist\t10k-labels-idx1-ubyte.gz
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "BestSplits" device_type: "CPU"') for unknown op: BestSplits
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "CountExtremelyRandomStats" device_type: "CPU"') for unknown op: CountExtrem
elyRandomStats
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "FinishedNodes" device_type: "CPU"') for unknown op: FinishedNodes
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "GrowTree" device_type: "CPU"') for unknown op: GrowTree
E c:\tf_jenkins\home\workspace\release-


win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "SampleInputs" device_type: "CPU"') for unknown op: SampleInputs
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "ScatterAddNdim" device_type: "CPU"') for unknown op: ScatterAddNdim
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TopNInsert" device_type: "CPU"') for unknown op: TopNInsert
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TopNRemove" device_type: "CPU"') for unknown op: TopNRemove
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TreePredictions" device_type: "CPU"') for unknown op: TreePredictions
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "UpdateFertileSlots" device_type: "CPU"') for unknown op: UpdateFertileSlots

E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "ReinterpretStringToFloat" device_type: "CPU"') for unknown op: ReinterpretS
tringToFloat
WARNING:tensorflow:From save_restore_model.py:66: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.
Starting 1st session...
Epoch: 0001 cost= 171.464307973
Epoch: 0002 cost= 43.585798355
Epoch: 0003 cost= 27.564399517
First Optimization Finished!
Accuracy: 0.9124
Model saved in file: C:\work\tensorflow001\model\result
Starting 2nd session...
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested


W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Out of range: Read fewer bytes than requested
Traceback (most recent call last):  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\client\session.py", line 1
022, in _do_call
    return fn(*args)
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\client\session.py", line 1
004, in _run_fn
    status, run_metadata)
  File "c:\python35\Lib\contextlib.py", line 66, in __exit__
    next(self.gen)
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\framework\errors_impl.py",
 line 469, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.OutOfRangeError: Read fewer bytes than requested
         [[Node: save/RestoreV2_3 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/t
ask:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_3/tensor_names, save/RestoreV2_3/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "save_restore_model.py", line 113, in <module>
    saver.restore(sess, model_path)
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\training\saver.py", line 1
439, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\client\session.py", line 7
67, in run
    run_metadata_ptr)
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\client\session.py", line 9
65, in _run
    feed_dict_string, options, run_metadata)
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\client\session.py", line 1
015, in _do_run
    target_list, options, run_metadata)
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\client\session.py", line 1
035, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: Read fewer bytes than requested
         [[Node: save/RestoreV2_3 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/t
ask:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_3/tensor_names, save/RestoreV2_3/shape_and_slices)]
]

Caused by op 'save/RestoreV2_3', defined at:
  File "save_restore_model.py", line 69, in <module>
    saver = tf.train.Saver()
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\training\saver.py", line 1
051, in __init__
    self.build()
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\training\saver.py", line 1
081, in build
    restore_sequentially=self._restore_sequentially)
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\training\saver.py", line 6
75, in build
    restore_sequentially, reshape)
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\training\saver.py", line 4
02, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\training\saver.py", line 2
42, in restore_op
    [spec.tensor.dtype])[0])
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 6
68, in restore_v2
    dtypes=dtypes, name=name)
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\framework\op_def_library.p
y", line 763, in apply_op
    op_def=op_def)
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\framework\ops.py", line 23
92, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "C:\work\tensorflow001\venv643\lib\site-packages\tensorflow\python\framework\ops.py", line 12
64, in __init__
    self._traceback = _extract_stack()

OutOfRangeError (see above for traceback): Read fewer bytes than requested
         [[Node: save/RestoreV2_3 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/t
ask:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_3/tensor_names, save/RestoreV2_3/shape_and_slices)]

expect result

why fallow message shows up ?

E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "BestSplits" device_type: "CPU"') for unknown op:

most import is when I using saver.restore, it always show fallowing error message

OutOfRangeError (see above for traceback): Read fewer bytes than requested
         [[Node: save/RestoreV2_3 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/t
ask:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_3/tensor_names, save/RestoreV2_3/shape_and_slices)]

why I cannot download example automatic by using

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("mnist", one_hot=True)

[jart]

Thanks for reaching out and we're sorry to hear about the issue you're facing. Could you please provide a minimal inline code example demonstrating the bug and showing how to reproduce? Downloading a zip file and troubleshooting another app is not a level of support we're able to offer.

[alarmz]

Thanks your response,
fallow is my code:
please download checkpoint that created by TensorFlow saver
https://www.dropbox.com/s/v5xl1dzssi58opf/model.zip?dl=0
copy it to "C:\work\tensorflow001\model"
in model will have 5 files, checkpoint, result, result.index.....
when TensorFlow restore checkpoint will crash.

from __future__ import print_function

# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("mnist", one_hot=True)

import tensorflow as tf

# Parameters
learning_rate = 0.001
batch_size = 100
display_step = 1
model_path = u"C:\\work\\tensorflow001\\model\\result"

# Initializing the variables
init = tf.initialize_all_variables()

# 'Saver' op to save and restore all the variables
saver = tf.train.Saver()

# Running a new session
print("Starting 2nd session...")

with tf.Session() as sess:
    # Initialize variables
    sess.run(init)

    # Restore model weights from previously saved model
    saver.restore(sess, model_path)

[jart]

Hmm if you insert a line into your script to trigger the lazy loading of the contrib library, e.g. _ = tf.contrib.tensor_forest then does it work?

[alarmz]

thanks your response,
I am insert that line of code, please check it out,
if that insert to right place, the result still crash.

from __future__ import print_function

# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("mnist", one_hot=True)

import tensorflow as tf

# Parameters
learning_rate = 0.001
batch_size = 100
display_step = 1
model_path = u"C:\\work\\tensorflow01\\model\\result"

# Initializing the variables
init = tf.initialize_all_variables()

# 'Saver' op to save and restore all the variables
saver = tf.train.Saver()

# Running a new session
print("Starting 2nd session...")
_ = tf.contrib.tensor_forest
with tf.Session() as sess:
    # Initialize variables
    sess.run(init)
    
    # Restore model weights from previously saved model
    saver.restore(sess, model_path)

[jart]

Can you put that line right after the import tensorflow as tf line? I ask because your stack trace indicates the saver = ... line is what crashes, and the code above inserted it after that.

I'm pretty confident that my suggestion will fix your problem. This is the result of a known issue we're planning to fix eventually, where the build graph and op loading needs to be smarter about loading libraries. It's a hard problem to solve.

If your issue is not solved by my suggestion, let me know and I'll re-open.

[alarmz]

Thanks your answer, but it not work, still crash
the fallow is my code:
can you help me to verify, may I miss understand your meaning, insert the code to wrong place.

from __future__ import print_function

# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("mnist", one_hot=True)

import tensorflow as tf

# Parameters
learning_rate = 0.001
batch_size = 100
display_step = 1
model_path = u"C:\\work\\tensorflow01\\model\\result"

# Initializing the variables
init = tf.initialize_all_variables()

# 'Saver' op to save and restore all the variables
saver = tf.train.Saver()

# Running a new session
print("Starting 2nd session...")
_ = tf.contrib.tensor_forest  #here is that code
with tf.Session() as sess:
    # Initialize variables
    sess.run(init)
    
    # Restore model weights from previously saved model
    saver.restore(sess, model_path)

[jart]

Can you try this?

from __future__ import print_function

# Import MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("mnist", one_hot=True)

import tensorflow as tf
_ = tf.contrib.tensor_forest  #here is that code

# Parameters
learning_rate = 0.001
batch_size = 100
display_step = 1
model_path = u"C:\\work\\tensorflow01\\model\\result"

# Initializing the variables
init = tf.initialize_all_variables()

# 'Saver' op to save and restore all the variables
saver = tf.train.Saver()

# Running a new session
print("Starting 2nd session...")
with tf.Session() as sess:
    # Initialize variables
    sess.run(init)
    
    # Restore model weights from previously saved model
    saver.restore(sess, model_path)

[alarmz]

It still crash
do you have any idea ?

W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel
.cc:975] Out of range: Read fewer bytes than requested
Traceback (most recent call last):
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\client\session.py", line 102
1, in _do_call
    return fn(*args)
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\client\session.py", line 100
3, in _run_fn
    status, run_metadata)
  File "c:\program files\anaconda3\Lib\contextlib.py", line 66, in __exit__
    next(self.gen)
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\framework\errors_impl.py", l
ine 469, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.OutOfRangeError: Read fewer bytes than requested
         [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/tas
k:0/cpu:0"](_recv_save/Const_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "save_restore_model.py", line 114, in <module>
    saver.restore(sess, model_path)
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\training\saver.py", line 138
8, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\client\session.py", line 766
, in run
    run_metadata_ptr)
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\client\session.py", line 964
, in _run
    feed_dict_string, options, run_metadata)
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\client\session.py", line 101
4, in _do_run
    target_list, options, run_metadata)
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\client\session.py", line 103
4, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: Read fewer bytes than requested
         [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/tas
k:0/cpu:0"](_recv_save/Const_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op 'save/RestoreV2', defined at:
  File "save_restore_model.py", line 69, in <module>
    saver = tf.train.Saver()
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\training\saver.py", line 100
0, in __init__
    self.build()
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\training\saver.py", line 103
0, in build
    restore_sequentially=self._restore_sequentially)
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\training\saver.py", line 624
, in build
    restore_sequentially, reshape)
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\training\saver.py", line 361
, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\training\saver.py", line 200
, in restore_op
    [spec.tensor.dtype])[0])
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 441
, in restore_v2
    dtypes=dtypes, name=name)
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\framework\op_def_library.py"
, line 759, in apply_op
    op_def=op_def)
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\framework\ops.py", line 2240
, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "C:\work\tensorflow01\venv64\lib\site-packages\tensorflow\python\framework\ops.py", line 1128
, in __init__
    self._traceback = _extract_stack()

OutOfRangeError (see above for traceback): Read fewer bytes than requested
         [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/tas
k:0/cpu:0"](_recv_save/Const_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

[jart]

Oh now the problem seems obvious to me. Windows support is still brand new. We haven't had a chance yet to make contrib Windows compatible. It's only possible to use the core TensorFlow codebase right now. See: https://github.com/tensorflow/tensorflow/blob/d5062e8/tensorflow/python/BUILD#L81 This means it's not possible at the moment to load a model on Windows that uses ops from contrib.

Since I don't see an issue tracking this, I'm going to re-open and rename this one.

[mrry]

@jart Can you explain what this issue has to do with tf.contrib? I don't see any reference to tf.contrib in the original code, and—while you're right that the contrib libraries aren't fully supported on Windows—I can't see how that's related to @alarmz's original problem.

[jart]

In the original post, the following error message is cited:

why fallow message shows up ?
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "BestSplits" device_type: "CPU"') for unknown op:

That op is defined in tensorflow/contrib/tensor_forest/core/ops/best_splits_op.cc.

[mrry]

Ah I see. We should definitely fix this so that TensorForest works on Windows, but I think this log message (which should really be a warning or a fatal error) is a red herring. It looks like the real problem @alarmz is facing is in tf.train.Saver code and/or its related ops. Perhaps @concretevitamin has seen this before?

[mrry]

(PR #6908 should fix the error message and make much of tf.contrib.tensor_forest work on Windows.)

[drpngx]

@alarmz can you confirm that tf.contrib.tensor_forest works after that PR?

[mrry]

Closing this issue, since tf.contrib.tensor_forest has (as far as I know) been working with tests passing on Windows for some time now. Feel free to open a new issue if there are other parts of tf.contrib for which you'd like to see us prioritize support!