Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feed_dict={K.learning_phase(): 1} while executing train_mitoses.py #3

Closed
sukruburakcetin opened this issue Oct 3, 2017 · 11 comments
Closed

feed_dict={K.learning_phase(): 1} while executing train_mitoses.py #3

sukruburakcetin opened this issue Oct 3, 2017 · 11 comments

Comments

@sukruburakcetin
Copy link

sukruburakcetin commented Oct 3, 2017 •
edited
Loading 

Traceback (most recent call last):
  File "E:/deep-histopath/train_mitoses.py", line 751, in <module>
    args.log_interval, args.threads, args.checkpoint, args.resume)
  File "E:/deep-histopath/train_mitoses.py", line 600, in train
    feed_dict={K.learning_phase(): 1})
  File "C:\Users\Burak\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 895, in run
    run_metadata_ptr)
  File "C:\Users\Burak\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\Burak\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1321, in _do_run
    options, run_metadata)
  File "C:\Users\Burak\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds.
	 [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
	 [[Node: data/IteratorGetNext = IteratorGetNext[output_shapes=[[?,64,64,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/cpu:0"](data/Iterator)]]
2017-10-03 11:37:55.026389: W C:\tf_jenkins\home\workspace\rel-win\M\windows\PY\35\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 11:37:55.036743: W C:\tf_jenkins\home\workspace\rel-win\M\windows\PY\35\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.

I extracted train and validation patches using "python preprocess_mitoses.py". Once I wanted to train the default vgg model with those patches, I got the error which is shown above.

Actually I do not know how to manage with the problem, due to fact that I do not get a clue about the error itself. Would you mind if I seek for your help? Thank you.

OS: Windows
tensorflow == 1.3.0(cpu loaded)
Keras== 2.0.8
anaconda == 3.5.2
conda == 4.3.27
@sukruburakcetin
Copy link
Author

@dusenberrymw

and these are other errors that I've got during compiling:

Traceback (most recent call last):
  File "C:\Users\Burak\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1327, in _do_call
    return fn(*args)
  File "C:\Users\Burak\Anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1306, in _run_fn
    status, run_metadata)
  File "C:\Users\Burak\Anaconda3\lib\contextlib.py", line 66, in __exit__
    next(self.gen)
  File "C:\Users\Burak\Anaconda3\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds.
         [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
         [[Node: data/IteratorGetNext = IteratorGetNext[output_shapes=[[?,64,64,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/cpu:0"](data/Iterator)]]

@sukruburakcetin
Copy link
Author

Now, I wanted to try something different:

I changed my Anaconda version from 3.5 to 3.6 and changed cudnn version from v5.1 to v6.0.

(tensorflow) C:\Users\Burak\Desktop\downloads2\tensorflow-status-checker>python tensorflow_self_check.py
TensorFlow successfully installed.
The installed version of TensorFlow includes GPU support.

After that, I rerun the code itself with the same exact configuration, but still getting same error.

(tensorflow) E:\deep-histopath>python train_mitoses.py
Using TensorFlow backend.
experiment directory: experiments\mitoses\171003_172944_vgg_mokoko
2017-10-03 17:29:45.435787: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2017-10-03 17:29:45.737953: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Found device 0 with properties:
name: GeForce GT 710 major: 3 minor: 5 memoryClockRate(GHz): 0.954
pciBusID: 0000:01:00.0
totalMemory: 1.00GiB freeMemory: 838.58MiB
2017-10-03 17:29:45.738055: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GT 710, pci bus id: 0000:01:00.0, compute capability: 3.5)
2017-10-03 17:30:28.563156: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.563205: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.563313: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.575001: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
         [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
2017-10-03 17:30:28.575028: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.585909: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.589385: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.597035: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.600166: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.606988: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
Traceback (most recent call last):
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1323, in _do_call
    return fn(*args)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1302, in _run_fn
    status, run_metadata)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\contextlib.py", line 88, in __exit__
    next(self.gen)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 467, in raise_exception_on_not_ok_status
    c_api.TF_GetCode(status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds.
         [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
         [[Node: data/IteratorGetNext = IteratorGetNext[output_shapes=[[?,64,64,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/cpu:0"](data/Iterator)]]
         [[Node: data/boolean_mask_2/Shape/_287 = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_668_data/boolean_mask_2/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train_mitoses.py", line 752, in <module>
    args.log_interval, args.threads, args.checkpoint, args.resume)
  File "train_mitoses.py", line 600, in train
    feed_dict={K.learning_phase(): 1})
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 889, in run
    run_metadata_ptr)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _do_run
    options, run_metadata)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds.
         [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
         [[Node: data/IteratorGetNext = IteratorGetNext[output_shapes=[[?,64,64,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/cpu:0"](data/Iterator)]]
         [[Node: data/boolean_mask_2/Shape/_287 = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_668_data/boolean_mask_2/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
2017-10-03 17:30:28.626792: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.626985: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.632310: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.639850: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.654266: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.656794: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.662539: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.671291: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.681184: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.684647: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.692484: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.710251: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.710270: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.710258: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.722299: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.732081: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.

@dusenberrymw
Copy link
Collaborator

Hi @Narthyard, thanks for reaching out, and thanks for trying out the code! Interesting issue -- I have not run into this issue on my current setup, so this will be a good exercise to find out what is going on in the case of different setups.

Looking at the error message, it looks like it is an issue either with the Dataset API, or with the tf.boolean_mask calls I'm using on the image tensors coming out of the Dataset object.

tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds.
         [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
         [[Node: data/IteratorGetNext = IteratorGetNext[output_shapes=[[?,64,64,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/cpu:0"](data/Iterator)]]
         [[Node: data/boolean_mask_2/Shape/_287 = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_668_data/boolean_mask_2/Shape", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
2017-10-03 17:30:28.626792: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.626985: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.632310: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.639850: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.654266: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.656794: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.662539: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.671291: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.681184: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.684647: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.692484: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.710251: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.710270: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.710258: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.722299: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-03 17:30:28.732081: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.

I'll investigate this a bit further with the TensorFlow debugger and see if I can determine the exact op and get back to you.

@sukruburakcetin
Copy link
Author

Hi @dusenberrymw, I'd like to say big thank you for your kind, quick response and your amazing effort.

Do you mind if I ask what is your current set up's version numbers?, I looked around but I couldn't find any info about it.

@sukruburakcetin
Copy link
Author

@dusenberrymw By the way, I didn't download the testing_image_data due to fact that I just wanted to train the model without validation. Is the error itself correlate with lacking test image data? I'm not sure about it, that may be a silly question.

@dusenberrymw
Copy link
Collaborator

dusenberrymw commented Oct 5, 2017 •
edited
Loading

@Narthyard I debugged this a bit, and it was indeed an issue with the output of the Dataset API. I believe it was due to a file pattern used to match train/val images that had paths specific to *nix operating systems, and thus was failing on Windows. I pushed an update to make this file patten generic -- can you grab the latest code and try it again?

As for the testing images, your original intuition was correct -- for training the model, the testing images are not necessary. For clarity, the preprocess script will still split the training images into a training and validation set, but those still come from the original training folder, and not the testing folder.

Here is my current setup:

  • OS: macOS & Linux
  • Python 3.6 (official release, and miniconda release)
  • Keras bleeding edge (not strictly necessary to be bleeding edge, but I make updates sometimes)
  • Tensorflow bleeding edge (not necessary, latest 1.3 release should be fine -- that being said, the Dataset API is still new (and it just moved out of staging in tf.contrib to it's new home in tf.data), so I might end up needing to bump that requirement up)
  • everything else is the latest version installed with pip

Please let me know if the latest change fixes the issue!

@sukruburakcetin
Copy link
Author

@dusenberrymw I grabbed your current update by hoping to solve the problem(your solution was indeed sounded plausible) , but unfortunately still facing with the same problem.

C:\Users\Burak\Anaconda3\envs\tensorflow\python.exe E:/dusenberrymw/deep-histopath/train_mitoses.py --exp_name MitosisTraining --exp_name_suffix Version_SupportWindowsPaths
Using TensorFlow backend.
experiment directory: experiments\mitoses\MitosisTraining_Version_SupportWindowsPaths
2017-10-05 11:30:13.586755: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2017-10-05 11:30:13.825582: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Found device 0 with properties: 
name: GeForce GT 710 major: 3 minor: 5 memoryClockRate(GHz): 0.954
pciBusID: 0000:01:00.0
totalMemory: 1.00GiB freeMemory: 837.65MiB
2017-10-05 11:30:13.825833: I C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1055] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GT 710, pci bus id: 0000:01:00.0, compute capability: 3.5)
2017-10-05 11:30:38.293369: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.293387: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.293413: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.296365: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.296572: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.299372: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.299383: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
	 [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
2017-10-05 11:30:38.299481: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.302215: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.304406: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.331863: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.341073: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.341304: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.343026: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.343388: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.346331: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.346564: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.348455: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.348663: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\frameworkTraceback (most recent call last):
\op_kernel.cc:  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1323, in _do_call
1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
    return fn(*args)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1302, in _run_fn
    status, run_metadata)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\contextlib.py", line 88, in __exit__
    next(self.gen)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 467, in raise_exception_on_not_ok_status
    c_api.TF_GetCode(status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds.
	 [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
	 [[Node: data/IteratorGetNext = IteratorGetNext[output_shapes=[[?,64,64,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/cpu:0"](data/Iterator)]]
	 [[Node: data/IteratorGetNext/_195 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_243_data/IteratorGetNext", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:/dusenberrymw/deep-histopath/train_mitoses.py", line 751, in <module>
    args.log_interval, args.threads, args.checkpoint, args.resume)
  File "E:/dusenberrymw/deep-histopath/train_mitoses.py", line 600, in train
    feed_dict={K.learning_phase(): 1})
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 889, in run
    run_metadata_ptr)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _do_run
    options, run_metadata)
  File "C:\Users\Burak\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index -1 of dimension 0 out of bounds.
	 [[Node: strided_slice = StridedSlice[Index=DT_INT32, T=DT_STRING, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1](StringSplit:1, strided_slice/stack, strided_slice/stack_1, strided_slice/stack_2)]]
	 [[Node: data/IteratorGetNext = IteratorGetNext[output_shapes=[[?,64,64,3], [?], [?]], output_types=[DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/cpu:0"](data/Iterator)]]
	 [[Node: data/IteratorGetNext/_195 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_243_data/IteratorGetNext", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
2017-10-05 11:30:38.360602: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.360683: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.360864: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.360961: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.363820: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.375785: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.375858: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.376007: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.376372: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.378195: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.380096: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.380135: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.380207: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.381332: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.382377: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.384195: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.385019: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.384275: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.385913: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.386316: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.387240: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.388187: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.388614: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.390043: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.398258: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.398334: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.398637: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.398770: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.400192: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.400990: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.414696: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.414768: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.415164: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.427581: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.427654: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.427609: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.427978: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.430165: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.430358: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.430729: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.432163: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.443497: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.443526: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.443728: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.455431: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.462065: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.471308: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.471325: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.480262: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.480291: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.480494: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.
2017-10-05 11:30:38.482919: W C:\tf_jenkins\home\workspace\tf-nightly-windows\M\windows-gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1192] Invalid argument: slice index -1 of dimension 0 out of bounds.

Process finished with exit code 1

@sukruburakcetin
Copy link
Author

Hi again @dusenberrymw, I'ld like to mention that I tried your work on Linux platform(ubuntu 16.04LTS), I reduced training batch size due to limited GPU ram availability and it is working flawlessly.

@dusenberrymw
Copy link
Collaborator

Hi @Narthyard, that's great that you were able to successfully run it on Linux. Although I didn't post an update here yesterday, we continued to look into it, and @gweidner was able to reproduce the exact issue on a Windows setup. Thus, it appears to be a core TensorFlow issue. We'll create a minimal example that causes the same issue, and then look into options.

For your particular workflow, is your Linux setup sufficient for you to be able to move forward for now, or is this issue on Windows still a blocker?

Thanks again for trying out the code!

@sukruburakcetin
Copy link
Author

Hi @dusenberrymw, first of all, I'd like to say thank you for taking the time to work with me. You really went out of your way to ensure my case is solved the way I like.

So far so good, I'll carry on with Linux platform for now. I'll limit the number of patches to examine the traning model efficiently and quickly. The current set up produces 782,490 patches, and my GPU and its ram capability are way too low. I tried to set them all but it takes ages to feed the model, so I interrupted it after one and a half day.

Now, I will just use limited number of patches to be ensure that system is producing the desired outputs. Then, I may try it on super computer to examine the environment in a different way, so I also log the traning time when the system setup is premium.

I'm looking forward to try your freshly added and updated "predict_mitosis" code. I'll report for the upcoming errors if any. You are doing amazing job, thanks again for both of you. @gweidner

@dusenberrymw
Copy link
Collaborator

@Narthyard Your welcome, and thank you for taking the time to try out the codebase and report issues! I'm going to close this issue now and keep track of the dependency bug as a separate task. Thanks for reporting it and trying out the code!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dusenberrymw @sukruburakcetin