-
Notifications
You must be signed in to change notification settings - Fork 45.4k
Description
(tf) ubuntu@2a366117057b:~/MyFiles/tensorflow/models/research/deeplab$ sh local_test.sh
testBuildDeepLabWithDensePredictionCell (main.DeeplabModelTest) ... 2019-05-10 06:34:46.116049: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-05-10 06:34:46.185820: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-05-10 06:34:46.186271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.392
pciBusID: 0000:02:00.0
totalMemory: 3.95GiB freeMemory: 3.89GiB
2019-05-10 06:34:46.186290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-10 06:34:46.493327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 06:34:46.493370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-05-10 06:34:46.493393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-05-10 06:34:46.493594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1212 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
ok
testBuildDeepLabv2 (main.DeeplabModelTest) ... 2019-05-10 06:34:48.059227: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-10 06:34:48.059282: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 06:34:48.059292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-05-10 06:34:48.059299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-05-10 06:34:48.059453: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1212 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2019-05-10 06:34:52.003082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-10 06:34:52.003142: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 06:34:52.003152: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-05-10 06:34:52.003160: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-05-10 06:34:52.003316: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1212 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2019-05-10 06:34:59.589931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-10 06:34:59.589990: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 06:34:59.589999: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-05-10 06:34:59.590007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-05-10 06:34:59.590164: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1212 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2019-05-10 06:35:01.237604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-10 06:35:01.237665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 06:35:01.237674: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-05-10 06:35:01.237682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-05-10 06:35:01.237838: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1212 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
ok
testForwardpassDeepLabv3plus (main.DeeplabModelTest) ... 2019-05-10 06:35:03.889859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-10 06:35:03.889912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 06:35:03.889921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-05-10 06:35:03.889929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-05-10 06:35:03.890081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1212 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
ok
testWrongDeepLabVariant (main.DeeplabModelTest) ... ok
test_session (main.DeeplabModelTest)
Returns a TensorFlow Session for use in executing tests. ... ok
Ran 5 tests in 19.901s
OK
Uncompressing VOCtrainval_11-May-2012.tar
Removing the color map in ground truth annotations...
Converting PASCAL VOC 2012 dataset...
2019-05-10 06:42:26.091092: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-05-10 06:42:26.160215: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-05-10 06:42:26.160685: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.392
pciBusID: 0000:02:00.0
totalMemory: 3.95GiB freeMemory: 3.89GiB
2019-05-10 06:42:26.160703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-10 06:42:26.509786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 06:42:26.509824: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-05-10 06:42:26.509846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-05-10 06:42:26.510033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3625 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2019-05-10 06:42:26.565549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-10 06:42:26.565613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 06:42:26.565629: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-05-10 06:42:26.565659: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-05-10 06:42:26.565797: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3625 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
Converting image 366/1464 shard 0
Converting image 732/1464 shard 1
Converting image 1098/1464 shard 2
Converting image 1464/1464 shard 3
2019-05-10 06:43:07.100551: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-10 06:43:07.100605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 06:43:07.100615: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-05-10 06:43:07.100623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-05-10 06:43:07.100754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3625 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2019-05-10 06:43:07.109954: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-10 06:43:07.109978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 06:43:07.110003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-05-10 06:43:07.110012: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-05-10 06:43:07.110146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3625 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
Converting image 729/2913 shard 0
Converting image 1458/2913 shard 1
Converting image 2187/2913 shard 2
Converting image 2913/2913 shard 3
2019-05-10 06:43:47.943785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-10 06:43:47.943907: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 06:43:47.943952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-05-10 06:43:47.943985: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-05-10 06:43:47.944339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3625 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
2019-05-10 06:43:47.953998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-05-10 06:43:47.954022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-10 06:43:47.954046: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2019-05-10 06:43:47.954079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2019-05-10 06:43:47.954178: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3625 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
Converting image 363/1449 shard 0
Converting image 726/1449 shard 1
Converting image 1089/1449 shard 2
Converting image 1449/1449 shard 3
--2019-05-10 06:44:02-- http://download.tensorflow.org/models/deeplabv3_pascal_train_aug_2018_01_04.tar.gz
Resolving download.tensorflow.org (download.tensorflow.org)... 172.217.25.16, 2404:6800:4005:800::2010
Connecting to download.tensorflow.org (download.tensorflow.org)|172.217.25.16|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 460058541 (439M) [application/x-tar]
Saving to: ‘deeplabv3_pascal_train_aug_2018_01_04.tar.gz’
deeplabv3_pascal_tr 100%[===================>] 438.75M 24.9MB/s in 16s
2019-05-10 06:44:23 (27.7 MB/s) - ‘deeplabv3_pascal_train_aug_2018_01_04.tar.gz’ saved [460058541/460058541]
INFO:tensorflow:Training on trainval set
Traceback (most recent call last):
File "/home/ubuntu/MyFiles/tensorflow/models/research/deeplab/train.py", line 513, in
tf.app.run()
File "/home/ubuntu/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/ubuntu/MyFiles/tensorflow/models/research/deeplab/train.py", line 463, in main
dataset.get_one_shot_iterator(), dataset.num_of_classes,
File "/home/ubuntu/MyFiles/tensorflow/models/research/deeplab/datasets/data_generator.py", line 325, in get_one_shot_iterator
.map(self._preprocess_image, num_parallel_calls=self.num_readers))
File "/home/ubuntu/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 882, in map
return ParallelMapDataset(self, map_func, num_parallel_calls)
File "/home/ubuntu/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1899, in init
super(ParallelMapDataset, self).init(input_dataset, map_func)
File "/home/ubuntu/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1868, in init
self._map_func.add_to_graph(ops.get_default_graph())
File "/home/ubuntu/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/framework/function.py", line 475, in add_to_graph
self._create_definition_if_needed()
File "/home/ubuntu/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/framework/function.py", line 331, in _create_definition_if_needed
self._create_definition_if_needed_impl()
File "/home/ubuntu/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/framework/function.py", line 340, in _create_definition_if_needed_impl
self._capture_by_value, self._caller_device)
File "/home/ubuntu/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/framework/function.py", line 804, in func_graph_from_py_func
outputs = func(*func_graph.inputs)
File "/home/ubuntu/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1833, in tf_map_func
ret = map_func(nested_args)
File "/home/ubuntu/MyFiles/tensorflow/models/research/deeplab/datasets/data_generator.py", line 286, in _preprocess_image
crop_height=self.crop_size[0],
TypeError: 'map' object is not subscriptable