Skip to content

Commit

Permalink
Adding no_rocm tag to the unit-test //tensorflow/python/keras/distrib…
Browse files Browse the repository at this point in the history 
…ute:saved_model_mixed_api_test_gpu

Three subtests started failing (as a consequence of some change within the last week or so?). Still need to root-cause the failures, but adding the no_rocm tag for now to get ROCm unit-tests passing

error messages we get are

```
======================================================================
ERROR: test_save_strategy_restore_strategy_test_distributionforrestoring_Mirrored1GPU_distributionforsaving_MirroredCPUAndGPU_mode_eager_modelandinput_SimpleSequentialModel_saveinscope_False (__main__.SavedModelSaveAndLoadTest)
test_save_strategy_restore_strategy_test_distributionforrestoring_Mirrored1GPU_distributionforsaving_MirroredCPUAndGPU_mode_eager_modelandinput_SimpleSequentialModel_saveinscope_False (__main__.SavedModelSaveAndLoadTest)
test_save_strategy_restore_strategy_test_distributionforrestoring_Mirrored1GPU_distributionforsaving_MirroredCPUAndGPU_mode_eager_modelandinput_SimpleSequentialModel_saveinscope_False(distribution_for_restoring=Mirrored1GPU, distribution_for_saving=MirroredCPUAndGPU, mode='eager', model_and_input=SimpleSequentialModel, save_in_scope=False)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/testing_utils.py", line 997, in decorated
    f(self, *args, **kwargs)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/absl_py/absl/testing/parameterized.py", line 263, in bound_param_test
    test_method(self, **testcase_params)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 366, in decorated
    execute_test_method()
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 349, in execute_test_method
    test_method(**kwargs_to_pass)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 517, in decorator
    test_method(self, **kwargs)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/distribute/saved_model_mixed_api_test.py", line 78, in test_save_strategy_restore_strategy
    save_in_scope)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/distribute/saved_model_test_base.py", line 248, in run_test_save_strategy_restore_strategy
    self._train_model(model, x_train, y_train, batch_size)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/distribute/saved_model_test_base.py", line 170, in _train_model
    model.fit(x=training_dataset, epochs=1, steps_per_epoch=100)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/engine/training.py", line 1188, in fit
    tmp_logs = self.train_function(iterator)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/def_function.py", line 885, in __call__
    result = self._call(*args, **kwds)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/def_function.py", line 913, in _call
    return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/function.py", line 3033, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/function.py", line 1957, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/function.py", line 596, in call
    ctx=ctx)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  Input to reshape is a tensor with 5 values, but the requested shape has 1145896960
	 [[node gradient_tape/mean_squared_error/Reshape (defined at usr/lib/python3.6/threading.py:916) ]]
	 [[SGD/AddN_1/_52]]
  (1) Invalid argument:  Input to reshape is a tensor with 5 values, but the requested shape has 1145896960
	 [[node gradient_tape/mean_squared_error/Reshape (defined at usr/lib/python3.6/threading.py:916) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_195971]

Function call stack:
train_function -> train_function

======================================================================
ERROR: test_save_strategy_restore_strategy_test_distributionforrestoring_Mirrored1GPU_distributionforsaving_OneDeviceGPU_mode_eager_modelandinput_SimpleSubclassModel_saveinscope_True (__main__.SavedModelSaveAndLoadTest)
test_save_strategy_restore_strategy_test_distributionforrestoring_Mirrored1GPU_distributionforsaving_OneDeviceGPU_mode_eager_modelandinput_SimpleSubclassModel_saveinscope_True (__main__.SavedModelSaveAndLoadTest)
test_save_strategy_restore_strategy_test_distributionforrestoring_Mirrored1GPU_distributionforsaving_OneDeviceGPU_mode_eager_modelandinput_SimpleSubclassModel_saveinscope_True(distribution_for_restoring=Mirrored1GPU, distribution_for_saving=OneDeviceGPU, mode='eager', model_and_input=SimpleSubclassModel, save_in_scope=True)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/testing_utils.py", line 997, in decorated
    f(self, *args, **kwargs)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/absl_py/absl/testing/parameterized.py", line 263, in bound_param_test
    test_method(self, **testcase_params)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 366, in decorated
    execute_test_method()
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 349, in execute_test_method
    test_method(**kwargs_to_pass)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 517, in decorator
    test_method(self, **kwargs)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/distribute/saved_model_mixed_api_test.py", line 78, in test_save_strategy_restore_strategy
    save_in_scope)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/distribute/saved_model_test_base.py", line 248, in run_test_save_strategy_restore_strategy
    self._train_model(model, x_train, y_train, batch_size)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/distribute/saved_model_test_base.py", line 170, in _train_model
    model.fit(x=training_dataset, epochs=1, steps_per_epoch=100)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/engine/training.py", line 1188, in fit
    tmp_logs = self.train_function(iterator)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/def_function.py", line 885, in __call__
    result = self._call(*args, **kwds)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/def_function.py", line 913, in _call
    return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/function.py", line 3033, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/function.py", line 1957, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/function.py", line 596, in call
    ctx=ctx)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  Input to reshape is a tensor with 10 values, but the requested shape has 0
	 [[node gradient_tape/mean_squared_error/Reshape (defined at root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/distribute/saved_model_mixed_api_test.py:78) ]] [Op:__inference_train_function_216342]

Function call stack:
train_function

======================================================================
ERROR: test_save_strategy_restore_strategy_test_distributionforrestoring_OneDeviceCPU_distributionforsaving_MirroredCPUAndGPU_mode_eager_modelandinput_SimpleSubclassModel_saveinscope_False (__main__.SavedModelSaveAndLoadTest)
test_save_strategy_restore_strategy_test_distributionforrestoring_OneDeviceCPU_distributionforsaving_MirroredCPUAndGPU_mode_eager_modelandinput_SimpleSubclassModel_saveinscope_False (__main__.SavedModelSaveAndLoadTest)
test_save_strategy_restore_strategy_test_distributionforrestoring_OneDeviceCPU_distributionforsaving_MirroredCPUAndGPU_mode_eager_modelandinput_SimpleSubclassModel_saveinscope_False(distribution_for_restoring=OneDeviceCPU, distribution_for_saving=MirroredCPUAndGPU, mode='eager', model_and_input=SimpleSubclassModel, save_in_scope=False)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/testing_utils.py", line 997, in decorated
    f(self, *args, **kwargs)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/absl_py/absl/testing/parameterized.py", line 263, in bound_param_test
    test_method(self, **testcase_params)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 366, in decorated
    execute_test_method()
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/framework/test_combinations.py", line 349, in execute_test_method
    test_method(**kwargs_to_pass)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/distribute/combinations.py", line 517, in decorator
    test_method(self, **kwargs)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/distribute/saved_model_mixed_api_test.py", line 78, in test_save_strategy_restore_strategy
    save_in_scope)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/distribute/saved_model_test_base.py", line 248, in run_test_save_strategy_restore_strategy
    self._train_model(model, x_train, y_train, batch_size)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/distribute/saved_model_test_base.py", line 170, in _train_model
    model.fit(x=training_dataset, epochs=1, steps_per_epoch=100)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/keras/engine/training.py", line 1188, in fit
    tmp_logs = self.train_function(iterator)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/def_function.py", line 885, in __call__
    result = self._call(*args, **kwds)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/def_function.py", line 913, in _call
    return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/function.py", line 3033, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/function.py", line 1957, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/function.py", line 596, in call
    ctx=ctx)
  File "/root/.cache/bazel/_bazel_root/efb88f6336d9c4a18216fb94287b8d97/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/keras/distribute/saved_model_mixed_api_test_gpu.runfiles/org_tensorflow/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  Input to reshape is a tensor with 5 values, but the requested shape has 1120437399847525866
	 [[node gradient_tape/mean_squared_error/Reshape (defined at usr/lib/python3.6/threading.py:916) ]]
  (1) Invalid argument:  Input to reshape is a tensor with 5 values, but the requested shape has 1120437399847525866
	 [[node gradient_tape/mean_squared_error/Reshape (defined at usr/lib/python3.6/threading.py:916) ]]
	 [[SGD/AddN_1/_52]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_300901]

Function call stack:
train_function -> train_function

----------------------------------------------------------------------
Ran 691 tests in 683.294s

FAILED (errors=3, skipped=421)
```
  • Loading branch information
@deven-amd
deven-amd committed May 14, 2021
1 parent 8f442e0 commit f4c6913
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions tensorflow/python/keras/distribute/BUILD
View file
Original file line number Diff line number Diff line change
Expand Up @@ -843,6 +843,7 @@ distribute_py_test(
shard_count = 7,
tags = [
"multi_and_single_gpu",
"no_rocm",
],
xla_tags = [
"no_cuda_asan", # times out
Expand Down

0 comments on commit f4c6913

Please sign in to comment.