Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ROCm] Fix for the broken ROCm CSB. #34104

Merged

Conversation

deven-amd
Copy link
Contributor

The following commit breaks the --config=rocm build

f72695e

The above commit adds a couple of subtests that require support for the StatefulUnirformFullInt Op on the GPU. Currently ROCm does not support that Op on the GPU, which leads to those subtests failing.

The "fix" is to skip those subtests on the ROCm platform.


/cc @chsigg @whchung

The following commit breaks the --config=rocm build

tensorflow@f72695e

The above commit adds a couple of subtests that require support for the `StatefulUnirformFullInt` Op on the GPU. Currently ROCm does not support that Op on the GPU, which leads to those subtests failing.

The "fix" is to skip those subtests on the ROCm platform.
@tensorflow-bot tensorflow-bot bot added the size:S CL Change Size: Small label Nov 8, 2019
@whchung whchung added the kokoro:force-run Tests on submitted change label Nov 8, 2019
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Nov 8, 2019
@gbaned gbaned self-assigned this Nov 11, 2019
@gbaned gbaned added the comp:keras Keras related issues label Nov 11, 2019
@gbaned gbaned added this to Assigned Reviewer in PR Queue via automation Nov 11, 2019
@gbaned gbaned added the awaiting review Pull request awaiting review label Nov 12, 2019
@deven-amd
Copy link
Contributor Author

@chsigg @tanzhenyu gentle ping

Copy link
Contributor

@tanzhenyu tanzhenyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why is this needed -- where is StatefulUniformFullInt used? I think the layer was only using stateless random ops.

@deven-amd
Copy link
Contributor Author

The error message we get for the failure indicates that StatefulUniformFullInt is getting instantiated in the TF graph.

tensorflow.python.framework.errors_impl.NotFoundError: No registered 'StatefulUniformFullInt' OpKernel for 'GPU' devices compatible with node {{node StatefulUniformFullInt}}                                                                                                                                                                                                                                                        
        .  Registered:  device='XLA_GPU'; shape_dtype in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, ..., DT_UINT16, DT_COMPLEX128, DT_HALF, DT_UINT32, DT_UINT64]; dtype in [DT_INT32, DT_INT64, DT_UINT32, DT_UINT64]                                                                                                                                                                                                      
  device='XLA_CPU'; shape_dtype in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, ..., DT_UINT16, DT_COMPLEX128, DT_HALF, DT_UINT32, DT_UINT64]; dtype in [DT_INT32, DT_INT64, DT_UINT32, DT_UINT64]                                                                                                                                                                                                                            
  device='XLA_CPU_JIT'; shape_dtype in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, ..., DT_UINT16, DT_COMPLEX128, DT_HALF, DT_UINT32, DT_UINT64]; dtype in [DT_INT32, DT_INT64, DT_UINT32, DT_UINT64]                                                                                                                                                                                                                        
  device='CPU'; dtype in [DT_UINT64]                                                                                                                                                                                                                                                                                                                                                                                                 
  device='CPU'; dtype in [DT_UINT32]                                                                                                                                                                                                                                                                                                                                                                                                 
  device='CPU'; dtype in [DT_INT64]                                                                                                                                                                                                                                                                                                                                                                                                  
  device='CPU'; dtype in [DT_INT32]                                                                                                                                                                                                                                                                                                                                                                                                  
  device='XLA_GPU_JIT'; shape_dtype in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, ..., DT_UINT16, DT_COMPLEX128, DT_HALF, DT_UINT32, DT_UINT64]; dtype in [DT_INT32, DT_INT64, DT_UINT32, DT_UINT64]                                                                                                                                                                                                                        
 [Op:StatefulUniformFullInt] name: model_1/random_crop/stateful_uniform_full_int/                                                                                                                                                                                                                                                                                                                                                    

@tensorflowbutler tensorflowbutler removed the awaiting review Pull request awaiting review label Nov 13, 2019
@deven-amd
Copy link
Contributor Author

@chsigg @tanzhenyu gentle ping

@gbaned gbaned added the awaiting review Pull request awaiting review label Nov 21, 2019
@deven-amd
Copy link
Contributor Author

@gbaned , anything we can do to help get this PR merged?

PR Queue automation moved this from Assigned Reviewer to Approved by Reviewer Nov 28, 2019
@tensorflow-bot tensorflow-bot bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Nov 28, 2019
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Nov 28, 2019
@gbaned gbaned removed the awaiting review Pull request awaiting review label Nov 28, 2019
tensorflow-copybara pushed a commit that referenced this pull request Nov 28, 2019
…ocm_fix_191108

PiperOrigin-RevId: 282922989
Change-Id: Id75e811dc0668a448800b712fe86975ac76ae991
@tensorflow-copybara tensorflow-copybara merged commit b1787be into tensorflow:master Nov 28, 2019
PR Queue automation moved this from Approved by Reviewer to Merged Nov 28, 2019
@deven-amd deven-amd deleted the google_upstream_rocm_fix_191108 branch December 20, 2019 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes comp:keras Keras related issues ready to pull PR ready for merge process size:S CL Change Size: Small
Projects
PR Queue
  
Merged
Development

Successfully merging this pull request may close these issues.

None yet

9 participants