Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix errors of incorrect session for evaluation in the clusterspec propagation test. #27108

Closed

Conversation

shishaochen
Copy link
Contributor

For a TensorFlowTestCase instance, its self.evalute(...) uses the default session to fetch a tensor value.
In testClusterSpecPropagationWorker1Placement and testMultipleLocalDevices, the default session has no graph registered, which brings errors like the following:

======================================================================
ERROR: testClusterSpecPropagationWorker1Placement (__main__.SessionClusterSpecPropagationTest)
testClusterSpecPropagationWorker1Placement (__main__.SessionClusterSpecPropagationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data00/home/shishaochen/.cache/bazel/_bazel_shishaochen/66654af5b8967893709f1bb873fdbb49/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/session
    yield
  File "/data00/home/shishaochen/.cache/bazel/_bazel_shishaochen/66654af5b8967893709f1bb873fdbb49/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/session
    testMethod()
  File "/data00/home/shishaochen/.cache/bazel/_bazel_shishaochen/66654af5b8967893709f1bb873fdbb49/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/session
    output = self.evaluate(const)
  File "/data00/home/shishaochen/.cache/bazel/_bazel_shishaochen/66654af5b8967893709f1bb873fdbb49/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/session
    return sess.run(tensors)
  File "/data00/home/shishaochen/.cache/bazel/_bazel_shishaochen/66654af5b8967893709f1bb873fdbb49/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/session
    return super(ErrorLoggingSession, self).run(*args, **kwargs)
  File "/data00/home/shishaochen/.cache/bazel/_bazel_shishaochen/66654af5b8967893709f1bb873fdbb49/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/session
    run_metadata_ptr)
  File "/data00/home/shishaochen/.cache/bazel/_bazel_shishaochen/66654af5b8967893709f1bb873fdbb49/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/session
    raise RuntimeError('The Session graph is empty.  Add operations to the '
RuntimeError: The Session graph is empty.  Add operations to the graph before calling run().

======================================================================
ERROR: testMultipleLocalDevices (__main__.SessionClusterSpecPropagationTest)
testMultipleLocalDevices (__main__.SessionClusterSpecPropagationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data00/home/shishaochen/.cache/bazel/_bazel_shishaochen/66654af5b8967893709f1bb873fdbb49/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/session
    yield
  File "/data00/home/shishaochen/.cache/bazel/_bazel_shishaochen/66654af5b8967893709f1bb873fdbb49/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/session
    testMethod()
  File "/data00/home/shishaochen/.cache/bazel/_bazel_shishaochen/66654af5b8967893709f1bb873fdbb49/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/session
    output = self.evaluate(sum3)
  File "/data00/home/shishaochen/.cache/bazel/_bazel_shishaochen/66654af5b8967893709f1bb873fdbb49/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/session
    return sess.run(tensors)
  File "/data00/home/shishaochen/.cache/bazel/_bazel_shishaochen/66654af5b8967893709f1bb873fdbb49/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/session
    return super(ErrorLoggingSession, self).run(*args, **kwargs)
  File "/data00/home/shishaochen/.cache/bazel/_bazel_shishaochen/66654af5b8967893709f1bb873fdbb49/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/session
    run_metadata_ptr)
  File "/data00/home/shishaochen/.cache/bazel/_bazel_shishaochen/66654af5b8967893709f1bb873fdbb49/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/python/session
    raise RuntimeError('The Session graph is empty.  Add operations to the '
RuntimeError: The Session graph is empty.  Add operations to the graph before calling run().

----------------------------------------------------------------------

Thus, we'd better explicitly use the created GRPC session registered with the wanted graph to evaluate tensors.

@rthadur rthadur added this to Assigned Reviewer in PR Queue via automation Mar 25, 2019
@rthadur rthadur self-assigned this Mar 25, 2019
@rthadur rthadur added the size:S CL Change Size: Small label Mar 25, 2019
@rthadur rthadur requested a review from saeta March 25, 2019 14:05
@shishaochen shishaochen changed the title Fix errors of incorrect session for evaluation. Fix errors of incorrect session for evaluation in the clusterspec propagation test. Mar 25, 2019
@shishaochen
Copy link
Contributor Author

Close this as #24466 is merged.

PR Queue automation moved this from Assigned Reviewer to Closed/Rejected Mar 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes size:S CL Change Size: Small
Projects
PR Queue
  
Closed/Rejected
Development

Successfully merging this pull request may close these issues.

None yet

4 participants