Can import meta graph KeyError: 'InfeedEnqueueTuple'

[sethtroisi]

    import tensorflow as tf
    tf.train.import_meta_graph("/home/eights/Projects/minigo/saved_models/000567-san-josef.meta")

I tensorflow (installed from pip3) and tf-nightly

full error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1666, in import_meta_graph
    meta_graph_or_file, clear_devices, import_scope, **kwargs)[0]
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py", line 1688, in _import_meta_graph_with_return_elements
    **kwargs))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/meta_graph.py", line 806, in import_scoped_meta_graph_with_return_elements
    return_elements=return_elements)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/importer.py", line 391, in import_graph_def
    _RemoveDefaultAttrs(op_dict, producer_op_list, graph_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/importer.py", line 158, in _RemoveDefaultAttrs
    op_def = op_dict[node.op]
KeyError: 'InfeedEnqueueTuple'

[amj]

this needs more details or i vote we close it

[wpstmxhs]

For me, the same problem has occurred:

Traceback (most recent call last):
  File "convert.py", line 11, in <module>
    saver = tf.train.import_meta_graph(sys.argv[1]+'.meta')
  File "C:\Users\Owner\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1674, in import_meta_graph
    meta_graph_or_file, clear_devices, import_scope, **kwargs)[0]
  File "C:\Users\Owner\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1696, in _import_meta_graph_with_return_elements
    **kwargs))
  File "C:\Users\Owner\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\meta_graph.py", line 806, in import_scoped_meta_graph_with_return_elements
    return_elements=return_elements)
  File "C:\Users\Owner\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "C:\Users\Owner\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\importer.py", line 391, in import_graph_def
    _RemoveDefaultAttrs(op_dict, producer_op_list, graph_def)
  File "C:\Users\Owner\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\importer.py", line 158, in _RemoveDefaultAttrs
    op_def = op_dict[node.op]
KeyError: 'InfeedEnqueueTuple'

I think this is due to difference between GPU and TPU.

[sethtroisi]

Things I've tried without success:

• clear_devices=True
• with tf.Session(config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)) as sess:

This seems to work in colab

!mkdir models
!gsutil cp gs://minigo-pub/v13-19x19/models/000673-crown* models
tf.train.import_meta_graph('models/000673-crown.meta')
``

https://colab.sandbox.google.com/notebooks/tpu.ipynb#scrollTo=ZKNff9YVVgWh

[amj]

can i get some more details about what we're trying to do and why we expect it to work?
I'm sorry the original bug here is that these two lines of python, devoid of context, don't work. Are they supposed to? why do we think tf.train.import_meta_graph with no preamble is just going to work?

I'm still not sure what the bug is. I'd like to see something about the problem we're trying to solve here.

[sethtroisi]

I didn't realize there was confusion.

Problem: you can't import a the .meta file. It certainly expected that import_meta_graph should work (e.g. there's no callout in the api documentation that it doesn't work for large models or anything).

Consequence: slightly harder to do model manipulation (for things like oneoff/l2_cost, and lz-to-mg converter)

I'm 98% sure this has to do with TPUs because the InfeedEnqueueTuple op is only(?) used on TPU specific and probably not loaded/supported if TPUs aren't found.

I see this affects other people. [1] but everyone seems to just suggest hacking around the problem (export as .pb, load tensors manually, ...)

[sethtroisi]

From a different thread the proposed solution is to "use one of the export functions, e.g. (TPU)Estimator.export_savedmodel() for inference"

we might try this at a later date but the main blocker (leela-zero/leela-zero#2133) has been fixed a different way