Support nested collections of TensorFlow tensors with t flag

[danijar]

Minimal example:

from qj import qj
import tensorflow as tf
tensors = {'foo': tf.constant(0), 'bar': tf.constant(1)}
tensors = qj(tensors, t=True)
with tf.Session() as sess:
  sess.run(tensors)

Output:

IndexError                                Traceback (most recent call last)
<ipython-input-1-6067c4d3b276> in <module>()
      2 import tensorflow as tf
      3 tensors = {'foo': tf.constant(0), 'bar': tf.constant(1)}
----> 4 tensors = qj(tensors, t=True)
      5 with tf.Session() as sess:
      6   sess.run(tensors)

~/.local/lib/python3.5/site-packages/qj/qj.py in qj(x, s, l, d, p, t, n, r, z, b, pad, tfc, tic, toc, time, catch, log_all_calls, _depth)
    192           if f.f_lasti not in fn_calls:
    193             qj._DEBUG_QJ and qj._DISASSEMBLE_FN(f.f_code, f.f_lasti)
--> 194             fn_calls[f.f_lasti] = _find_current_fn_call(f.f_code, f.f_lasti)
    195             qj._FN_MAPS[f.f_code.co_code] = fn_calls
    196           s = fn_calls.setdefault(f.f_lasti, '').strip()

~/.local/lib/python3.5/site-packages/qj/qj.py in _find_current_fn_call(co, lasti)
   1698 
   1699   # Apply stack effects backwards until we arrive at the stack entries for a complete function call
-> 1700   fn_stack = _collect_pops(stack[:-1], stack[-1].stack_depth, [], 0)
   1701 
   1702   if not fn_stack and stack[-1].stack_depth == 0:

~/.local/lib/python3.5/site-packages/qj/qj.py in _collect_pops(stack, depth, pops, skip)
   1511     skip = set_skip_after_current_entry + max(0, skip)
   1512 
-> 1513   pops = _collect_pops(stack, depth + popped_depth, pops, skip)
   1514 
   1515   if len(tokens):  # pylint: disable=g-explicit-length-test

~/.local/lib/python3.5/site-packages/qj/qj.py in _collect_pops(stack, depth, pops, skip)
   1511     skip = set_skip_after_current_entry + max(0, skip)
   1512 
-> 1513   pops = _collect_pops(stack, depth + popped_depth, pops, skip)
   1514 
   1515   if len(tokens):  # pylint: disable=g-explicit-length-test

~/.local/lib/python3.5/site-packages/qj/qj.py in _collect_pops(stack, depth, pops, skip)
   1511     skip = set_skip_after_current_entry + max(0, skip)
   1512 
-> 1513   pops = _collect_pops(stack, depth + popped_depth, pops, skip)
   1514 
   1515   if len(tokens):  # pylint: disable=g-explicit-length-test

~/.local/lib/python3.5/site-packages/qj/qj.py in _collect_pops(stack, depth, pops, skip)
   1511     skip = set_skip_after_current_entry + max(0, skip)
   1512 
-> 1513   pops = _collect_pops(stack, depth + popped_depth, pops, skip)
   1514 
   1515   if len(tokens):  # pylint: disable=g-explicit-length-test

~/.local/lib/python3.5/site-packages/qj/qj.py in _collect_pops(stack, depth, pops, skip)
   1369   expect_extracted_tokens = []
   1370 
-> 1371   se = stack.pop()
   1372   pops_len = len(pops)
   1373   if (pops_len > 1

IndexError: pop from empty list

[iansf]

Thanks for the report and minimal repro! This is dying in the code decompilation code, which happens before anything having to do with t=True. I suspect it's getting confused by the code above the qj call. The minimal work-around is to pass the second argument to qj, which is the label string: qj(tensors, 'tensors', t=1) However, having done that, it still won't do what you want (but I claim it won't crash, and it will log something reasonable), as t=1 currently only works if the object is directly a tf object type. I agree that supporting collections of tensors would be nice. I'll take a look at both of these issues soon.

…

On Wed, Jun 13, 2018 at 3:45 AM Danijar Hafner ***@***.***> wrote: Minimal example: from qj import qjimport tensorflow as tf tensors = {'foo': tf.constant(0), 'bar': tf.constant(1)} tensors = qj(tensors, t=True)with tf.Session() as sess: sess.run(tensors) Output: IndexError Traceback (most recent call last) <ipython-input-1-6067c4d3b276> in <module>() 2 import tensorflow as tf 3 tensors = {'foo': tf.constant(0), 'bar': tf.constant(1)} ----> 4 tensors = qj(tensors, t=True) 5 with tf.Session() as sess: 6 sess.run(tensors) ~/.local/lib/python3.5/site-packages/qj/qj.py in qj(x, s, l, d, p, t, n, r, z, b, pad, tfc, tic, toc, time, catch, log_all_calls, _depth) 192 if f.f_lasti not in fn_calls: 193 qj._DEBUG_QJ and qj._DISASSEMBLE_FN(f.f_code, f.f_lasti) --> 194 fn_calls[f.f_lasti] = _find_current_fn_call(f.f_code, f.f_lasti) 195 qj._FN_MAPS[f.f_code.co_code] = fn_calls 196 s = fn_calls.setdefault(f.f_lasti, '').strip() ~/.local/lib/python3.5/site-packages/qj/qj.py in _find_current_fn_call(co, lasti) 1698 1699 # Apply stack effects backwards until we arrive at the stack entries for a complete function call -> 1700 fn_stack = _collect_pops(stack[:-1], stack[-1].stack_depth, [], 0) 1701 1702 if not fn_stack and stack[-1].stack_depth == 0: ~/.local/lib/python3.5/site-packages/qj/qj.py in _collect_pops(stack, depth, pops, skip) 1511 skip = set_skip_after_current_entry + max(0, skip) 1512 -> 1513 pops = _collect_pops(stack, depth + popped_depth, pops, skip) 1514 1515 if len(tokens): # pylint: disable=g-explicit-length-test ~/.local/lib/python3.5/site-packages/qj/qj.py in _collect_pops(stack, depth, pops, skip) 1511 skip = set_skip_after_current_entry + max(0, skip) 1512 -> 1513 pops = _collect_pops(stack, depth + popped_depth, pops, skip) 1514 1515 if len(tokens): # pylint: disable=g-explicit-length-test ~/.local/lib/python3.5/site-packages/qj/qj.py in _collect_pops(stack, depth, pops, skip) 1511 skip = set_skip_after_current_entry + max(0, skip) 1512 -> 1513 pops = _collect_pops(stack, depth + popped_depth, pops, skip) 1514 1515 if len(tokens): # pylint: disable=g-explicit-length-test ~/.local/lib/python3.5/site-packages/qj/qj.py in _collect_pops(stack, depth, pops, skip) 1511 skip = set_skip_after_current_entry + max(0, skip) 1512 -> 1513 pops = _collect_pops(stack, depth + popped_depth, pops, skip) 1514 1515 if len(tokens): # pylint: disable=g-explicit-length-test ~/.local/lib/python3.5/site-packages/qj/qj.py in _collect_pops(stack, depth, pops, skip) 1369 expect_extracted_tokens = [] 1370 -> 1371 se = stack.pop() 1372 pops_len = len(pops) 1373 if (pops_len > 1 IndexError: pop from empty list — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMC8E6UiHimOz6qaGouW9MYg88DVsa-6ks5t8O1VgaJpZM4Ul9fj> .

[danijar]

Thanks, sounds great!