tf_upgrade_v2 fails if the file contains f-strings and gives pasta.base.annotate.AnnotationError

[MaximilianPavon]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS 10.14.3
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: NA
• TensorFlow installed from (source or binary): pip install tensorflow==2.0.0-alpha0
• TensorFlow version (use command below): 2.0.0-alpha0
• Python version: 3.6.8
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version: NA
• GPU model and memory: NA

Describe the current behavior
tf_upgrade_v2 fails if the file contains f-strings

Describe the expected behavior
tf_upgrade_v2 does not fails if the file contains f-strings

Code to reproduce the issue
File foo.py:

print(f'tf_upgrade_v2 fails to convert f-strings, like this one: {42}')

Command that produces the error: tf_upgrade_v2 --infile foo.py --outfile foo_tf20.py

Other info / logs

Traceback (most recent call last):
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 1161, in visit
    super(AstAnnotator, self).visit(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 127, in visit
    super(BaseVisitor, self).visit(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/ast.py", line 253, in visit
    return visitor(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 47, in wrapped
    f(self, node, *args, **kwargs)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 1213, in visit_Num
    self.attr(node, 'content', contentargs, deps=('n',), default=str(node.n))
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 1352, in attr
    attr_parts.append(attr_val())
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 1210, in <lambda>
    contentargs = [lambda: self.tokens.next_of_type(token_number_type).src]
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/token_generator.py", line 347, in next_of_type
    self.lines[token.start[0] - 1]))
ValueError: Expected 'NUMBER' but found ')'
line 1: print(f'tf_upgrade_v2 fails to convert f-strings, like this one: {42}')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda3/envs/tf2.0/bin/tf_upgrade_v2", line 10, in <module>
    sys.exit(main())
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/tensorflow/tools/compatibility/tf_upgrade_v2_main.py", line 110, in main
    args.input_file, output_file, upgrade)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/tensorflow/tools/compatibility/tf_upgrade_v2_main.py", line 33, in process_file
    upgrader.process_file(in_filename, out_filename)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/tensorflow/tools/compatibility/ast_edits.py", line 494, in process_file
    temp_file)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/tensorflow/tools/compatibility/ast_edits.py", line 548, in process_opened_file
    self.update_string_pasta("".join(lines), in_filename))
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/tensorflow/tools/compatibility/ast_edits.py", line 510, in update_string_pasta
    t = pasta.parse(text)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/__init__.py", line 25, in parse
    annotator.visit(t)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 1161, in visit
    super(AstAnnotator, self).visit(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 127, in visit
    super(BaseVisitor, self).visit(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/ast.py", line 253, in visit
    return visitor(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 47, in wrapped
    f(self, node, *args, **kwargs)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 211, in visit_Module
    self.generic_visit(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/ast.py", line 261, in generic_visit
    self.visit(item)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 1161, in visit
    super(AstAnnotator, self).visit(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 127, in visit
    super(BaseVisitor, self).visit(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/ast.py", line 253, in visit
    return visitor(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 47, in wrapped
    f(self, node, *args, **kwargs)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 574, in visit_Expr
    self.visit(node.value)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 1161, in visit
    super(AstAnnotator, self).visit(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 127, in visit
    super(BaseVisitor, self).visit(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/ast.py", line 253, in visit
    return visitor(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 47, in wrapped
    f(self, node, *args, **kwargs)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 703, in visit_Call
    any_args = self.visit_Call_arguments35(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 770, in visit_Call_arguments35
    self.visit(arg)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 1161, in visit
    super(AstAnnotator, self).visit(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 127, in visit
    super(BaseVisitor, self).visit(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/ast.py", line 253, in visit
    return visitor(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/ast.py", line 261, in generic_visit
    self.visit(item)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 1161, in visit
    super(AstAnnotator, self).visit(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 127, in visit
    super(BaseVisitor, self).visit(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/ast.py", line 253, in visit
    return visitor(node)
  File "/anaconda3/envs/tf2.0/lib/python3.6/ast.py", line 263, in generic_visit
    self.visit(value)
  File "/anaconda3/envs/tf2.0/lib/python3.6/site-packages/pasta/base/annotate.py", line 1163, in visit
    raise AnnotationError(e)
pasta.base.annotate.AnnotationError: Expected 'NUMBER' but found ')'
line 1: print(f'tf_upgrade_v2 fails to convert f-strings, like this one: {42}')

[ziky90]

I'm having exactly the same problem when trying to migrate to tensorflow==2.0.0-alpha0 using the tf_upgrade_v2 script.

[ghost]

I've encountered the same bug. A test file:

#! /usr/bin/env /usr/bin/python3

import tensorflow as tf

tf.enable_eager_execution()
print(f"tf.executing_eagerly() = {tf.executing_eagerly()}")

I've attached the Traceback.
tf2_fstring_traceback.txt

[alextp]

This is an issue in google_pasta, a library we use to write that script in. I've escalated it to the maintainers and will ping back here soon.

[soupytwist]

Sorry for the trouble here, this will be addressed as part of google/pasta#58.

[soupytwist]

The fstring failures should be fixed as of pasta 0.1.5 (pasta#59 + pasta#61).

[lc0]

same here, waiting for this to upgraded, so I can upgrade it as well for http://tf2up.ml/

[kriskorrel-cw]

@soupytwist Should this problem be fixed now? I am using google_pasta 0.1.6 now and am still experiencing this issue.

[lc0]

@kriskorrel-cw yes, it's fixed in nightly builds for quite some time

[lc0]

I think we can also close the issue

[kriskorrel-cw]

I have the following (minimal) code example that tf_upgrade_v2 can't pass.

import os
import Tensorflow
path = "/a/b/c/d/e"
c = f'/{(os.sep).join(path.split(os.sep)[2:])}'

I believe I have the correct pasta version:

$ pip3 install google_pasta

Collecting google_pasta
  Using cached https://files.pythonhosted.org/packages/f9/68/a14620bfb042691f532dcde8576ff82ee82e4c003cdc0a3dbee5f289cee6/google_pasta-0.1.6-py3-none-any.whl
Installing collected packages: google-pasta
Successfully installed google-pasta-0.1.6

Do you also experience this error @lc0 ? In that case, I don't think it can be closed

[lc0]

@kriskorrel-cw I think the truth is somewhere in the middle. It does work for simpler cases like

foo = 'bar'
a, b = 1, 2
print(f"{foo} : {a+b}")

But does fail for your case. For more examples please see the colab example below

https://colab.research.google.com/drive/1_hWyz2x_SwN0R0Jo1G0OCBeybmPkLlzC#scrollTo=a9l0KRPYu0JK

I think in this case would make sense to reference this case for pasta

[soupytwist]

Thank you for the test case to reproduce this. Revisiting this now.

[soupytwist]

It should be fixed in 0.1.7.

[lc0]

I just tested with todays nightly and the issue is resolved - I can convert the file that failed before. Thanks @soupytwist!

I think if @MaximilianProll does not have any objections we can close this issue

cc @alextp

[tensorflow-bot]

Are you satisfied with the resolution of your issue?
Yes
No

[richardwth]

I received the pasta.base.annotate.AnnotationError but I used str.format() instead of f-string. What should I do?

The detailed error is:

pasta.base.annotate.AnnotationError: Expected '_flatten' but found 'yield'
line 119:                 yield from _flatten(tensor)

The current code was written in TF-1.14. google-pasta version is 0.1.8.

update: I fixed this by removing all code that includes yield...

[soupytwist]

Hi @richardwth,
Thanks for reporting that. The error you posted isn't what I would expect, but it did reveal that the yield from <expression> syntax is unsupported in pasta. Fixed this in 43cc7c6.

[JDE65]

I have tried to upgrade to tf2 :
tf_upgrade_v2 --intree tf2/ --outtree tf2/upgraded/ --reportfile report.txt
where tf2 is the directory with all my codes ready for upgrade and tf2/upgraded/ the subdirectory where I'd like to get the update codes.
I also tried on single .py files with
tf_upgrade_v2 --intree tf2/util_model.py --outtree tf2/upgraded/util_model.py --reportfile report_utile_model.txt

I am working with with Windows 10 - anaconda - spyder 4.1.5 - python 3.7 and
google-pasta 0.2.0 then 0.1.5 then 0.1.7 then 0.1.8.... I always get the same message

  File "D:\Anaconda3\Scripts\tf_upgrade_v2-script.py", line 6, in <module>
    from tensorflow.tools.compatibility.tf_upgrade_v2_main import main
  File "D:\Anaconda3\lib\site-packages\tensorflow\tools\compatibility\tf_upgrade_v2_main.py", line 24, in <module>
    from tensorflow.tools.compatibility import ast_edits
  File "D:\Anaconda3\lib\site-packages\tensorflow\tools\compatibility\ast_edits.py", line 30, in <module>
    import pasta
ModuleNotFoundError: No module named 'pasta'

I don't know what to do anymore.