Autograph "Failed to parse source code" error when using lambda in for loop #30149
Comments
ravikyram
added
comp:autograph
|
I have reproduced the issue in Colab using TF VERSION=2.0.0-dev20190625.Thanks! |
|
This is related to a limitation in Python's The workaround is to declare the lambda function on a single line, as the OP indicates. Normally, the error message should describe that (albeit in more detail), but it should definitely suggest the workaround of declaring the lambda on a separate line - @ageron can you confirm that the error message included that guidance? Related, we should remove the extraneous imports from the error message. The message should spell just: |
|
Yes, I can confirm that the message ValueError: Failed to parse source code of <function <lambda> at 0x134b81488>, which Python reported as:
for window in ds.flat_map(lambda window: window.batch(5)):
The error may be avoided by creating the lambda in a standalone statement.Alternatively, isn't it possible to parse this line to extract the lambda? After all, it's right there. :) |
|
I agree - will simplify the error message. Yes, we do attempt to parse the line, but in this case is it not well-formed Python code - in our example, it's a for loop without a body. One could imagine a partial parser which attempts to parse as much as possible of the code that is well-formed, but the Python parser doesn't know how to do that, and even then there may still be situations of ambiguity where the results would be incorrect. A much more robust fix would be to fix the parser so that it records the exact extents of the lambda, with column numbers. Currently it only records the line number, which is the root of the problem. |
Currently, for some function, TF2 autograph will fail. See tensorflow/tensorflow#35765 tensorflow/tensorflow#30149 tensorflow/autograph#3 If autograph failed, the functions will be run eagerly and TF cannot optimize them. So we'd better address them. ## Manually test ~~~python df1 = spark.range(100) from petastorm.spark import make_spark_converter # Set a cache directory on DBFS FUSE for intermediate data. spark.conf.set("petastorm.spark.converter.parentCacheDirUrl", "file:///dbfs/ml/tmp/petastorm/QA/bugs/") converter1 = make_spark_converter(df1) with converter1.make_tf_dataset(num_epochs=1) as dataset: for batch in dataset: print(batch.id) ~~~ * Before Output includes: ``` WARNING:tensorflow:AutoGraph could not transform <function _NamedtupleCache.get at 0x7f0bfbe6f200> and will run it as-is. Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: expected exactly one node node, found [<gast.gast.FunctionDef object at 0x7f0bfad5d050>, <gast.gast.Return object at 0x7f0bfad5d7d0>] WARNING:tensorflow:AutoGraph could not transform <function make_petastorm_dataset.<locals>.<lambda> at 0x7f0bf8da03b0> and will run it as-is. Cause: could not parse the source code: .map(lambda row: _set_shape_to_named_tuple(reader.schema, row, reader.batched_output)) This error may be avoided by creating the lambda in a standalone statement. ``` * After The warnings listed above disappear.
Currently, for some function, TF2 autograph will fail. See tensorflow/tensorflow#35765 tensorflow/tensorflow#30149 tensorflow/autograph#3 If autograph failed, the functions will be run eagerly and TF cannot optimize them. So we'd better address them. ## Manually test ~~~python df1 = spark.range(100) from petastorm.spark import make_spark_converter # Set a cache directory on DBFS FUSE for intermediate data. spark.conf.set("petastorm.spark.converter.parentCacheDirUrl", "file:///dbfs/ml/tmp/petastorm/QA/bugs/") converter1 = make_spark_converter(df1) with converter1.make_tf_dataset(num_epochs=1) as dataset: for batch in dataset: print(batch.id) ~~~ * Before Output includes: ``` WARNING:tensorflow:AutoGraph could not transform <function _NamedtupleCache.get at 0x7f0bfbe6f200> and will run it as-is. Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: expected exactly one node node, found [<gast.gast.FunctionDef object at 0x7f0bfad5d050>, <gast.gast.Return object at 0x7f0bfad5d7d0>] WARNING:tensorflow:AutoGraph could not transform <function make_petastorm_dataset.<locals>.<lambda> at 0x7f0bf8da03b0> and will run it as-is. Cause: could not parse the source code: .map(lambda row: _set_shape_to_named_tuple(reader.schema, row, reader.batched_output)) This error may be avoided by creating the lambda in a standalone statement. ``` * After The warnings listed above disappear.
System information
Yes
MacOSX 10.13.6
N/A
binary
VERSION=2.0.0-dev20190625
GIT_VERSION=v1.12.1-4885-g71241a6afd
3.6.8
N/A
N/A
N/A
N/A
Describe the current behavior
I get an autograph error when running the following code (see the full stacktrace below):
The error is
ValueError: Failed to parse source code of <function <lambda> at 0x11194c488>Describe the expected behavior
Everything works fine when I define the dataset on the previous line like this:
Code to reproduce the issue
See above.
Other info / logs
Full stack trace with
AUTOGRAPH_VERBOSITY=10:The text was updated successfully, but these errors were encountered: