Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coverage not working for TensorFlow Model call function #856

Open
NathanDotTo opened this issue Oct 11, 2019 · 9 comments
Labels
bug

Comments

@NathanDotTo
Copy link

@NathanDotTo NathanDotTo commented Oct 11, 2019

This problem was first reported at https://youtrack.jetbrains.com/oauth?state=%2Fissue%2FPY-38366

1 - I am running a unit test that is invoking the call(...) function of a TensorFlow model. I can see the trace message, that I have placed in the function, printed so I know the function is being called.
2 - The coverage analysis shows that code in red, as though it was not invoked.
3 - It should show green as it was invoked.

See test_model.py in https://github.com/Data-Science-Projects/demo-routenet.

See also the requirements.txt in that project.

@NathanDotTo NathanDotTo added the bug label Oct 11, 2019
@nedbat

This comment has been minimized.

Copy link
Owner

@nedbat nedbat commented Oct 14, 2019

I've never used TensorFlow before. I'm going to guess that the execution of Python code within call(...) doesn't trigger the Python trace function, but I don't know.

Can you provide very very explicit reproduction steps? Include the version of Python, the commands to run to install everything, and the commands to run the program. Thanks.

@NathanDotTo

This comment has been minimized.

Copy link
Author

@NathanDotTo NathanDotTo commented Oct 26, 2019

Python 3.7.4 on macOS 10.14.6.

The steps so far are:

git clone https://github.com/Data-Science-Projects/demo-routenet.git
cd demo-routenet/bin
. ./create_routenet_venv.sh
cd ../tests/unit
pytest -s test_model.py --cov

Since I am looking the results in PyCharm, I am not quite sure how to recreate what I see in PyCharm at the CLI. What seems to be missing, in PyCharm, is any indication that the call function in the src/routenet/model/routenet_model.py code is covered. It is clear that the code is called, as one can see the output from the print statement.

@nedbat

This comment has been minimized.

Copy link
Owner

@nedbat nedbat commented Oct 26, 2019

Thanks, this lets me reproduce the problem. I can see that line 65 in routenet_model.py is reported as uncovered. I'll dig into it.

@NathanDotTo

This comment has been minimized.

Copy link
Author

@NathanDotTo NathanDotTo commented Oct 26, 2019

@nedbat

This comment has been minimized.

Copy link
Owner

@nedbat nedbat commented Oct 27, 2019

@NathanDotTo this was an interesting one! It turns out that TensorFlow is making a transformed copy of your code, and then running it. So your original file isn't actually executed. In a way, coverage.py was right! I've opened tensorflow/tensorflow#33759 with TensorFlow to see what we can do about it.

@NathanDotTo

This comment has been minimized.

Copy link
Author

@NathanDotTo NathanDotTo commented Oct 27, 2019

@Zminghua

This comment has been minimized.

Copy link

@Zminghua Zminghua commented Oct 31, 2019

I have a similar question, when I debug tensor2tensor and set a breakpoint in t2t_model.py at line 316, the debugger doesn't break as if there is no breakpoint. Then I run the code step by step, in the end, I also found a temporary Python file in /tmp/.

  • tensorflow 1.15
  • python 3.7
@nedbat

This comment has been minimized.

Copy link
Owner

@nedbat nedbat commented Oct 31, 2019

@Zminghua debugging is not a coverage.py concern, but sharing my new-found expertise, it should work if you put a "pdb.set_trace()" line in your t2t_model.py file. That line will then be copied to the temporary file, and the debugger will break. You will be in a slightly strange world, since your file has been changed, but you can at least be in the debugger.

@Zminghua

This comment has been minimized.

Copy link

@Zminghua Zminghua commented Nov 1, 2019

@nedbat Thanks for your reminding. I have learned that it is AutoGraph conversion in Tensorflow keras module. After I added a "pdb.set_trace()" line, I was really in a slightly strange world. However, I have modified tensorflow source code to prevent the conversion. Thank you very very very much.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.