-
Notifications
You must be signed in to change notification settings - Fork 284
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Travis CI Enhancement #94
Conversation
129ace2
to
5c4d997
Compare
This is a Travis CI Rework. The idea is to split C++ build with Python and R build. Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
a38656a
to
80d1191
Compare
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
68aa6b4
to
5fbcbd0
Compare
So that we could use `(cd tests && python -m pytest -s .)` Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
to run `python -m pytest -s tests` without install. Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
decc60a
to
ff81ef8
Compare
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
@dmitrievanthony @terrytangyuan @BryanCutler This PR should be ready now. The testing is closer to Also dropped shell script for whl generation. All are done through |
Once this PR is merged, I will update the 1.13 support PR. 1.13 support is almost ready as well. We will need 1.13 support soon to match the TF release, which likely will happen in a week or two. |
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
echo "options(repos = c(CRAN='http://cran.rstudio.com'))" >> ~/.Rprofile | ||
R -e 'install.packages(c("tensorflow", "tfdatasets"), quiet = TRUE)' | ||
R -e 'install.packages(c("testthat", "devtools"), quiet = TRUE)' | ||
R -e 'install.packages(c("forge"), quiet = TRUE)' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason for splitting them into three different commands?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@terrytangyuan Travis CI has a limit of several minutes to see any output, otherwise it will assume the job hangs and cancel. Travis CI also has a limit of max length of logs. Previously we install all packages in one command without quiet
. That caused the max length of logs reached. So changed to one command installing every package with quiet = TRUE
. But that caused several minutes without any output to stderr or stdout. So I split them into several, just to show something in the output (by command itself echoed in shell, not the install.packages).
.travis/install.test.sh
Outdated
R -e 'install.packages(c("testthat", "devtools"), quiet = TRUE)' | ||
R -e 'install.packages(c("forge"), quiet = TRUE)' | ||
|
||
V=$(python3 -c 'import sys; print(str(sys.version_info[0])+str(sys.version_info[1]))') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe a more meaningful variable name here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@terrytangyuan Update to use CPYTHON_VERSION
to make it easy to read.
|
||
# NOTE: __datapath__ is a hidden value used by test, | ||
# in case .so files are on a different path. | ||
__datapath__ = None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why would .so
files be on different paths?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is to allow us to test two scenarios:
- Install package from whl files. In this situation,
.so
file will be part of the path so__datapath__
will always be None, no change needs. - Not install the package (in development). In this situation, as we split the C++ build to use bazel and pytest for python, the
.so
files are generated inbazel-bin
and the python code are in the original source tree. However, bazel does not allow us to write.so
file into the original source tree. So__datapath__
is added to capturebazel-bin
.
This is mainly for easy local development:
$ bazel build .....
$ python -m pytest tests --binary bazel-bin -s
The above will be all we need for local development and debugging. No installation required.
to address review feedback Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good. Thanks for the clarification!
This looks great, thanks for the effort @yongtang ! Trying it out now |
@BryanCutler Here is the complete script I used to do a local development:
Note with the new changes it is not necessary to build with custom-op. I am using Ubuntu 18.04. Let me know if you encounter any issues. |
Yeah, works great! Would you mind if I put the above commands in a script and update the |
@BryanCutler Sure, that would be great! |
This is a Travis CI rework. The idea is to split C++ build from Python and R build. Also, bazel will only be used for C++ build, and python testing has been switched to pytest.
[Update ] This PR is now ready. The Travis CI now spend around 15 min each and total (3 jobs) is about 1 hour. This is about 25% of our previous Travis CI CPU usage.
Major changes:
python -m pytest tests
.Instead, all are packed into setup.py (with
setuptoools.sandbox.run_setup
)Note: The above 6) means:
We should provide Python 3.7 support to align TF 2.0, not there yet.
Some major changes related to development:
Pretty much switched from bazel to pytest, which is common in python projects. It is possible to test with out packaging the whl file and install:
Signed-off-by: Yong Tang yong.tang.github@outlook.com