Make N_CPUS check in ci_sanity.sh work on MacOS #29711
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The pylint sections of
tensorflow/tools/ci_build/ci_sanity.sh
do not run on MacOS because they contain code that assumes the existence of/proc/cpuinfo
.This is a problem for Mac users because pylint is extremely slow on Docker on Mac -- over 12 hours to to run a single pass of linting on my machine. The same pylint check runs in 15 minutes outside of Docker.
The changes in this PR allow the linting portions of
ci_sanity.sh
to run on MacOS with Docker. After my modifications,ci_sanity.sh
uses the POSIX standard commandgetconf
to query the number of cores if/proc/cpuinfo
does not exist.To get reliable results when running pylint locally, users will need a Python environment that is close to that of the TensorFlow Docker images. I have written a script that will set up such an environment. My script is currently checked into my TensorFlow config files repository at https://github.com/frreiss/tf-dev-conf/blob/master/lintenv.sh. I would be happy to contribute this script if there's interest.