Setup py improvements #38
Conversation
Prevents accidental overwriting of changes (or unnecessary pulling of more stuff when I explicitly *not* init a subset of the modules).
facebook-github-bot
added
the
CLA Signed
setup.py
Outdated
| # Init & update all submodules if not already (the user might be pinned | ||
| # on some particular commit or have working tree changes, don't destroy | ||
| # those) | ||
| if in_git() and not os.path.exists(".git/modules"): |
There was a problem hiding this comment.
I like the idea of this -- I have resorted to some pretty hacky hacks to get git to not update submodules (adding a random space or newline to a file and then not committing it being my favorite).
However, I also like that currently when people pull a new build, they don't have to worry about updating submodules. Would adding a --no-update-modules flag be sufficient (I am about to submit a PR adding a --cmake-args flag, so I can add it to that)?
CC: @msavva
There was a problem hiding this comment.
I see your point. Yes that would work too, I would just need to remember to always add that flag (which is okay) :)
A more general question from a setup.py noob -- do setuptools have something similar to cmake's cache? Like, you enable / set up some flags and properties and they get remembered for the next invocation; so you don't have to specify the whole command-line but can do just ./setup.py and it will pick up the rest from somewhere, which is (like cmake's build dir) not commited to the git repository but private to your working copy? Or am I asking for the impossible?
There was a problem hiding this comment.
Yeah... unfortunately setup.py doesn't do that. We could roll our own as pickling out args and reloading if it exists would work.....
There was a problem hiding this comment.
I also like the idea. I agree with @erikwijmans 's suggestion to make this into a --no-update-modules flag. It's good to accommodate for the expectation that everything should work through a naive pull-and-build. I'm not aware of a way to make setup.py cache flags etc. :(
There was a problem hiding this comment.
Made a PR into this PR with a go at caching (it contains more than just that tho).
* Argparse * --cmake-args * Caching of arguments to build_ext and argparse
|
Merged #40 into this. This should be good to go too! |
* setup.py: option to don't run git submodule update if modules already exist. * setup.py: make it possible to *reduce* the amount of jobs. * Setup.py -- Argprase, Cmake Args, and caching (facebookresearch#40) * Argparse * --cmake-args * Caching of arguments to build_ext and argparse
Motivation and Context
While #24 said that
I'm actually one of those crazy multitaskers :) 100% CPU wouldn't be a problem, but when running on all cores, something in the build demands 8 GB of RAM which I usually don't have, so my system comes to a grinding halt. (I still want to investigate what's responsible and if it's possible to tell
ninjato not overschedule such jobs.)While the default is still all the cores, it now allows to override the setting using the setuptools' builtin
--paralleloption. Together with redirecting it to use a pre-existing CMake build (which was set up manually for #35) dir my command-line looks like this:Apart from this, because I use some system deps, I'm not interested in
setup.pyimplicitly pulling submodules that I am not interested in, so it runsgit submodule updateonly the very first time.How Has This Been Tested
I'm now able to both have 163 tabs open in my browser and build Habitat :)
Types of changes
dependency upgradebuildsystem updateChecklist