Create a python app for dependencies instead of venv #16694
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This started in PR chapel-lang#9137 but the reason is lost to time. Leaving this flag out will allow pip to use its local cache and reduce reliance on the network. --- Signed-off-by: Michael Ferguson <mppf@users.noreply.github.com>
--- Signed-off-by: Michael Ferguson <mppf@users.noreply.github.com>
it is included in python3 --- Signed-off-by: Michael Ferguson <mppf@users.noreply.github.com>
Started to do this as a zipapp but sphinx doesn't like to exist within a zipfile (we get errors when it tries to open a theme). So now we just install the dependencies and extend the python path to include them. For convenience, the 'chpldeps' Python app will run included dependencies, e.g. chpldeps sphinx-build arg... --- Signed-off-by: Michael Ferguson <mppf@users.noreply.github.com>
--- Signed-off-by: Michael Ferguson <mppf@users.noreply.github.com>
--- Signed-off-by: Michael Ferguson <mppf@users.noreply.github.com>
--- Signed-off-by: Michael Ferguson <mppf@users.noreply.github.com>
Can't have a regular target depend on a phony one. --- Signed-off-by: Michael Ferguson <mppf@users.noreply.github.com>
--- Signed-off-by: Michael Ferguson <mppf@users.noreply.github.com>
There was a problem hiding this comment.
What is the behavior:
- on a Mac after a clobber?
- on Linux after a clobber?
- when calling
make chpldoc/make test-venvwhen nothing has changed in chpl-venv since the last time it was run? - with a
make docscall when chpldoc is not built? - with a
make docscall when chpldoc has been built?
ditto
nothing is downloaded
it builds chpldoc. it does not rebuild the chpldeps if they have not changed.
nothing is downloaded because |
For now, this uses the third-party/chpl-venv/build-venv because I'm not so sure it makes sense to include the chplspell dependency in what gets installed since it is a developer-only tool. --- Signed-off-by: Michael Ferguson <mppf@users.noreply.github.com>
--- Signed-off-by: Michael Ferguson <mppf@users.noreply.github.com>
mppf
added a commit
to mppf/chapel
that referenced
this pull request
Nov 12, 2020
I meant to remove this in PR chapel-lang#16694 which makes it no longer used. --- Signed-off-by: Michael Ferguson <mppf@users.noreply.github.com>
mppf
added a commit
that referenced
this pull request
Nov 12, 2020
Remove python3-wrapper I meant to remove this in PR #16694 which makes it no longer used. Trivial and not reviewed.
6 tasks
mppf
added a commit
that referenced
this pull request
Nov 13, 2020
Tidy up Makefile error handling for chpl-venv Follow-up to PR #16694 * Removes check for a working pip since `python3 -m venv` creates a venv that is required to have pip. * Adjusts the Makefile targets running pip to better handle errors. Before, it was creating the build/chpl-venv or install/chpldeps directories even if the `pip` command failed. Stop doing that. * Adjust run-in{-test}-venv.bash to check for `__main__.py` - To avoid problems if the directory exists but nothing within it does, which we observed in nightly testing when the `pip` command failed due to network errors. Reviewed by @lydia-duncan - thanks! - [x] verified that the `chpldeps` directory is not added when causing pip to fail by setting `PIP_INDEX_URL` to something bogus - [x] verified that `run-in-test-venv.bash` reports an error when I `mkdir` the `chpldeps` directory - [x] `make docs` works on Ubuntu 16.04 with Python 3.5 - [x] `make docs` works on a Cray XC - [x] `make docs` works on Mac OS X - [x] full local testing
mppf
added a commit
that referenced
this pull request
Nov 17, 2020
Fix chpl_home_utils.py --test-venv calls take 2 Follow-up to PR #16694 and #16560 Replaces #16722. The --test-venv option was removed in #16694, so adjust scripts that were using it. To support chpl-scripts-completion.bash, added `register-python-argcomplete` as a subcommand for the chpldeps application. While there, improved the usage checking for chpldeps. Reviewed by @ronawho - thanks!
daviditen
pushed a commit
to daviditen/chapel
that referenced
this pull request
Dec 3, 2020
I meant to remove this in PR chapel-lang#16694 which makes it no longer used. --- Signed-off-by: Michael Ferguson <mppf@users.noreply.github.com> Signed-off-by: David Iten <daviditen@users.noreply.github.com>
CaptainSharf
pushed a commit
to CaptainSharf/chapel
that referenced
this pull request
Jan 16, 2021
I meant to remove this in PR chapel-lang#16694 which makes it no longer used. --- Signed-off-by: Michael Ferguson <mppf@users.noreply.github.com> Signed-off-by: CaptainSharf <mohdsharf7407@gail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Follow-up to PR #16663 #16644 and #16560
We run into problems when including a venv in the installed code for
Chapel (e.g. in a Chapel RPM) because the venv encodes the path to the
python interpreter (both as symbolic links and also in pyvenv.cfg). We
tried to work around this in PRs #3571 #16663 and #16680.
This PR instead changes the strategy away from trying to create a
long-lasting venv in the
third-party/chpl-venv. Instead, it creates avenv during installation (so we can have
pipeven on systems that don'tinclude it by default) and then uses that venv to install the
dependencies into
chpldepswhich is a Python application. Now weinstall the dependencies with
python3 -m pip --target path/to/chpldeps.Additionally we no longer include
getpipsupport in the Makefiles as itshould no longer be used and we stop passing
--cache-dirto thepipcommands (so that the cache can be used night-to-night in nightly
testing).
Originally I had hoped to use
zipappto storechpldepsas a zipfilebut that causes problems with Sphinx (when it looks for theme files). So,
here we just have
chpldepsas a directory (which Python considers akind of application) just as we would before zipping it with
zipapp.zipappdoes not do much other than storing the files into a .zip and sothis should have similar portability.
To keep it convenient, Python programs needing
chpldepscan be run aspython3 path/to/chpldeps some/python/program.pyor for certain includedtools,
python3 path/to/chpldeps sphinx-build. This avoids the need forutil/config/run-in-venv.bashin many cases.Lastly, I wasn't sure what to do with chplspell for this PR since
chplspell is a developer-only tool. For now, it uses the same venv that
we install pip into.
Reviewed by @ben-albrecht and @lydia-duncan - thanks!
test_rpm_module.bash