Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Consolidate builder into an Invoke Task #1596

Merged
merged 2 commits into from

3 participants

@dstufft
Owner

This moves contrib/builder-installer into an invoke task which can be executed as invoke generate.installer. It also fixes a small thing where we were using get-pip.py as a temporary staging file.

@dstufft dstufft merged commit 2795578 into pypa:develop

1 check passed

Details default The Travis CI build passed
@dstufft dstufft deleted the dstufft:consolidate-builder branch
@pfmoore
Owner

This does bother me somewhat more than making the authors generation an invoke task, as I do on occasion run build-installer manually (for testing, making a dev copy, etc).

Can we not preserve the ability to run build-installer without needing invoke (for Windows users)? Does it have to be an all-or-nothing change?

@pfmoore
Owner

While I don't want to get into a "competing task runners" debate, can I ask for some clarification on how invoke is impacting this code? From what I see:

  1. The functions implementing the tasks are decorated with @invoke.task.
  2. authors() runs git using invoke.run rather than (say) subprocess.check_call
  3. There are new print statements added to log progress (which is essentially irrelevant other than in terms of the format of the message).

It seems to me that invoke has a pretty minimal footprint, and it wouldn't be hard to make the code compatible with running outside of invoke:

Wrap the use of invoke in a try..except (or move the driver functions - see below - into a separate file.

Have the functions and the tasks separate:

    def authors_impl():
        ...
    authors = invoke.task(authors_impl)

Then people without invoke could call authors_impl direct from a driver script.

I have no idea what invoke.run gains over subprocess.check_output (given that we don't use the pty support) but if it's important to not use the latter when running via invoke, we could factor out the subprocess runner to use.

I'm willing to make these changes (someone should justify the use of invoke.run or I'll probably just replace it with subprocess) but only if there's agreement that future additions will follow the same pattern. If we don't want to go down this route, can we just factor out the installer task so it remains usable without invoke?

(I'm still looking into doit as an alternative to invoke for task running. If someone can list the key benefits to pip of invoke, that'll help me know what's important to replicate).

@Ivoz
Owner

I'd prefer to keep on using invoke and make it windows compatible. envoy could be a good candidate to replace invoke's current use of pexpect.

@dstufft
Owner

Sorry if maybe I got a little overzealous :) Mostly I'm trying to centralize stuff and get a single entry for this stuff that can be a little more standardized.

Besides pty support I think the only thing invoke.run() brings to the table is a nicer API that is more unified. We probably don't need pty support (I can't think of anything we'd want to work with that would need a pty) and certainly if we find something that we do want to work with that requires a pty we can revisit the use of invoke.run()vs subprocess.

As far as invoke vs something else, I admit that I didn't really bother to look around. Invoke is what I use for other projects and I'm friends with the maintainer/creator so I just reached for my standard tool. I would not be entirely opposed to switching as I think the benefit of getting all of these into one central location with one central method of calling them is still there but if Windows support can be added to invoke I'd prefer that over switching. I'm also OK with providing a contrib/build-installer that just reaches into the tasks module and calls the function.

I don't know how big of a deal adding Windows support to invoke would be (to be honest I'm not sure of the differences that would affect something like that at all) and I don't want to "volunteer" or invent busy work for you @pfmoore. What do you think is the best path forward here?

@pfmoore
Owner

I'm happy to help with porting invoke to Windows. (BTW, @Ivoz I don't think we need envoy - the only use of pexpect and pty in invoke is when the pty=True flag is set and I'd be perfectly happy just to raise an error in that case).

My main issue with invoke is that when I went and looked at the github project (which is annoyingly hard to find from the PyPI page, btw!) there's a month-old patch to provide Windows support. I know a month isn't long, and it's a self-described "quick and dirty" job, but it's about where I'd have started (the poster picked up a few points I'd not spotted). But other than a ping and a "me, too" message, there's no comment from anyone. I've added a comment saying I'm willing to help with Windows support, but I'm concerned whether Windows patches will get attention (let alone whether they'll get applied!)

@dstufft if you know the maintainer, maybe you could point him at pyinvoke/invoke#113? I'd be happy to have a discussion with him under that issue about how best to add Windows support while still keeping in line with how he wants the project to go. I can appreciate that if he's not a Windows user he's taking patches somewhat on trust, let's work out how we can make that OK for him. (Heck, if he wants, I can be the "annoying Windows guy" on invoke just as easily as I currently am for pip :-))

@dstufft
Owner

Sure I can poke him :) He's on California time so probably not awake yet :)

@pfmoore
Owner

No problem, I'm away this weekend so I'll probably not get back to this till next week anyway.

@Ivoz
Owner

@bitprophet seems to get around to things slowly. He's not completely useless though afaik :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
Showing with 176 additions and 164 deletions.
  1. +0 −164 contrib/build-installer
  2. +171 −0 tasks/generate.py
  3. +5 −0 tasks/paths.py
View
164 contrib/build-installer
@@ -1,164 +0,0 @@
-#!/usr/bin/env python
-import base64
-import os
-import sys
-import zipfile
-
-WRAPPER_SCRIPT = b"""
-#!/usr/bin/env python
-#
-# Hi There!
-# You may be wondering what this giant blob of binary data here is, you might
-# even be worried that we're up to something nefarious (good for you for being
-# paranoid!). This is a base4 encoding of a zip file, this zip file contains
-# an entire copy of pip.
-#
-# Pip is a thing that installs packages, pip itself is a package that someone
-# might want to install, especially if they're looking to run this get-pip.py
-# script. Pip has a lot of code to deal with the security of installing
-# packages, various edge cases on various platforms, and other such sort of
-# "tribal knowledge" that has been encoded in it's code base. Because of this
-# we basically include an entire copy of pip inside this blob. We do this
-# because the alternatives are attempt to implement a "minipip" that probably
-# doesn't do things correctly and has weird edge cases, or compress pip itself
-# down into a single file.
-#
-# If you're wondering how this is created, the secret is
-# "contrib/build-installer" from the pip repository.
-
-ZIPFILE = b\"\"\"
-{zipfile}
-\"\"\"
-
-import base64
-import os.path
-import pkgutil
-import shutil
-import sys
-import tempfile
-
-
-def bootstrap(tmpdir=None):
- # Import pip so we can use it to install pip and maybe setuptools too
- import pip
-
- # We always want to install pip
- packages = ["pip"]
-
- # Check if the user has requested us not to install setuptools
- if "--no-setuptools" in sys.argv or os.environ.get("PIP_NO_SETUPTOOLS"):
- args = [x for x in sys.argv[1:] if x != "--no-setuptools"]
- else:
- args = sys.argv[1:]
-
- # We want to see if setuptools is available before attempting to
- # install it
- try:
- import setuptools # noqa
- except ImportError:
- packages += ["setuptools"]
-
- delete_tmpdir = False
- try:
- # Create a temporary directory to act as a working directory if we were
- # not given one.
- if tmpdir is None:
- tmpdir = tempfile.mkdtemp()
- delete_tmpdir = True
-
- # We need to extract the SSL certificates from requests so that they
- # can be passed to --cert
- cert_path = os.path.join(tmpdir, "cacert.pem")
- with open(cert_path, "wb") as cert:
- cert.write(pkgutil.get_data("pip._vendor.requests", "cacert.pem"))
-
- # Use an environment variable here so that users can still pass
- # --cert via sys.argv
- os.environ.setdefault("PIP_CERT", cert_path)
-
- # Execute the included pip and use it to install the latest pip and
- # setuptools from PyPI
- sys.exit(pip.main(["install", "--upgrade"] + packages + args))
- finally:
- # Remove our temporary directory
- if delete_tmpdir and tmpdir:
- shutil.rmtree(tmpdir, ignore_errors=True)
-
-
-def main():
- tmpdir = None
- try:
- # Create a temporary working directory
- tmpdir = tempfile.mkdtemp()
-
- # Unpack the zipfile into the temporary directory
- pip_zip = os.path.join(tmpdir, "pip.zip")
- with open(pip_zip, "wb") as fp:
- fp.write(base64.decodestring(ZIPFILE))
-
- # Add the zipfile to sys.path so that we can import it
- sys.path.insert(0, pip_zip)
-
- # Run the bootstrap
- bootstrap(tmpdir=tmpdir)
- finally:
- # Clean up our temporary working directory
- if tmpdir:
- shutil.rmtree(tmpdir, ignore_errors=True)
-
-
-if __name__ == "__main__":
- main()
-""".lstrip()
-
-
-def getmodname(rootpath, path):
- parts = path.split(os.sep)[len(rootpath.split(os.sep)):]
- return '/'.join(parts)
-
-
-def main():
- sys.stdout.write("Creating pip bootstrapper...")
-
- here = os.path.dirname(os.path.abspath(__file__))
- toplevel = os.path.dirname(here)
- get_pip = os.path.join(here, "get-pip.py")
-
- # Get all of the files we want to add to the zip file
- all_files = []
- for root, dirs, files in os.walk(os.path.join(toplevel, "pip")):
- for pyfile in files:
- if os.path.splitext(pyfile)[1] in {".py", ".pem", ".cfg", ".exe"}:
- all_files.append(
- getmodname(toplevel, os.path.join(root, pyfile))
- )
-
- with zipfile.ZipFile(get_pip, "w", compression=zipfile.ZIP_DEFLATED) as z:
- # Write the pip files to the zip archive
- for filename in all_files:
- z.write(os.path.join(toplevel, filename), filename)
-
- # Get the binary data that compromises our zip file
- with open(get_pip, "rb") as fp:
- data = fp.read()
-
- # Write out the wrapper script that will take the place of the zip script
- # The reason we need to do this instead of just directly executing the
- # zip script is that while Python will happily execute a zip script if
- # passed it on the file system, it will not however allow this to work if
- # passed it via stdin. This means that this wrapper script is required to
- # make ``curl https://...../get-pip.py | python`` continue to work.
- with open(get_pip, "wb") as fp:
- fp.write(WRAPPER_SCRIPT.format(zipfile=base64.encodestring(data)))
-
- # Ensure the permissions on the newly created file
- if hasattr(os, "chmod"):
- oldmode = os.stat(get_pip).st_mode & 0o7777
- newmode = (oldmode | 0o555) & 0o7777
- os.chmod(get_pip, newmode)
-
- sys.stdout.write("done.\n")
-
-
-if __name__ == "__main__":
- main()
View
171 tasks/generate.py
@@ -1,7 +1,14 @@
+import base64
import io
+import os
+import shutil
+import tempfile
+import zipfile
import invoke
+from . import paths
+
@invoke.task
def authors():
@@ -26,3 +33,167 @@ def authors():
with io.open("AUTHORS.txt", "w", encoding="utf8") as fp:
fp.write(u"\n".join(authors))
fp.write(u"\n")
+
+
+@invoke.task
+def installer(installer_path=os.path.join(paths.CONTRIB, "get-pip.py")):
+ print("[generate.installer] Generating installer")
+
+ # Define our wrapper script
+ WRAPPER_SCRIPT = b"""
+#!/usr/bin/env python
+#
+# Hi There!
+# You may be wondering what this giant blob of binary data here is, you might
+# even be worried that we're up to something nefarious (good for you for being
+# paranoid!). This is a base64 encoding of a zip file, this zip file contains
+# an entire copy of pip.
+#
+# Pip is a thing that installs packages, pip itself is a package that someone
+# might want to install, especially if they're looking to run this get-pip.py
+# script. Pip has a lot of code to deal with the security of installing
+# packages, various edge cases on various platforms, and other such sort of
+# "tribal knowledge" that has been encoded in its code base. Because of this
+# we basically include an entire copy of pip inside this blob. We do this
+# because the alternatives are attempt to implement a "minipip" that probably
+# doesn't do things correctly and has weird edge cases, or compress pip itself
+# down into a single file.
+#
+# If you're wondering how this is created, it is using an invoke task located
+# in tasks/generate.py called "installer". It can be invoked by using
+# ``invoke generate.installer``.
+
+ZIPFILE = b\"\"\"
+{zipfile}
+\"\"\"
+
+import base64
+import os.path
+import pkgutil
+import shutil
+import sys
+import tempfile
+
+
+def bootstrap(tmpdir=None):
+ # Import pip so we can use it to install pip and maybe setuptools too
+ import pip
+
+ # We always want to install pip
+ packages = ["pip"]
+
+ # Check if the user has requested us not to install setuptools
+ if "--no-setuptools" in sys.argv or os.environ.get("PIP_NO_SETUPTOOLS"):
+ args = [x for x in sys.argv[1:] if x != "--no-setuptools"]
+ else:
+ args = sys.argv[1:]
+
+ # We want to see if setuptools is available before attempting to
+ # install it
+ try:
+ import setuptools # noqa
+ except ImportError:
+ packages += ["setuptools"]
+
+ delete_tmpdir = False
+ try:
+ # Create a temporary directory to act as a working directory if we were
+ # not given one.
+ if tmpdir is None:
+ tmpdir = tempfile.mkdtemp()
+ delete_tmpdir = True
+
+ # We need to extract the SSL certificates from requests so that they
+ # can be passed to --cert
+ cert_path = os.path.join(tmpdir, "cacert.pem")
+ with open(cert_path, "wb") as cert:
+ cert.write(pkgutil.get_data("pip._vendor.requests", "cacert.pem"))
+
+ # Use an environment variable here so that users can still pass
+ # --cert via sys.argv
+ os.environ.setdefault("PIP_CERT", cert_path)
+
+ # Execute the included pip and use it to install the latest pip and
+ # setuptools from PyPI
+ sys.exit(pip.main(["install", "--upgrade"] + packages + args))
+ finally:
+ # Remove our temporary directory
+ if delete_tmpdir and tmpdir:
+ shutil.rmtree(tmpdir, ignore_errors=True)
+
+
+def main():
+ tmpdir = None
+ try:
+ # Create a temporary working directory
+ tmpdir = tempfile.mkdtemp()
+
+ # Unpack the zipfile into the temporary directory
+ pip_zip = os.path.join(tmpdir, "pip.zip")
+ with open(pip_zip, "wb") as fp:
+ fp.write(base64.decodestring(ZIPFILE))
+
+ # Add the zipfile to sys.path so that we can import it
+ sys.path.insert(0, pip_zip)
+
+ # Run the bootstrap
+ bootstrap(tmpdir=tmpdir)
+ finally:
+ # Clean up our temporary working directory
+ if tmpdir:
+ shutil.rmtree(tmpdir, ignore_errors=True)
+
+
+if __name__ == "__main__":
+ main()
+""".lstrip()
+
+ # Get all of the files we want to add to the zip file
+ print("[generate.installer] Collect all the files that should be zipped")
+ all_files = []
+ for root, dirs, files in os.walk(os.path.join(paths.PROJECT_ROOT, "pip")):
+ for pyfile in files:
+ if os.path.splitext(pyfile)[1] in {".py", ".pem", ".cfg", ".exe"}:
+ path = os.path.join(root, pyfile)
+ all_files.append(
+ "/".join(
+ path.split("/")[len(paths.PROJECT_ROOT.split("/")):]
+ )
+ )
+
+ tmpdir = tempfile.mkdtemp()
+ try:
+ # Get a temporary path to use as as staging for the pip zip
+ zpth = os.path.join(tmpdir, "pip.zip")
+
+ # Write the pip files to the zip archive
+ print("[generate.installer] Generate the bundled zip of pip")
+ with zipfile.ZipFile(zpth, "w", compression=zipfile.ZIP_DEFLATED) as z:
+ for filename in all_files:
+ z.write(os.path.join(paths.PROJECT_ROOT, filename), filename)
+
+ # Get the binary data that compromises our zip file
+ with open(zpth, "rb") as fp:
+ data = fp.read()
+ finally:
+ shutil.rmtree(tmpdir, ignore_errors=True)
+
+ # Write out the wrapper script that will take the place of the zip script
+ # The reason we need to do this instead of just directly executing the
+ # zip script is that while Python will happily execute a zip script if
+ # passed it on the file system, it will not however allow this to work if
+ # passed it via stdin. This means that this wrapper script is required to
+ # make ``curl https://...../get-pip.py | python`` continue to work.
+ print(
+ "[generate.installer] Write the wrapper script with the bundled zip "
+ "file"
+ )
+ with open(installer_path, "wb") as fp:
+ fp.write(WRAPPER_SCRIPT.format(zipfile=base64.encodestring(data)))
+
+ # Ensure the permissions on the newly created file
+ oldmode = os.stat(installer_path).st_mode & 0o7777
+ newmode = (oldmode | 0o555) & 0o7777
+ os.chmod(installer_path, newmode)
+
+ print("[generate.installer] Generated installer")
View
5 tasks/paths.py
@@ -0,0 +1,5 @@
+import os.path
+
+PROJECT_ROOT = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
+
+CONTRIB = os.path.join(PROJECT_ROOT, "contrib")
Something went wrong with that request. Please try again.