documentation

BUGS.md

@@ -2,6 +2,19 @@ This is a simple listing of bugs previously encountered:
 
 Known Bugs
 ============
+This is just a place to brain-dump bugs I've found so I don't go insane trying to remember them.
+It's much lighter weight than filing tickets, and I like that it's version controlled.
+
+    * venv-update shows the same `> virtualenv venv` line twice in a row
+
+    * the first venv-update fails with "filename too long" in a download cache file,
+        but subsequent run succeeds
+        TESTCASE: add a super-extra-really-obscenely-long-named-package
+        FIX: don't set download-cache within pip-faster. i dont think it was speeding anything up. the wheels are what
+        matter.
+
+    * if the "outer" pip is >6, installing pip1.5 shows "a valid SSLContext is not available" and "a newer pip is available"
+        we can suppress these with PIP_NO_PIP_VERSION_CHECK and python -W 'ignore'
 
     * venv-update can `rm -rf .`, if '.' is its first argument.
 
@@ -25,7 +38,6 @@ Known Bugs
 Annoyances
 ==========
 
-    * `print 1; print 2` is coming from somewhere during py.test -s
     * `capture_subprocess` doesn't properly proxy tty input/output.
       see: https://github.com/bukzor/ptyproxy
 
@@ -48,6 +60,16 @@ pip-faster install:
       Planned fix: for the purposes of pruning, pip-faster should be added to the list of requirements
       Stopgap fix: whitelist argparse along with pip-faster, pip, setuptools, and wheel to never be pruned
 
+test:
+    * during make-sdists, the setup.py for pip-faster went missing, once
+      Cause: parallel test fixtures were stomping on each others' egg-info
+      Fix: set a --egg-dir for egg-info
+
+
+Magically Fixed
+===============
+
+    * `print 1; print 2` is coming from somewhere during py.test -s
 
 Fixed and Tested
 ================

README.md

@@ -1,26 +1,58 @@
-# pip-faster
-[![Build Status](https://travis-ci.org/Yelp/pip-faster.svg?branch=master)](https://travis-ci.org/Yelp/pip-faster)
-[![Coverage Status](https://img.shields.io/coveralls/Yelp/pip-faster.svg?branch=master)](https://coveralls.io/r/Yelp/pip-faster)
+# pip-faster [![Build
+Status](https://travis-ci.org/Yelp/pip-faster.svg?branch=master)](https://travis-ci.org/Yelp/pip-faster)
+[![Coverage
+Status](https://img.shields.io/coveralls/Yelp/pip-faster.svg?branch=master)](https://coveralls.io/r/Yelp/pip-faster)
 
-`venv-update` is a tool for keeping a virtualenv up-to-date with version-controlled requirement lists.
-It's optimized for big projects (hundreds of requirements), and as such makes heavy use of caches, wheels and wheel-caches. We use it when working on the main Yelp codebase. The tool also ensures that no "extraneous" packages are installed after an update; the goal is that your virtualenv in the same state as if you had rebuilt it from scratch, but *much* more quickly.
+`venv-update` is a tool for keeping a virtualenv up-to-date with
+version-controlled requirement lists.  It's optimized for big projects
+(hundreds of requirements), and as such makes heavy use of caches, wheels and
+wheel-caches. We use it when working on the main Yelp codebase. The tool also
+ensures that no "extraneous" packages are installed after an update; the goal
+is that your virtualenv in the same state as if you had rebuilt it from
+scratch, but *much* more quickly.
 
 
 ## Installation
 
-Because this tool is meant to be the entry-point for handling requirements and dependencies, it's not meant to be installed via pip usually. The design allows the single venv_update.py to be vendored (directly checked in) to your project, and run without any dependencies beyond the python standard library.
+Because this tool is meant to be the entry-point for handling requirements and
+dependencies, it's not meant to be installed via pip usually. The design allows
+the single venv_update.py to be vendored (directly checked in) to your project,
+and run without any dependencies beyond the python standard library.
 
 
 ## Usage
 
 
-Simply running `venv_update.py` will create a virtualenv named `virtualenv_run` in the current directory, using `requirements.txt` in the current directory. These are default values that can be overridden by providing arguments. Pass `--help` for more detail.
+Simply running `venv_update.py` will create a virtualenv named `virtualenv_run`
+in the current directory, using `requirements.txt` in the current directory.
+These are default values that can be overridden by providing arguments. Pass
+`--help` for more detail.
 
 
 ## Features:
 
- * Caching: All downloads and wheels are cached (in `~/.pip/cache` and `~/.pip/wheelhouse`, respectively). You shouldn't have to wait for anything to download or build twice.
- * All packages are built to wheels before installation. This means that if you're using a package that takes a bit to compile (`lxml`) or on a platform that doesn't have public-pypi wheel support (linux), you still get the speed advantages associated with wheels.
- * Extraneous packages are uninstalled. This helps ensure that your dev environment isn't polluted by any previous state of your project. "Extraneous" packages are those that are neither directly required, nor required by any direct requirement.
- * Dependency conflict detection: stock pip will happily install two packages with conflicting requirements, with undefined behavior for the conflicted requirement. For now, venv-update gives the same result as pip, but at least throws you a yellow warning when such a situation arises. In future (once we fix our code base's issues) this will be an error.
- * Minimize pypi round-trips: We've taken great pains to reduce the number of round-trips to pypi, which makes up the majority of time spent on what should be a no-op update. With a properly warmed cache, you should be able to rebuild your virtualenv with no network access.
+ * Caching: All downloads and wheels are cached (in `~/.pip/cache` and
+   `~/.pip/wheelhouse`, respectively). You shouldn't have to wait for anything
+   to download or build twice.
+ * All packages are built to wheels before installation. This means that if
+   you're using a package that takes a bit to compile (`lxml`) or on a platform
+   that doesn't have public-pypi wheel support (linux), you still get the speed
+   advantages associated with wheels.
+ * Extraneous packages are uninstalled. This helps ensure that your dev
+   environment isn't polluted by any previous state of your project.
+   "Extraneous" packages are those that are neither directly required, nor
+   required by any direct requirement.
+ * Dependency conflict detection: stock pip will happily install two packages
+   with conflicting requirements, with undefined behavior for the conflicted
+   requirement. For now, venv-update gives the same result as pip, but at least
+   throws you a yellow warning when such a situation arises. In future (once we
+   fix our code base's issues) this will be an error.
+ * Minimize pypi round-trips: We've taken great pains to reduce the number of
+   round-trips to pypi, which makes up the majority of time spent on what
+   should be a no-op update. With a properly warmed cache, you should be able
+   to rebuild your virtualenv with no network access.
+
+
+
+<!-- vim:textwidth=79:
+-->

TODO.md

@@ -3,14 +3,15 @@ NEXT
 ====
 We plan to work on these as soon as possible.
 
+* neither pip-faster nor venv-update report their *own* versions
 
 - [ ] testing
   - [ ] PIP\_FIND\_LINKS=https://...
   - [ ] pip-faster install nonsense
 
 
  - [ ] dogfood venv-update during travis, tox
-   - [ ] recommended tox config: install_command=pip-faster --prune {opts} {packages}
+   - [ ] recommended tox config: `install_command=pip-faster --prune {opts} {packages}`
 
 * change host-python requirement to simply virtualenv, any version
 
@@ -19,6 +20,23 @@ We plan to work on these as soon as possible.
    - [ ] run from virtualenv which doesn't have virtualenv installed
    - [ ] update an active virtualenv which wasn't created by venv-update
 
+* upgrade-install of unpinned requirements is ~50% slower than vanilla pip.
+  compare:
+
+  - time pip install --find-links file://$HOME/.pip/wheelhouse --upgrade -r
+    requirements.d/dev.txt -vvvv > fast
+
+    real    0m3.107s
+    user    0m2.424s
+    sys     0m0.427s
+
+  - time make venv > slow
+
+    real    0m4.870s
+    user    0m4.021s
+    sys     0m0.631s
+
+