asv continuous using existing environment (like "--python same")

[jorisvandenbossche]

When developing on a branch, you can use asv continuous branch to check the performance of that branch to master. I think this is a really useful during development.
But, this will also create new environments for both (which can take some time). However, when I am developing on a branch, I already have set up a full development environment (and I think this is a common case). Would it be possible to give asv continuous the option to use that?

For example, considering you only touched python code and you don't need to rebuild the project, a manual approach for the above would be:

git checkout master
asv run --python same
git checkout feature_branch
asv run --python same

and then compare the result of both runs manually.
(and if you touched some C extensions, then doing a python setup.py build_ext before each run is can be much quicker than creating a full new environment, certainly if only part of the extensions have to rebuilt)

[pv]

To compare, asv needs to rebuild. However, it cannot do code checkouts or build outside the virtual environment. It also cannot save the results, because it does not what is the current commit for the installed version.

[pv]

The environment setup is done only when this is done the first time, so it should amortize to 0 in principle. However, clean rebuilds can still take time.

Note that you probably should have wheel_cache_size set to some nonzero value to avoid rebuilds required for benchmark discovery.

[pv]

To make this sort of mode of operation possible, best way would probably be to add a new environment type inplace would need to be added (alongside conda and virtualenv).
Additionally, it should be possible to select the environment type on command line.

The inplace environment would then (i) use python setup.py build_ext -i to build the project, (ii) only work with local repositories, and (iii) also do checkouts in-place, clobbering the working tree. Note that step (iii) also clobbers the benchmark suite if it's in the same repository, so I'm not sure if this sort of approach can be made to work.

[jorisvandenbossche]

Ah, yes, indeed, the issue with the benchmarks itself also possibly changing in a checkout, is something I overlooked. This makes this approach indeed less attractive.

[mdboom]

Not to mention the various issues with inplace building and C/C++ extensions, particularly with multiple Python versions. I personally really prefer a strong separation between source and build products -- it makes a lot of things much simpler.

[yarikoptic]

I am in dire desire for such a feature as well. I guess for "existing" (if results are stored for benchmarks ran in commits, not in dirty trees), contineous could just reload the results for the base (if present, if not -- error out) and then do its regular run -E existing and do compare against the stored one. Would be super useful.