Could pipfile be designed to support a MetaPathFinder implementation that installs the requirements 'on-demand' ?

[lonetwin]

I learned about MetaPathFinder class from importlib and created this hack:

http://lonetwin.github.io/blog/html/2016/05/16/auto_install_missing_python_modules.html

Since I discovered this hack, I have been thinking that something of an 'install-on-missing-import' feature would be, in theory, a possibility,

I've been wrestling with getting an implementation which works with requirements.txt for and in fact, I do have a rough implementation which parses the requirements.txt and installs the requirements at import time. This works for most simple installations, although I haven't tested it extensively.

I would like to suggest that maybe pipfile could designed (and possibly provide a reference implementation) of a MetaPathFinder which ensures that the developers and users of modules don't even have to worry about running a command to install anything. Just executing or importing the module should install the necessary dependencies.

This is just a suggestion and I'm ok if people think this idea is ridiculous. That said, knowing the reasons why such a thing would be ridiculous would be appreciated.

[FRidh]

Just executing or importing the module should install the necessary dependencies.

Module names are not unique; multiple packages can provide modules with the same name.

Furthermore, pipfile shall provide a deterministic way to describe an environment; installing on demand defies the point because you introduce state. (How will one know what version of django will be installed when importing?)

[lonetwin]

Just executing or importing the module should install the necessary dependencies.

Module names are not unique; multiple packages can deliver modules with the same name.

Right, but resolving that ambiguity would be done by checking the pipfile. That's no different than what pip/pipfile will do, right ? A MetaPathFinder implementation would load a pipfile . What I am suggesting is moving the call to pip install .. into a hook that gets called at import time.

Furthermore, pipfile shall provide a deterministic way to describe an environment; installing on demand defies the point because you introduce state. (How will one know what version of django will be installed when importing?

By loading the pipfile/Pipfile.lock in the MetaPathFinder implementation ?

[FRidh]

By loading the pipfile/Pipfile.lock in the MetaPathFinder implementation ?

And why would one ever want that? If you need an environment with packages, then you build that environment and then start using it. Why would you build a smaller version first and then install during runtime? Just so you don't have to do a pip install Pipfile.lock in advance?

Anyway, Pipfile.lock is supposed to be easily readable so I don't see any issue preventing you from parsing it and installing from it during runtime.

[lonetwin]

Why would you build a smaller version first and then install during runtime? Just so you don't have to do a pip install Pipfile.lock in advance?

I'm sorry but I'm not sure what you mean by "build a smaller version first and then install during runtime".

The example above doesn't do that, neither was I suggesting that we do that. The reason one would want to do what I suggested is you can tell users, "download and run foo" rather than telling them to "download, run pip install -r/-p whatever and run foo" -- yeah it doesn't seem like a big deal, I just was thinking from a distribution point of view where the only necessary command line to (for instance) run a UI based python package can be avoided.

Anyway, Pipfile.lock is supposed to be easily readable so I don't see any issue preventing you from parsing it and installing from it during runtime.

Yep, thanks. On IRC there were a number of valid reason why doing something like this might be fraught with difficulties so I'll close this. Although, I think packaging and distributing modules this way in predictable environments would not be a bad thing.

Thanks for your attention.

[FRidh]

The example above doesn't do that, neither was I suggesting that we do that. The reason one would want to do what I suggested is you can tell users, "download and run foo" rather than telling them to "download, run pip install -r/-p whatever and run foo" -- yeah it doesn't seem like a big deal, I just was thinking from a distribution point of view where the only necessary command line to (for instance) run a UI based python package can be avoided.

That is an interesting use case. You do not want to pollute your global site packages I would imagine (or whatever env you're in at that moment), so a better approach would be to create a new Python env first.

With Nix we have something like that actually:

#! /usr/bin/env nix-shell
#! nix-shell -i python -p 'python.withPackages(packages: [ packages.prettytable])'

import prettytable

# Print a simple table.
t = prettytable.PrettyTable(["N", "N^2"])
for n in range(1, 10): t.add_row([n, n * n])
print t

When executed, the script builds an environment where there's a Python interpreter along with that package.

I guess you could do something similar to this in Python by running the file twice (a bit similar to what is done with setup.py and setup_requires, but it won't ever be exactly reproducible, because you don't know the state of the first interpreter.

[xavfernandez]

you can tell users, "download and run foo" rather than telling them to "download, run pip install -r/-p whatever and run foo"

@lonetwin You might want to take a look at https://pypi.org/project/rwt/

[lonetwin]

@xavfernandez thanks! Yes, rwt appears to be pretty much what @FRidh describes as a possible approach.