Add pip-run support for extensions

[the-allanc]

After discovering ulauncher a couple of days ago, and experimenting with plugins, I've created a small change which would allow extensions to indicate what third-party dependencies they would need using pip-run.

By adding a __requires__ definition within main.py, a script can indicate which dependencies it needs to be installed. These dependencies are then installed in a transient virtualenv from which the script can run from.

My changes assumes that pip-run is available somewhere on the system path, and only attempts to run a script using pip-run if indicated in the manifest.

One slight issue is that this adds some overhead (even if the libraries are cached locally), and there is enough of a delay (at least with testing locally) between Ulauncher executing the extension and the extension being ready that you often won't get any options being presented during the first invocation.

I've had to make these changes against the dev branch rather than the v6 branch, as I've been struggling to run the v6 branch locally (I've filed #947 in relation to this).

Anyway, interested in what your thoughts are as a proof-of-concept.

[friday]

This relates to #273 and #721. We haven't decided on how to solve them yet, but I'm not as knowledgeable about venvs and similar techniques as Troy so I think it's better he has a look at it. We have discussed namespace packages as the solution ifnot using the proposal in #721.

As mentioned in the other issue, you don't need Docker any more to contribute. It's easier now. In fact I'm not sure where you found the references to Docker installation as they were not in the v6 branch and removed recently from the dev branch also.

[the-allanc]

Finally got the v6 branch running locally, and so I've updated my changes against that branch.

[the-allanc]

This relates to #273 and #721. We haven't decided on how to solve them yet, but I'm not as knowledgeable about venvs and similar techniques as Troy so I think it's better he has a look at it. We have discussed namespace packages as the solution ifnot using the proposal in #721.

Just some reasons why I advocate for pip-run as a solution:

1. Having a virtualenv created on demand (and deleted afterwards) saves any hassle of trying to maintain that environment.
2. Being able to write single file scripts which can also state their dependencies makes the script much more convenient - although extensions require multiple files anyway, so that's not as much of a selling point here.

[troycurtisjr]

I'll take a close look this weekend, it seems really interesting, I've not come across pip-run before.

[friday]

One big issue I can see with this is that pip-run isn't supported as a dependency in distros, while virtualenv is. pip-run is just a convenient wrapper for virtualenv, so I think we can do the same thing as pip-run does without needing pip-run.

If I understand things right it also seem that every time Ulauncher would launch, it would download and install the same extension dependencies, install the extensions to a temporary location and then afterwards delete them? I think this would slow down the initial start of Ulauncher by a lot if this is the case.

[the-allanc]

pip-run would need to be installed by a user manually, yes - but then it would be the only dependency that a user would need to installed.

Ulauncher could be in charge of creating environments, but pip-run tears them down after use, so you would have to work out either a similar strategy or if they were persisted, then you would have to determine how they are maintained if dependencies change.

Regarding how the code currently works - the overhead of initialising the environment is in the spawned process for the plugin. So currently in this branch, Ulauncher starts up just fine, but when the plugin is initialised, it usually isn't ready quickly enough to respond to the first keyword query event.

To counter that, it might be suitable (either as a preference or just default behaviour by Ulauncher) to start the plugin before the user attempts to use it; that way, a user shouldn't notice the overhead.

[troycurtisjr]

I haven't had the chance yet to really dig into the PR itself, but from the description of the pip-run implementation I doubt that this specific approach will be the right way to go. The overall idea is on the right track, but I don't think it's reasonable to have users install, and presumably need to keep updated, pip-run when ulauncher itself is available in a distro's package manager. Individual extensions may require manual installation now, which is not ideal obviously, but at least if they install using their package manager it will then be kept up-to-date automatically. Additionally for me personally having the churn of each environment created and torn down every invocation is pretty much a deal-breaker as a user, which means I wouldn't want to pass it on to general users. It would also make extension development pretty painful. I presume this also needs an Internet connection at runtime to work? I see that it relies on the pip cache to not require the download of individual packages each time, but I think it still needs the connection to decide what to install, and what is needed, right?

…

On Sun, Jan 9, 2022 at 8:43 PM Allan Crooks ***@***.***> wrote: pip-run would need to be installed by a user manually, yes - but then it would be the only dependency that a user would need to installed. Ulauncher could be in charge of creating environments, but pip-run tears them down after use, so you would have to work out either a similar strategy or if they were persisted, then you would have to determine how they are maintained if dependencies change. Regarding how the code currently works - the overhead of initialising the environment is in the spawned process for the plugin. So currently in this branch, Ulauncher starts up just fine, but when the plugin is initialised, it usually isn't ready quickly enough to respond to the first keyword query event. To counter that, it might be suitable (either as a preference or just default behaviour by Ulauncher) to start the plugin before the user attempts to use it; that way, a user shouldn't notice the overhead. — Reply to this email directly, view it on GitHub <#948 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD4FBALABDVX7ETKWWDSKZDUVI2SJANCNFSM5LAHS5YQ> . You are receiving this because your review was requested.Message ID: ***@***.***>

[friday]

Yeah. It sounds like you found a great tool that you enjoy to use. But it doesn't seem to fit with our needs. Appreciate you taking the time to read our guidelines, contributing and rebasing over v6 and all.

[the-allanc]

Let me reply to those comments in brief:

Individual extensions may require manual
installation now, which is not ideal obviously, but at least if they
install using their package manager it will then be kept up-to-date
automatically.

Maybe there is a difference between how you and I would be imagining how users would be managing third-party dependencies required by extensions; rather than installing a package via apt, I imagined that the user would have to use pip to install their dependencies. In that case, pip-run is actually a better alternative when it comes to using up-to-date dependencies.

Additionally for me personally having the churn of each
environment created and torn down every invocation is pretty much a
deal-breaker as a user

I wrote this pull request as I'm writing my own plugin which requires external libraries. The overhead of the initial initialisation is annoying, but once the extension is running, then performance isn't an issue. Maybe you still find that unacceptable (which is why I suggested in my previous comment to allow a plugin to be run as soon as Ulauncher is launched), but I do want to make it clear that the overhead only applies on the first invocation, not every time you send a new keyword query event.

It would also make extension development pretty painful.

Using this branch, I define a single use_pip_run option in the manifest, and add a single __requires__ = ['thedependency'] line in main.py. That was all I needed to be able to start writing my extension.

I presume this also needs an Internet connection at runtime to work?

Yes. Annoyingly, there doesn't seem to be a good way to utilise the local pip cache without a call to Pypi first. The best offline solution is to run a Devpi mirror, which is the recommendation from the pip-run author himself.

Making changes to work around that give a stronger case to instead managing environments directly within Ulauncher - ideally, you want environments to be persisted (which pip-run doesn't do), and it's easy to read dependencies from the manifest, so the advantage of being able to mention the dependencies in the main script file isn't compelling here.

Anyway, as a proof-of-concept, this was pretty straight-forward to implement. If adding virtualenv support for extensions takes a particularly long time to achieve, then you might want to consider pip-run as a stop-gap solution.

I'll let you guys decide whether you want to close the MR or if you think it might have some merit at some point. Thanks for taking a look.

[troycurtisjr]

@the-allanc You may be interested in my latest comment I just added to #273 where I outline an approach using virtualenv that I think could be workable for ulauncher's usage. My plan was to take a stab at actually implementing that idea, however if you have interest in giving it a shot you are more than welcome to. I just wanted to check in to determine if this is an area that you have a strong interest in contributing in as I don't want to take away the opportunity if you do. Just let me know!