Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] Proposal: Engineering a Common "Plugin" Infrastructure #59344

Open
deliciouslytyped opened this issue Apr 12, 2019 · 33 comments

Comments

@deliciouslytyped
Copy link
Contributor

commented Apr 12, 2019

Edit: I made a WIP PoC library https://github.com/deliciouslytyped/nix-rootedoverlay/ , it still needs a lot of work but seems to be functional.
Edit: Help wanted: someone to hand-hold me through the fixpoint stuff needed for overrides in this infrastructure I think I figured it out.

There are various separate, probably incompatible implementations of plugin infrastructure across nixpkgs, examples being at least:

(NOT REVIEWED indicates the author is tentatively aware of a plugin system but has not inspected the implementation)

  • TODO, list here
  • a not-yet-upstreamed implementation for Ghidra cannibalized from Idris
  • NOT REVIEWED: Idris (see above)
  • NOT REVIEWED: firefox / other mozilla software
  • NOT REVIEWED: Eclipse
  • package managers of interpreted languages (TODO: any 'sub'-package manager?)
    • NOT REVIEWED: python packages
    • NOT REVIEWED: ruby
  • NOT REVIEWED: Jekyll
  • NOT REVIEWED: redmine
  • NOT REVIEWED: perl #59215
  • HYPOTHETICAL wine package management
  • NOT REVIEWED retroarch
  • NOT REVIEWED gnome-shell
  • NOT REVIEWED terraform (was suggested for it's simplicity)
  • NOT REVIEWED r/rstudio/rwrapper/rpackages
    No implementation:
  • jenkins (?)

CONSIDER: asking maintainers of the various systems for input

The more advanced implementations usually (TODO: substantiate this) use some kind of fix-point mechanism which can be quite hard to understand.

Would it be possible and reasonable to factor out a plugin handling API that can be specialized to various desired implementations, such that the internal mechanisms do not need to be understood by the user? This could also result in the overall simplification of these components of nixpkgs.

A generic plugin system should:

  • provide one or more (possibly extensible) methods of applying plugins to a package:
    • feeding plugins back into the original derivation, e.g.
      {..., plugins ? ... }: mkDerivation ...
    • merging base and plugin derivations via something like symlinkJoin.
      This has the advantage of being faster and less wasteful than rebuilding the base package, but may fail with improper symlink handling? This also may have other failure modes such as runner makeWrapper scripts referring to the original derivation, resulting in the plugins being ignored because they are in a new derivation. (TODO: elaborate on this, this is an issue I ran into.)
    • ...other...?
  • provide the ability to have in-tree (nixpkgs) plugins
  • provide the ability to have out-of-tree plugins via overrides, overlays, or some similar mechanism
  • ...other...?

Pending questions:

  • What makes a plugin system a plugin system? (I.e. what is the scope of this?)
  • how does #44196 relate?
@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Apr 12, 2019

If a desirable solution already exists, then other plugin systems listed in the (TODO) aggregation should be migrated to it.

@deliciouslytyped deliciouslytyped changed the title Proposal: Engineering a Common Plugin Infrastructure Proposal: Engineering a Common "Plugin" Infrastructure Apr 12, 2019

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Apr 12, 2019

This post: syntax/semantics/ergonomics

I quite like the rootPackage.withPackages (p: with p; [ a b c ]) pattern.

Under what circumstances should it be considered meaningful to include non-child packages of the rootPackage in the list? Are there conditions where it's useful to pass in external packages? Say, something like python37.withPackages (p: [ pkgs.hello ]), or should the intended result of such a thing be achieved some other way?
Edit: a use case may be hybrid environments like interfacing between Python and R?

This should not be the same as accepting out-of-tree plugins; I think it would be reasonable to accept something like functions in the plugin list, where during evaluation the root package is applied to the plugins if necessary, and passed on to another builder (or some similar approach).
This is done in my current implementation of ghidra plugins because they need access to a build script located in the root package. (TODO substantiating example)

Alternatively, external overrides should be applied strictly through some pattern like the following:
(rootPackage.overridePackages { d = parent: ...; }).withPackages (p: with p; [ a b c d ])
The parent: parameter may lead to infinite recursion with improper use? (This is the kind of thing that might be useful to mention in API docs, if it's not possible to reasonably prevent altogether)
Though I find the parenthesization a bit awkward.

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Apr 12, 2019

Per suggestions I have attempted to explore makeScope/newScope/... , but I haven't been able to understand them yet enough to get them to work right. My current understanding is that would at least cover my minimum desired functionality for providing overrides.

@joepie91

This comment has been minimized.

Copy link
Contributor

commented Apr 16, 2019

Some quick additional notes before I forget, though I don't have much time to elaborate right now:

  • Mods for games are also plugins, not sure whether those require special handling.
  • There's currently a disconnect in cases where plugins can be managed from within the application as well as from Nix configuration; there should probably be some sort of way to reconcile that, though it may involve extra tooling.
  • PlatformIO orchestrates a bunch of different tools for interacting with different embedded devices and architectures; it currently downloads those on runtime. Not sure if that would fall under the conceptual meaning of 'plugins' here.
@eighal

This comment has been minimized.

Copy link

commented Apr 16, 2019

Packages managed by the package managers of (at least interpreted) languages, like Ruby gems, are also plugins, right?

@joepie91

This comment has been minimized.

Copy link
Contributor

commented Apr 16, 2019

More: Jekyll has plugins, if I recall correctly.

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Apr 18, 2019

@joepie91
To address your points in some order:

2 )
The application managed plugins problem is something that has bothered me for a while and I'm not sure what to do about it.

My knee-jerk reaction is that applications need to support application-external plugin/setting management if we don't want strange things to happen? The reason I mention settings management, is because that's usually how plugins get added to the "current set of running plugins", right? My mental model is currently based off of firefox - for which I'm not sure of the details (but there's this mess of notes rycee/home-manager#606 ) - which is to say, I think (?) I'm agreeing with you.

1 )
I don't see yet why game mods are special? I'm imagining that specific installation functionality would be handled by domain specific handlers anyway. Application specific hooks.

3 )
I will talk to you about this on IRC and update this post.

@eighal yeah I'm inclined to think so. My current heuristic for "plugins" is whether there is some central thing that needs to have plugins fed back into it, that being the interpreter/'s package environment.
But maybe I'm going about this from the wrong direction. Instead of thinking about plugins, maybe we should be thinking about "things that have a domain specific set of 'sub'packages, that must be fed-back the packages"? I wouldn't be surprised if this basically reduces to the thing going on with the fixpoint stuff.

Misc:
added ruby and jekyll to tentative list.

Edit:
Just a subpackageset is easy to make, it gets hairy with this fixpoint stuff which is needed for overrides.

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Apr 18, 2019

TL;DR: if it has something that looks like a .withPackages (p: ...), or looks like it would want a .withPackages (p: ...) then at least that is relevant.

@aanderse

This comment has been minimized.

Copy link
Contributor

commented Apr 18, 2019

The redmine service has plugins and themes. This is currently not implemented well.

@coreyoconnor

This comment has been minimized.

Copy link
Contributor

commented Apr 20, 2019

Projects with internal package managers for handling plugins I've worked with: jenkins, retroarch, gnome-shell. These all can download their own plugins/extensions which, sometimes, works. Other times there are requirements to the .so/.jar/.js that are not part of the provided environment. Hence: those need to be built via nix and, preferably, block the installation of built artifacts dynamically.

As @deliciouslytyped mentions: applications themselves need some consideration to prevent bad things happening.

EG: in retroarch an update for bundled core can be requested but will fail with an odd error as that .so is RO.

@coreyoconnor

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2019

I noticed the question mark next to jenkins and decided to dig in a bit mroe.

  • The relevant code is at nixos/modules/services/continuous-integration/jenkins/default.nix
  • the tool referenced jenkinsPlugins2nix does not (afaict) build the plugin. Only packages a pre-built plugin.
  • only runtime dependencies would impact actually using the plugin
  • jenkins itself is not built by nix but only packaged

I agree this puts jenkins in a different group than the above.

@joepie91

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2019

Addition: KDE has a plugin manager as well, for "plasma widgets" and such, I believe.

Edit: and Thunar has file manager plugins too, though no interactive manager I think - the Thunar package in nixpkgs also already has a plugins option.

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Apr 21, 2019

@coreyoconnor The reason I "separated" it was because on a cursory look I didn't see an implementation of handling plugins - hence: "No implementation". If it had an implementation I would have listed it under the rest. (This is just a comment on that and not really addressing anything else.)

Edit: which is to say, it has plugins, there is just no implementation of handling them in nixpkgs. Though now that I think about it, the list might get really long if we start considering everything everywhere that takes plugins. :P - which doesn't mean that I think we shouldn't consider everything. However, I guess after the first 20 or 30 examples it will be time to start trying to figure out archetypes.

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Apr 22, 2019

I'm currently poking around overlays and...that stuff. Still trying to figure out how it works.
I spent the day prototyping stuff with a type system but I'm not quite at the goal of the previously mentioned syntax I want yet...

https://www.youtube.com/watch?v=W85mF1zWA2o (@nbp NixCon talk on overlays) is helpful.

@deliciouslytyped deliciouslytyped changed the title Proposal: Engineering a Common "Plugin" Infrastructure [RFC] Proposal: Engineering a Common "Plugin" Infrastructure Apr 24, 2019

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Apr 24, 2019

I have a preliminary system for Ghidra I will try upstreaming soon, otherwise I'm still exploring the problem space.

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Apr 24, 2019

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Apr 28, 2019

Possibly related NixOS/rfcs#3

@aanderse

This comment has been minimized.

Copy link
Contributor

commented May 3, 2019

Just noticed another item which might be of interest. squirrel-sql has database drivers that can be considered plugins. See nixpkgs/pkgs/development/tools/database/squirrel-sql/default.nix for details.

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented May 4, 2019

Misc wishlist: It would be nice if enableDebugging worked properly across package sets (where appropriate).

Edit:
Possible mechanism: enableDebugging should check if the target package has an enableDebugging attribute, if yes it should call it - this should be nailed down as an api convention.

Currently what I'm doing for plugin sets is merging a .pkgs attribute into to the package sets base overlay and merging it into .callPackage's scope. So maybe functions that rely on overriding something should also override it in a pkgs passed to these, but that sounds kind of messy.

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented May 14, 2019

Forgot to link my WIP POC: #60664

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented May 22, 2019

I turned it into a WIP library https://github.com/deliciouslytyped/nix-rootedoverlay/ , it's pretty broken right now. Help wanted.

Edit: I think I fixed stuff but I'm still trying to write tests.

@aanderse

This comment has been minimized.

Copy link
Contributor

commented Jun 22, 2019

As a new data point I found that moodle plugins are scattered all throughout the directory structure of the package. Typically plugins for an application should go under 1 or maybe 2 different directories (say plugins/ and/or themes/, for example), but with moodle they go all over the place, depending on what type of plugin you're looking at.

For example plugins classified as Enrollment plugins go under enrol/, while those classified as Blocks go under blocks/, etc... At this time it appears moodle supports upwards to 50 different types of plugins, each with a different folder to be installed under.

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Jun 25, 2019

Where can I find the moodle stuff?

From the perspective of my WIP rootedoverlay stuff:
Not sure which of the two you're trying to point out here? - I'm assuming the latter:

  • From a packaging standpoint personally I'd just shove everything in a top level plugins directory, and the top level plugin list can handle exposing things to the user
  • From the standpoint of "applying" plugins, I think this would still fit into a supplying a custom Root -> Plugin -> Root function, such that this function can handle a plugin that specifies it's installation details.

Does that sound reasonable?
Edit: rewording

@aanderse

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2019

@deliciouslytyped I was mostly just providing another data point... but yes, the second point sounds good 😄

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Jun 25, 2019

All good, thanks :D

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Jul 4, 2019

I'm currently working on a POC for the R package set.

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Jul 5, 2019

There's a somewhat broken implementation at https://github.com/deliciouslytyped/nix-rer/ currently.
The major bugs off the top of my head are

  • propagated inputs aren't handled correctly (I'm not sure how the original implementation dealt with this) so surrogate.cran.curl doesn't build if it's the dependency of something, but it builds by itself.
  • some packages are not found both because I don't have a way to search the global namespace yet, and I haven't added the bioc package sets.
    Also the codebase is still quite a mess.

but it seems to basically work.

Edit:
Oneliner example:

nix-shell -E "(import ./. {}).surrogate.withPackages (p: [ p.cran.purrr ])" -v --run "R --quiet -e 'library(purrr); map(list(1,2,3), ~(..1+1))'"
@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Jul 17, 2019

@barakber I'm thinking of writing a library set for Prolog. I'm not yet familiar with how prolog libraries work, can you maybe explain a bit how

packInstall = swiplPath: pack:
is intended to work? Thanks!

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Jul 17, 2019

I'm currently working on porting the Idris infrastructure.

@barakber

This comment has been minimized.

Copy link
Contributor

commented Jul 17, 2019

Sure. packInstall is a Nix function that is using https://www.swi-prolog.org/pldoc/man?section=prologpack from the shell, to install a list of "packs" (from https://www.swi-prolog.org/pack/list) during the installation phase of the SWI-prolog Nix derivation.
The goal is really to have a "swipl.withPackages" that does all pack_install()s for you.

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2019

@deliciouslytyped

This comment has been minimized.

Copy link
Contributor Author

commented Aug 11, 2019

It seems like a good idea to read through and steal some ideas from https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md , there appear to have been developments since I last read it. (edit: it's more likely that I just didn't notice a bunch of things...)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.