Initial stab at extensible subsystems

[jaen]

Hi, I come bearing actual discussion material regarding allowing dream2nix to be extended from outside.
This is the least amount of changes that allowed me to specify new discoverers, translators, fetchers and builders using the config attribute. Right now this is pretty un-fancy – for example it would be tedious to combine multiple extensions together, as you would have to manually merge attrsets together. An overlay-based mechanism, with each extension providing an overlay adding language-subsystem-specific functionality would probably make more sense, but wanted to keep it as simple as possible and have really no idea how much changes would be required to accommodate that.

Now, I don't quite understand what I'm doing here, so some changes might've been unnecessary or better accomplished another way – for example I had to push down the dlib like that and thread config through more places, because otherwise I ended up with translators that alternatively had the specified config or not, probably closing over different things – but hopefully at least better explains what I thought dream2nix being extensible might be useful and could be a starting port for a discussion.

Here's a repo using it to get a barebones Ruby subystem running to show it actually works:
https://github.com/jaen/iwai

[DavHau]

Some thoughts:

• Currently still a lot of things are changing in dream2nix. Therefore I cannot guarantee a stable interface for external extensions.
• Maybe better have an argument extensions for dream2nix.lib.init, like:

  dream2nix.lib.init {
    extensions.translators.ruby.pure.bundler-impure = ...;
    extensions.builders.ruby = ...;
  }

• Currently the structure of translators and builders is different in dream2nix. For translator the translator name is derived from the file name of the nix file and for builders the names are defined in builders/default.nix. We should probably unify that structure to be the same for every module kind.

[jaen]

Currently still a lot of things are changing in dream2nix. Therefore I cannot guarantee a stable interface for external extensions.

Right. Though from my PoV it would be better to already offer some extensibility, even if the interface won't be stable and will change in the future – it's not a big deal for the consumer to pin a dependency with flakes and update it whenever. Also if a potential new language subsystems would be in-tree from the start, then you would have more churn when refactoring – if the subsystem is external, then you offload the churn to early adopters.

Maybe better have an argument extensions for dream2nix.lib.init, like:

The attribute is already in init, though? Unless you're just bikeshedding for it to be under a different attribute from config. Which, sure, I'm not married to, naming it extensions would certainly be more self-explanatory as well.

Currently the structure of translators and builders is different in dream2nix. For translator the translator name is derived from the file name of the nix file and for builders the names are defined in builders/default.nix. We should probably unify that structure to be the same for every module kind.

Yeah, I've noticed, it wasn't really an issue for me though, since I was providing everything as a data structure, so I had to name everything explicitly anyway. But if we're doing this for real, then probably something more unified with how this is set up in dream2nix proper would make sense (and unifying it internally, as you suggest) – for example allowing to specify an extension directory and have dream2nix discover it in an unified manner to the internal subsystems (or optionally, still as a data structure?).

Incidentally, if we're talking about unification of how it is structured in dream2nix, then how about slicing it vertically instead of horizontally (if you're unfamiliar with the distinction then this seems like a simple enough overview)?

As in, instead of having discoverers/ruby, translators/ruby (and so on), then have something like subsystems/ruby/discoverer.nix, subsystems/ruby/translator.nix (and so on) and a unifying subsystems/ruby/default.nix. Never the mind that to me a feature-based grouping makes more sense than implementation-detail-based one, I think it would make it easier to make sure the importing is uniform and make it easier to extend to external subsystems.

Orthogonally, another thing I was mulling about is that what I've done here is a quick ad-hoc hack that doesn't look like it would compose well and maybe an overlay-based approach would make more sense? Additionally, it could make it possible for external libraries to extend and modify other subsystems with additional functionality.

For example the above example of extending with a Ruby subsystem could look like this instead:

# The language ecosystem library exports an overlay like this:
rubyOverlay = (final: prev
  {
    # Not terribly sure how exactly should this play along with `callDreamPkg`, but I probably
    # each leaf should be a callable compatible with it? I'm not quite sure if that's the case
    # for all the things, I think discoverers at least seem to not be called with it?
    discoverers = prev.discoverers // { ruby = rubyDiscoverer; };
    # other subsystem elements…
  })

# And an user project's just uses the overlay
dream2nix.lib.init {
  extensions = [
    rubyOverlay
  ];
}

# An overlay could even extend existing subsystems with new functionality
magicCompanyVpnOverlay = (final: prev
  {
    fetchers = lib.mapAttrsRecursive wrapFetcherWithVpnAccess prev.fetchers;
  })

# And now all fetches go through your paranoid company's VPN
dream2nix.lib.init {
  extensions = [
    magicCompanyVpnOverlay
  ];
}

and if we decided to use NixOS module system for that, then it could probably be simplified even more, because we could make subsystems element to merge sensibly by default and have all the priority/force/etc helpers of modules.

If those two proposals sounds sensibly to you, I could try to hack together a working example (hopefully, developing non-trivial stuff in the nix language is a bit painful).

Sorry if this is disjointed or hard to follow, feel free to ask for any clarifications.

[jaen]

Here's the module version – #153

[jaen]

Closing in favour of #155