dynamic modules

[dominictarr]

sometimes it seems necessary to have a dependency you didn't know that you had, or have something like a router, which different add handlers to.

I showed depject to @nkrn and he tried it and wrote this about his experience https://github.com/nrkn/classical-vs-depject/blob/master/readme.md

under "Cons" he mentions:

"I can't figure out how once you've called combine you can then plug more modules into your sockets object post the fact - this is probably not so much a con as a lack of comprehension on my part!"

So, opening this issue as a place to discuss this question. I have had this problem recently, although I wasn't using depject at the time. (I was using secret-stack's [crappy] plugin system [which led to depject])

Here, two "plugin" apis collide. flume takes a simple interface and then wraps it a bit (taking database views and adding a thing to delay the request to the view to ensure read consistency) but the secret-stack plugins are supposed to export the apis that it creates. I can't have a flume sbot plugin that doesn't know what it's gonna export, and other plugins that know what they want to export but expect that to be exported by another thing.

But then i figured out that I didn't actually need it!
https://github.com/ssbc/scuttlebot/blob/flume/plugins/friends.js#L35

So what I did, was just have the flume method return the wrapped view, _flumeUse (not how you'd normally use flume) and then the plugin (in this case, friends, which shows who follows who, etc) just exports it's own api, which it happens to initialize through another plugin.

The only limitation here is that the since the friends plugin has to call another plugin during initialization (create method in depject) that you can't have cyclic dependencies (but you should avoid that anyway!)

I'm sure there are other ways you might need dynamic modules though, this is just one example.

[dominictarr]

oh I could also imagine applications where you want to load plugins via the ui, dynamically, like new kinds of image manipulation scripts. that really would require a dynamic combine - this is probably possible though?

[ahdinosaur]

@dominictarr what if we iterate on your idea from #17:

var App = require('depject')

// static modules
var app = App([
  {
    needs: {decorate: 'reduce'},
    gives: 'hello',
    create: function (sockets) {
      return function (name) {
        return sockets.decorate(name)
      }
    }
  },
  {
    gives: 'decorate',
    create: function () {
      return function (name) {
        return 'Hello, ' + name
      }
    }
  }
])

// dynamic module
app.use({
  gives: 'decorate',
  create: function () {
    return function (name) {
      return name.toUpperCase()
    }
  }
})

// ...

and somehow have a way to "start" the app (using something like #20, but needs to observe when modules are added).

[nrkn]

You can maybe do something like this? It's backwards compatible and the tests still pass, but I've only tried it on a trivial example (it worked) and I've really got a lot of work to get on with so I can't write any further tests to ensure it's robust just now:

module.exports = combine

function combine () {
  return combineInto({}).apply(this, arguments)
}

combine.dynamic = combineInto

function combineInto (combinedModules) {
  return function addModules () {
    var nestedModules = Array.prototype.slice.call(arguments)

    var modules = flattenNested(nestedModules)

    assertDependencies(modules)

    for (var key in modules) {
      var module = modules[key]
      var needed = getNeeded(combinedModules, module.needs)
      var given = module.create(needed)

      assertGiven(module.gives, given, key)

      addGivenToCombined(given, combinedModules, module)
    }

    if (isEmpty(combinedModules)) {
      throw new Error('could not resolve any modules')
    }

    return combinedModules
  }
}

And then to use it (I modified the hello example):

var combine = require('../')

var hi = {
  needs: {decorate: 'reduce'},
  gives: 'hello',
  create: function (sockets) {
    return function (name) {
      return sockets.decorate(name)
    }
  }
}

var capitalize = {
  gives: 'decorate',
  create: function () {
    return function (name) {
      return name.toUpperCase()
    }
  }
}

var greet = {
  gives: 'decorate',
  create: function () {
    return function (name) {
      return 'Hello, ' + name
    }
  }
}

var insult = {
  gives: 'decorate',
  create: function () {
    return function (name) {
      return name + ', you pillock.'
    }
  }
}

module.exports = {
  hi: hi, capitalize: capitalize, greet: greet
}

if (!module.parent) {
  var sockets = combine([hi, capitalize, greet])

  console.log(sockets.hello[0]('dominic'))

  var extended = combine.dynamic(sockets)([insult])

  console.log(extended.hello[0]('nik'))
}

[nrkn]

It's easy enough to just keep the array of modules that you originally passed to combine, modify that when you need extra modules and just call combine again.

If the use case came up enough and there were a lot of modules involved it would be worth doing for performance gains, but otherwise it's a simple work around.

[dominictarr]

@nrkn when you get the time, could you describe what you used this for? it helps a great deal to know something about how something is actually being used, especially when it comes time to weigh up competing merits of various proposals.

[nrkn]

@dominictarr do you mean what I used the above code for, or what I use post-plugins in general for?

If a) I didn't use it for anything, was just an idea/proof of concept

For b) it's a workflow thing - I build pretty much every with the adapter/common core/plugin structure and I often use post-plugins for when I've got a package with a pre-baked interface (like a convenience API that's friendly for consumers rather than making them deal with adapters/plugins etc.) and I want to just install it and prototype new features, add things that don't belong in the core on a single use basis, add a plugin that wraps an existing function for logging etc. etc. - so these convenience apis usually have a .plugin( plugins ) function to enable that

You could still do that with depject without making it support dynamic plugins explicitly because you can just keep track of all the modules that you're putting into depject and when a plugin is added you add it to the list of modules and generate a whole new sockets instance from your list of modules each time a plugin is added. It feels a bit wasteful, but would work.

[dominictarr]

I mean, like, how does this touch the real world? did you use it for a CMS? or ecommerce or blog? what did the plugins do and when did you change them? When something is explained abstractly, it's way easier to think that I understand you, but if you add some real world context I can double check my understanding with what would make sense against my understanding of the real world.

Like imagine if you asked me how to make wood waterproof - I might assume you where building a boat - but maybe you are actually building a dog-house. I want context

[nrkn]

Primarily for a website builder/CMS. But I reuse a lot of the code for various personal projects as well, I mean, I have a game that I've been dicking around with that uses a lot of the same packages and patterns, even though the problem domain is different, I need to work with trees and graphs and I need it to run in the browser, so huge overlap.

Looking through my code for specifics to answer your question, it looks like I'm primarily (almost solely in fact) using it to build custom tree types (a dom tree, a json object tree, a schema tree) on top of my base tree api. Here it is in use in the dom tree:

https://github.com/mojule/mojule-dom/blob/master/src/index.js

It's to control the order in which functions are wrapped. The base tree has some functions that in almost every case need to wrap user-provided plugins - hence, the plugins passed to TreeFactory by the user are added before the important-need-to-wrap-everything-else functions. But then sometimes, and it's an edge case but one that comes up from time to time, I know that I need to wrap those important-functions-that-normally-wrap-everything-else:

const DomTree = TreeFactory( dom, parse, select, stringify, types, vdom )

// add afterwards because the original createTree doesn't exist until now
DomTree.plugin( createTree )
// add afterwards so that it wraps parentMap and not the other way round
DomTree.plugin( insertBefore )

createTree above is one of the important-etc. functions, and I'm post-wrapping it via a plugin to make it take some extra argument types specific to working with a dom vs a generic tree.

Post plugins of this kind would be useless in depject for first, but I think they'd be handy for reduce.

It's hard to say without actually taking the plunge and moving these things to depject.