Edgeconf Takeaways & Duo's place

[matthewmueller]

Edgeconf Takeaways

Just wanted to give you an update on my thinking around Duo and it's place. I had the chance to meet most of the other main package manager maintainers yesterday at Edgeconf. Here were some takeaways:

NPM wants to become the one "package manager to rule them all"

It seems like a lot of people support this idea that the client-side should be handled by NPM. NPM is interested in supporting this eventually, but their priority is business sustainability (with enterprise stability). Addressing the concerns of how to manage CSS and other non-JS assets is potentially on their TODO.

I strongly disagree with the premise that it's a given that NPM should be the one main package manager. I'm becoming increasingly convinced that there should be a clear separation between Node and the client. To me I have to ask, where does it no longer make sense? Should NPM be installing my Mac Apps? What about my Linux Packages? I feel like NPM can rule the node world, but we need a client-side package manager that specializes in the asset pipeline, which can be interoperable with other programming languages (like ruby, go, python, etc.)

Bower and Duo should forge a closer relationship

Bower really surprised me. I thought it was a vote by committee project backed by Twitter, Google, etc. It's pretty much running itself by a few devoted contributors on cheap heroku software. What's weird to me is it seems like the big weights are behind it for the most part (ex. Google's Polymer project seems to be backing it), but they're not contributing much financially or with dedicated engineers. This might change in the future.

Bower's registry is optional, it just points to HTTP URLs

The registry is merely there for discoverability and ease of typing. It's only slightly more than a key-value store in the cloud. You can install from any HTTP endpoint though just with bower install <url>. Github uses bower to install, sprockets to build.

---

Duo's Place

It seems to me that Duo's place is to make it as easy as possible to consume and use code. Not a lot of people agree with the premise that the installer and builder should be packaged together (like Component & Duo). I believe when these are put together in the right way with the right hooks, they greatly compliment each other. I also don't believe that a lot of the "big names" in frontend dev have given Component or Duo a fair shot. The people most interested in Duo and Component were the attendees, not so much the panelists.

The other thing to note is that Bower or NPM don't work standalone, they require additional libraries to glue everything together. This may not matter when HTTP2 lands, but right now concatenation is essential. Even then, I think a build step will be necessary for non-JS/CSS/Template compiling.

I think if we want Duo to have a place in this ecosystem, we need to focus our efforts in the following areas:

Include from anywhere (literally anywhere)

Duo-package should be able to be given a URI and it should be able to take it from there. If it's an HTTP remote to a JS or CSS file, install and bundle that. If it's a tarball, unzip and look for a main in either a component.json, bower.json, or package.json. It should have hooks for new providers like bitbucket, or private remote git servers.

For dependencies, I think we're going to want to make calls to other libraries to resolve. For bower and NPM, curl the registry. If you think about it, this is no different than what we do now with git-ls-remote

Allow for a way to run $ duo install ...

I think everyone has been wanting this, but no one has made it happen yet. I think this is actually a really easy fix. Move most the duo#run logic into duo#install and then have duo#run call duo#install.

Hooks for everything

Nearly every piece of the system should be swappable. From the fetch process, to the parsing, to the resolving, to the packing. People will feel more comfortable using Duo knowing they can customize it at any point.

Support implicit transforms, instruct Duo users when packages need additional compilers.

Some die-hard Component devs might cringe, but I think there's benefit in creating higher-level plugins that get compiled down at build.

So how this could work is:

1. plugin authors write their CSS component in SASS.
2. The whole repo gets downloaded by Duo
3. Duo reaches a .sass file and internally looks at a map of extensions.
4. If there's a extname match, either compile or instruct the user to download the given compiler.

If you notice, this workflow is very similar to how https://github.com/visionmedia/consolidate.js works.

After talking with @josh more about this, it seems like Bower might be interested in this feature as well. So I think we could potentially collaborate on a plugin that works for both package managers.

---

/cc @yields @ianstormtaylor @stephenmathieson @stephenway @dominicbarnes @lancejpollard @visionmedia

[howardroark]

I am so happy to have read this!

The browser will increasingly interact with services via smart interfaces that deliver JSON. This means that the code which runs in the browser and that which runs on the server have less and less need to be a single project. We can't forget that things like Python exist and there is so much amazing (non-nodejs) code out there for building APIs.

We need smart solutions that favour building beautiful interfaces!

We don't need central package authorities any more and I am delighted to see that this project could be an opportunity to move that direction. In the end a package is nothing more than a set of files with one that says what to do with it. Packages should be able to be stored anywhere, and right now GitHub is where they all are anyway!

[josh]

@matthewmueller hey, it was great finally meeting you. And great observations in this writeup.

Not a lot of people agree with the premise that the installer and builder should be packaged together (like Component & Duo).

I think this is a totally awesome and great opinion for duo to have. If we're in a position where the manifests and registries are clearly specified, theres room for many implementations with other opinions on how the user tooling should work.

Some sort of community effort to spec this details clarifies how "Include from anywhere (literally anywhere)" should work. Though, its definitely not a blocker. This is also key to avoiding the centralized authority that just doesn't feel like the web we love.

Bower and Duo should forge a closer relationship

Given the current situation we do have a separate bower.json that duo would need to support. That may change in the future, but at least for now bower should be clear about how its manifest registry apis should work. I have some other ideas to attempt to simply that part of the stack.

[matthewmueller]

@josh great meeting you too. I thought we had some really productive conversations.

Given the current situation we do have a separate bower.json that duo would need to support. That may change in the future, but at least for now bower should be clear about how its manifest registry apis should work. I have some other ideas to attempt to simply that part of the stack.

+1, adding full bower.json support is on the roadmap. We'll need to abstract out the component-specific stuff and make the manifest implementations pluggable.

[ismay]

I was wondering, without the manifest, will duo install ... just install a component and then forget about it? Or will the installation of a component that isn't referenced in css or js somehow be remembered?

[sankargorthi]

@ismay Well, that's how bower works.

[ismay]

@sankargorthi

You mean that bower can --save dependencies to bower.json? Because I know that. What I was wondering is whether and how duo install will be storing dependencies. I think that you've misinterpreted my question to pertain to the bower related discussion in this thread (which is not what duo install will be using, right?).

[sankargorthi]

Afaict that's all it's meant to do. In the absence of component's dev dependency manifest, this would help install dev dependencies.

[ismay]

@sankargorthi

Because I've understood this:

Allow for a way to run $ duo install ...
I think everyone has been wanting this, but no one has made it happen yet. I think this is actually a really easy fix. Move most the duo#run logic into duo#install and then have duo#run call duo#install.

to mean something else (i.e.: custom duo install logic). But if you're right, then why wrap bower install with a duo install? I'm not sure that that is what @matthewmueller meant.

[stephenmathieson]

Support implicit transforms

I thought about proposing this earlier. I decided against it because it enforces users to use the "official" foo to bar transform. Users don't like this, which is why there are two sass plugins already (duosass and duo-sass).

Outside of that, I'd love to not have to manually include duo-jade in every one of my builds... haha

off topic: @tetsuo and @stephenway you two should really merge your efforts...

[bmcmahen]

The ability to include from anywhere would be really nice -- especially if bower and package.json were supported. In my experience, fragmentation is the primary headache when dealing with packages. When choosing a package manager, you are not just choosing a tool, you are choosing an entire ecosystem of modules. In effect, I chose component back in the day primarily because of the high quality components written by @visionmedia @matthewmueller and @yields (among many others). Decoupling the manager/builder from a narrow ecosystem of modules would be really, really cool...

Having an install prompt would be nice. But a question:

what role will component.json have in the project going forward?

My sense is that this already confuses people. If we install --save something (from any path ) should it go into a component.json? Should component.json support other, non-github, non-component paths? Should newly created duo modules have a component.json in the root?

[stephenway]

@stephenmathieson I agree, that would be nice.

[howardroark]

It seems that no matter what you choose to focus on will need to evolve... be it bower.json, package.json or component.json. I think the real question is which community is most likely to gravitate to the idea behind duo and push it forward. My gut is the community around component.json is this.

Then you can start deciding things like... If we could pull packages from any Git remote, how would that look? Trying to support everything out there may just leave you in a tug-of-war.

IMO you should only need to "install" things that are to be available in your bin path.

If the desire is to better enable duo to support things like Zurb's foundation... I don't think that is right. Duo should encourage CommonJS and the Unix philosophy. If you want to build a fancy carousel plugin, then separate it into two projects. One project that offers up the JS functionality and maybe injects some content into the DOM, and another that offers CSS and assets. This way you are forced to decouple the two and better offer the ability to invent your own CSS... rather than trying to awkwardly build over top of it. This way once you run duo it can get everything you need on the spot.

You should expect new habits to be formed.

[ismay]

I do wonder whether going with a component.json (or bower.json, or any other kind of manifest) will be the best solution though. Duo's main innovation is that it doesn't need a separate manifest.

I'm already using bower to fetch remote dependencies that I can't require in my js or import in my css and don't see the reason to duplicate bower's or component's functionality for that. I'm only saying this from my own perspective (as an end user), but I'd feel it would be far more elegant to keep going without a manifest.

[matthewmueller]

@ismay the manifest will always remain optional. this is about being able to interface with existing packages from other package managers.

[tj]

Yeah I agree, regardless of intent npm is pretty far from being ubiquitous or anything close to it. Even if it was on that path, that's not something I want to see, I don't see many (any?) benefits from that personally. Especially having not touched npm for a few months now I can easily say that I'm much more productive without it.

Bower I don't have much to comment on since I haven't used it much, but I haven't noticed anything appealing about it. Not sure I even like the idea of introducing a centralized namespace thingy again, but if duo was abstract enough it could support an arbitrary number of similar things where people collected "canonical" paackages like jquery etc without typing jquery/jquery. I'm impartial there, the primary thing for me is good discoverability, not so much the typing aspect.

Build-time compiles would be on the installer's end? Not sure how I feel about that, it would couple to node like crazy, but with pipelines being less necessary in the future I guess it doesn't realllllly matter. Always the headaches of dealing with broken local shit, I guess that's where I like the idea of delegating as much as you can to the author in some sense, you get a single view of the build vs whatever happens to happen on the client with semver madness etc.

I still haven't tried Jonathan's normalize stuff either but part of me really loves the idea of not having to deal with any stuff locally. At most maybe describing that you want your project to use sass 1.x or 2.x and everything else just happens. I haven't written any client-side in ages now but even the idea of setting up manual stuff to get myth working etc is daunting haha.

Going to hack on my first Duo project right now :D

[matthewmueller]

Not sure I even like the idea of introducing a centralized namespace thingy again, but if duo was abstract enough it could support an arbitrary number of similar things where people collected "canonical" paackages like jquery etc without typing jquery/jquery. I'm impartial there, the primary thing for me is good discoverability, not so much the typing aspect.

Yah, I think we could do something like require('bower:jquery') to support their registry

Always the headaches of dealing with broken local shit, I guess that's where I like the idea of delegating as much as you can to the author in some sense

This is a good point, I'm just wondering if we should be giving authors opportunities for higher-level plugins. But broken local shit sucks, completely agree :-D

I still haven't tried Jonathan's normalize stuff either but part of me really loves the idea of not having to deal with any stuff locally

Yah that'll be sweet, you'd probably still need to download it when you deploy, but

Going to hack on my first Duo project right now :D

Awesome! Let me know how it goes. There's still a lot to do with it, just haven't had enough time yet. Even moreso, need to have more documentation/videos to show people how great the workflow is for new apps.

[jasonkuhrt]

@ everyone, required reading: http://blog.npmjs.org/post/101775448305/npm-and-front-end-packaging

Thoughts?

Seems to me that the idea here is basically all package managers' days are numbered. You may disagree, but the current situation obviously will not last (half-a-dozen managers) so something does have to give sooner or later.

[matthewmueller]

Yah, I've been aware of most of these new features and I have no idea how this is going to play out.

What I do know is that it's in their interest as a business to be the package manager for everything. Browserify is their solution to client-side development, which leaves CSS and the rest of the assets scratching their heads.

I see Duo being able to support NPM and Bower packages before NPM gets around to the client-side. At the end of the day though, it's a decision of what tool works best for the job at hand. That's not clear yet.

[tj]

My confidence in npm for anything is relatively low, when you've worked without it for quite a while (Go etc) it becomes pretty clear how awesome it is not having a centralized package manager. Pros and cons of course but npm is pretty far from being the obvious choice, maybe some day.

[jasonkuhrt]

@tj depends on what your usual assortment of libraries you want to use are though. If your really want to use Immutable or React or Kefir or bluebird then you're going to want simple interop and lightweight builds e.g. gluing a whole of bunch of dist.js files is going to loose the benefit of sharing deps and thus we become beholden to what other projects are doing. Seems like bower.json and package.json + browserify are coming options offered by projects but I don't see how duo can as-is just jump in there and create optimized builds. I think it could and I hope to help it do that but its going to require that, most likely, duo become really good at working with package.json and modules bias to npm; too much engineering effort has gone into that sphere to disregard.

[jasonkuhrt]

And if duo takes the approach that any library over ~500 LOC is too heavy weight than that's another path to irrelevance.

[bmcmahen]

My general predicament right now is that most of the libraries I want to use tend to only support browserify (basically any React lib uses browserify)... but when it comes to anything that requires assets (css, mostly) it becomes a complete mess. If duo could support package.json and resolve npm dependencies, I would be a very happy man.

[tj]

Yeah that's the thing, npm sort of wins via critical mass, but that doesn't mean it's the right solution, or an enjoyable/robust one.

[howardroark]

My gut is that browserify was the first solid solution to get away from the AMD model and that caused a rush of people to go that route. If Duo had been around before browserify then I think we would be talking less about package.json. Like @bmcmahen pointed out... browserify does nothing to help with CSS. You end up having to use a browserify/bower hybrid combo, which is not fun. I don't think the critical mass has gone any which way yet. I do think they will go the direction that makes the most sense for building "web apps" and "cordova apps". That is applications that have no server-side at all, or if they do it is some sort of remote storage cluster or 3rd party API.

I'd love to see a trend of people separating their front-end projects into Javascript and CSS repos. Why bundle it all together in the first place, things would be much more reusable that way.

If Duo wants success it should give future "web app" developers what they need. Stop thinking about the Web 2.0/SQL approach.

[jasonkuhrt]

Great discussion here webpack/webpack#378 and one fallout leading to Webpack 2: https://github.com/webpack/concord

The stories outlined above do seem compelling indeed.

[tj]

There are definitely cases where you want js/css/fonts etc, many IMO. This will be even more true moving into Web Component land.

I can't see any of this stuff working out long-term until we can get packages that can be as abstract as a tarball and work well for all cases, hopefully Web Components helps drive towards that, any special-casing at the PM level is required for now but it'll make less sense in the future

Go's stuff is a good example of that, there are no manifests, anything in a tarball could work just fine assuming it has .go files. Eventually client-side will be similar (hopefully).

[howardroark]

@tj It is all very true. What about the idea of applying an object style schema to a set of files after the fact. You grab your tarball from where you choose and after extraction you tell something like duo to use an object to deal with the files accordingly. The object decides what to do with the web component or node module.

In my head there are two things duo needs to deal with... Pure JavaScript "modules" and full featured web components. In regards to browserify... I would love to see this project forge way for the creator/project scheme. I think it just makes so much human sense that way. Why should only one source have dibs on something like "zip".

Web components sort of end up with the need to be loose. That is where it gets fuzzy. So much stuff out there done in so many ways. There are patterns to all of it though, that could be applied after the fact.

[howardroark]

@matthewmueller I just had a thought related to my last comment. What if duo offered a service that let people submit JSON schemas that pair with GitHub "creator/project" paths. This way a third party could submit an JSON object (hosted on GitHub) which helps duo use an existing web component elsewhere on GitHub. With OAUTH you could make it simple for the original owner to "take it over".

If it has no object available, just assume it is a "module" and use it the way browserify would. 99% of npm modules are on GitHub anyway. Heck I would support standard node "require" methods by default just to ease the burden of phasing them out to the "creator/project" schema.

Just a thought (gotta think big!)...

[jasonkuhrt]

@tj @howardroark I'm assuming your thoughts aren't in response to my comment, I don't see how they address it at all. The links, if you read them, long as they are, point to a few things:

• its not about package management
• its about builds
• that are separated from modules
• and modules
• that are separated from builds
• based on a standard module system
• that different builds obey so that users aren't locked into a single builder
• and may choose whichever package manager they want

-- EDIT

Thus, see Concord, for that matter duo could choose to use it too.

[tj]

@jasonkuhrt nope, just a response to @howardroark's "I'd love to see a trend of people separating their front-end projects into Javascript and CSS repos.". I don't blame package management for the state we're in at all, it seems more like a tooling issue in the browser to serve things up in chunks like we need, seems like we're almost there though!

[tetsuo]

@stephenmathieson You're right
@stephenway Sorry I acted like a little bitch, what is your npm user?

[tetsuo]

Added @kevva as owner to duo-sass, tell me when you publish, I'll put a deprecated link to duojs/sass

[kevva]

@tetsuo, done. Thank you.

[tetsuo]

And I agree with that, compiling a .sass file and spitting it out is not quite enough actually, that introduces more trouble than solution. CSS is one thing but that's another.

But my real concern about duo is require statements. It breaks compatibility with other package managers, browserify and sounds like it is turning into another programming language on its own.

[stephenmathieson]

@tetsuo :)

[stephenway]

Thanks @tetsuo, https://www.npmjs.org/~stephenway

[tetsuo]

@stephenway Done

[johntron]

Gettting back on topic, these's are why I think there's no "best" package manger/builder:

1. Manifests

Working without a manifest, but using one to specify explicit values seems like the right approach. Sometimes manifests have been necessary, but package maintainers don't like supporting all the different manifests. package.json is the most common and obviously doesn't work for Duo, but there are ways we could extend it to do our bidding.

The concord spec suggests namespacing things in package.json (the "concord" field), but even then maintainers will have to maintain separate lists of dependencies - now they're just in a single file instead of many. Maybe we could allow extension of the npm's "dependencies" field like so:

{
   "dependencies": {
      "pkg-without-author-prefix": "1.*",
      "pkg-with-fs-dependency": "*"
   }
   "duo": {
      "deps-map": {
         "pkg-without-author-prefix": "//github/some-author/pkg-without-author-prefix@[inherit]",
         "pkg-with-fs-dependency": "//github/author/pkg-with-fs-dependency@browser"
      }
   }
}

This way, the maintainers only have a single version number, but can support dependencies on different branches, versions, repositories, etc. for the browser.

The webpack guys seem to have put a lot of thought into a generic manifest format (thanks for the reminder @jasonkuhrt). With all the fragmentation in the JS world, allowing developers to replace Duo in the future (because it uses a standard manifest format) may help adoption. Martin Fowler has been harping on "substitutability" lately, and this mentality makes a lot of sense for package managers/builders. Personally, I'm already using Component, but am hesitant to use Duo on our new project because of the lock-in.

2. Extension

Every application I've ever built has required some custom code in our build toolchain. Sometimes it's a defect in one of the plugins, and other times I wanted to incorporate a new idea that didn't have an existing plugin. The more I use build tools, the more I want to write my own. Thank you @matthewmueller for supporting hooks as a candidate solution, but I think we need more control. For instance, order is really important in a build process; using hooks will make it difficult to put one step before or after another.

I like how Component and Duo provide a nice API, but they're too limited. Instead of the CLI, I'd rather start with a build.js (sane defaults) that uses the Duo API, even if every step of the build process is explicit. I already know JavaScript, I have full control, and it's better than the alternative: installing plugins hoping they do exactly what I need, then banging my head when they don't (convention vs configuration). It sucks having a fairly complex build process, then having to convert everything to JS just so I can do something unique. Why not start with the JS, so developers have something to work with down the road? In my mind the right solution is: declarative syntax, straightforward support for plugins and anonymous functions, a bootstrap, and great documentation.

3. Non-JS files

I think these are the application developer's responsibility, so maybe maintainers should just help identify the assets, and give the application developer tools to easily preprocess and put in the correct location. Assuming Duo made it easy for developers to craft a build process for these assets, there's still the problem of resolving paths (i.e. a dust template with: <img src="/some-image-{device}.png">). Concord has an idea with the "modules" field, but I'm not sure this is the right approach - it seems like this should be done at the application level, rather than the package. Since I've been out of the loop lately, maybe there's already an answer to this.

4. Registry

At the start of a project, there's nothing better than being able to run: duo search model, duo install component/model, and so on. Really, this is why I started using a package manager in the first place - building the project was just an added bonus. Apparently, maintaining a registry is a PITA, but Duo needs to support more than Github. I have no ideas on this one, but it's crucial.

[stereokai]

So where does this stand now?