Make bower Github Release Asset aware

[trek]

Apologies if this idea has been floated before. I did a cursory search and couldn't find anything.

The impetus behind this proposal is a long exploration of the painful JavaScript modularity story. I'll save you the read and summarize the most salient points:

1. In Bower the average JavaScript library dependency depth rounds to 0.
2. Most JavaScript is not authored in the format that people consume it.
3. Modularity only Just Works™ when your each item in your dependency tree has a standard way to expose itself.

These general problems also apply to other asset types (e.g. css, images), but I'll focus on JavaScript for this proposal.

Dependency Depth

@rayshan was kind enough to give me a data dump for the last 30 days of bower installs. I'm still playing around with that data, but I think I have enough to make some points. Typical dependency depth for libraries published to Bower is 0. I don't have any data on npm, but just looking into some node based projects on my computer, it's not uncommon to see dependency depths of 5 or more.

One exception to this in Bower is the clusters of libraries around foundational projects like jQuery, Ember, or Angular. Here, dependency depth is between two and three. This appears to be because these projects' communities either have a shared build ecosystem and/or a mandated modular format (ES6 for Ember, the $ namespace for jQuery, angular.module for Angular, etc).

JavaScript outside these silos tends to lack (or not expose) modularity. This lack of modularity is leading to fragmentary solutions which complicates the process of getting dependencies into projects without a lot of additional effort on the part of the package consumer.

The Gold Standard for this should be systems like Rubygems and npm where coarse (package-level) and fine-grained (specific object import) dependency is so frictionless, you'd be crazy not to use it.

To summarize:

npm install foo

Gets the foo package as a copy locally

var aFoo = require('foo');

Is really a shorthand for the loader to go find node_modules/foo/package.json, look for its main entry, and bring the contents of that file into the execution environment.

In Bower the process is more like:

bower install foo

# lots of complicated custom tooling to convert
# between module formats or wrap un-modular
# code in IIFEs for safety or wrap in module shims

require(["foo"], function(foo) {});

Problems

The lack of that common module system is forcing authors to engage in various painful-to-maintain patterns:

• Checking in compiled code (or multiple versions of it) into source control. A.k.a. the dist/ pattern. Pain points: having a main entry in a package manifest is basically meaningless, forgetting to build before publish leads to problems where a tagged version's dist/ is still the old code, not every project uses the same folder for builds, so each consumer has to go find where code is.
• Creating special module-specific forked repos that are not associated with the main project (https://github.com/amdjs/backbone, https://github.com/amdjs/underscore) and then publishing those as new Bower packages. Problems: "This branch is 16 commits ahead, 489 commits behind jashkenas:master".
• Creating official shim repos https://github.com/components/ember. Problems: polluting Github with "read-only" projects, maintenance pains around people not knowing that the repo isn't the real one and opening issues and pull requests against it.
• The "¯\_(ツ)_/¯" approach of shipping one module system. I chatted with @jeremyckahn about how he handles the module problem for https://github.com/jeremyckahn/rekapi/ and he basically said "what module problem?". Turns out he ships only in AMD/one global variable exposed and other format users are just not in scope. This is also true for everyone who publishes as CJS and says "just use browserify!".
• Embedding dependencies: jQuery embeds Sizzle. Ember embeds RSVP.js, router.js, route-recognizer.js, etc (about a dozen libraries). This makes combining these libraries complicated. If I build a library on top of Sizzle but not jQuery and you also pull jQuery into your project you effectively have two copies of Sizzle. You can work around this by creating custom builds (jQuery compat and non-jQuery compat) but that just adds to the "where do builds go!?!" problem.
• Thinking UMD will help. (It does if your dependencies have no dependencies – "UMD works great when you have no dependencies in your lib" – which contributes to the overall shallow dependency tree).

Ideal Solution

We all author in ES6 modules and compile to AMD for runtime. When node 0.12 ships it lets people use ES6 modules natively. The node community rapidly shifts over. By December 2015 all major browsers parse ES6 modules natively. I get a pony.

Actual Solution.

It's been 5 long years already. We're not going to convince everyone using one module system in the near future. Maybe in 2016 or 2017. Nobody likes alternate module systems. Everyone thinks their module system has "won" or will "win". Everyone is wrong.

Until that all gets resolved, I'd like to suggest we make Bower aware of Github Releases and, more specifically, release assets. Release assets:

You can also attach binary assets (such as compiled executables, minified scripts, documentation) to a release. Once published, the release details and assets are available to anyone that can view the repository.

The 10,000ft view of how this could work:

• Authors can remove built code from source control.
• Authors can author in whatever module format or language they prefer.
• Authors can publish builds as release assets that transpile from their preferred authoring format into executable JavaScript wrapped in multiple module syntaxes
• Authors can use what ever build tools they prefer to prepare these publications
• Consumers can specify preferred module formats in their projects. Bower will attempt to provide this to them.
• Consumers will get the default behavior (plain source checkout) if an asset of the preferred format is unavailable.

The lower level view of how consumers use this (which we can bikeshed):

• In a project .bowerrc, I can specify acceptable build types in order

    {
      "buildAssets": ["es6", "amd"]
    }

  or maybe grouped by asset type?

    {
      "buildAssets": {
        "javascript": ["amd", "global"],
        "css": ["sass", "css"],
        "icon": ["svg"]
      }
    }

• In addition to sha/tag mappings, the bower local cache gets populated with
  available build assets by name for each release when using the GitHubResolver.

• When a consumer attempts an install Bower checks builds by name as part of the resolution phase

• If a build is available for the resolved release version, that build is downloaded instead of the release build.
  So for https://github.com/trek/multi-publish-example, asking for amd gets you
  https://github.com/trek/multi-publish-example/releases/download/v0.0.1/amd.zip instead of https://github.com/trek/multi-publish-example/archive/v0.0.1.tar.gz

• If no matching build is available, the current behavior of just obtaining the source code applies.

• Builds can be specified explicitly and stored in the bower file so this check can be skipped later.

  bower install jquery@amd
  # or
  bower install jquery#2.1@amd

Some problems with this proposal:

• You need to use the Github API, which is rate limited to 5000 requests/hr. With caching this may not be an actual problem in the wild.
• Projects that ship multiple asset types (e.g. Bootstrap). Do they need to make a build for every asset type combination? Convert into a metaproject that references one JavaScript, one CSS, and one Font project? Maybe they just don't participate at all?
• What is special about module formats over, say, precompiled-code formats (coffee, clojurescript, etc)? I'd argue that because they represent the standard way a library exposes its public interface, module systems have more in common with wrapping code in IIFE's than compiling from coffeescript to executable JavaScript. In my mind, if you authored in coffeescript, that code is black-boxed inside the module so your publish phase would compile from coffee to javascript, then convert your authoring module format to all compatible module formats. Other people (@domenic) disagree.

[trek]

cc @timrwood who is experimenting with ES6 module authoring for moment.js https://github.com/timrwood/moment-es6

[searls]

👍 to the careful thought, background, and reasoning. It's long been clear that anyone arguing a single module (or packaging) format is going to emerge as dominant—especially across all of the contexts in which JavaScript is written—is probably delusional or trying to sell you something.

I think Github releases and its API are one potential path (I'm woefully unfamiliar with what they have exposed via the API). I have to ask:

• is there an opportunity for a decentralized or at least re-implementable alternative to Github Releases to accomplish this? Realizing Github is incredibly dominant, especially insofar as bower is concerned, I'd be content just to know someone could write an adapter to another release source of record in a pinch.
• is this an opportunity to specify more concrete metadata about assets? Getting the types of assets and how they reveal themselves is certainly useful, but the concept of an entry point is difficult to pin down with a lot of projects that currently publish full source, minified source, localized sources, plugins, CSS, fonts, images, and so forth. Until Bower or another tool finds a way to popularize people formalizing this metadata, I worry improvements like this won't eliminate the need for custom build configurations for each one-off dependency.

[danvarga]

I like the proposal. I am currently working on a project which has put off listing on bower explicitly because we did not want to check in compiled or concatenated assets. If bower were aware of GitHub releases this would alleviate a huge pain point for us.

[trek]

@searls

is there an opportunity for a decentralized or at least re-implementable alternative to Github Releases to accomplish this? Realizing Github is incredibly dominant, especially insofar as bower is concerned, I'd be content just to know someone could write an adapter to another release source of record in a pinch.

For sure. Each resolver would need to figure out how to get this meta data from a package source, but nothing is stopping a package source from exposing this.

is this an opportunity to specify more concrete metadata about assets?

I wish, but I don't think the bike shedding on that would even stop. My hope is that, for JavaScript at least, we can get main to mean "have your loader/resolver start at this file and then walk the dependency tree from there".

Minification is an interesting bit. I feel like people who prefer global builds also prefer to have a minified option available to them.

People who care about module systems are more likely to use a build tool and know that minification is not enough. They're probably most interested in tree-shaking and whole-app minification rather than having dependancies pre-minimized for you. Minification is a fickle process, but I've noticed that delaying minification as late in the build process as possible often leads to smaller builds as the minifier gets more aggressive finding repeated strings in your apps and across all dependencies.

[sheerun]

Uh.. I agree about the problems, but not about the solution. You said "We're not going to convince everyone using one module system in the near future." but at the same time you want authors to create builds for different module systems. Authors don't care. Only consumers care. We can't use GitHub builds for that, because only authors can create them.

Btw. did you read #1520?

Please think about other solution, so builds can be automated, and consumers can define them.

[trek]

Authors don't care. Only consumers care.

I'm a library author and I care. This problem literally keeps me up at night. If I didn't care I'd just say "I write in CJS and publish on npm. Just go user browserify. It works." I know many, many other authors who care.

Please think about other solution, so builds can be automated, and consumers can define them.

Putting this onus on consumers doesn't make a ton of sense to me. When a compile fails and the consumer sees an error, they're the least likely person to know how to fix my library.

[sheerun]

I believe you care, but ultimately many (most?) of authors don't. Even if they do, they don't have the time to do so. Even you: https://github.com/trek/grunt-neuter/issues/48 I hope you get what I mean, and I don't mean to offend you.

As a consumer I don't want to wait month for author to fix their bower.json or create build for my module system. People write custom solutions for those problems, like bower-installer for overriding bower.json entries or SystemJS for loading modules in any format (AMD, CJS, ES6).

[timrwood]

I'm a library author and I care.

[trek]

Even if they do, they don't have the time to do so.

That's fine. For authors who don't care, the current behavior remains unchanged.

Even you: trek/grunt-neuter#48 I hope you get what I mean, and I don't mean to offend you.

I fail to see how this is related to the issue at hand. Maybe you can help me understand why you feel its related to publishing built assets? He's asking for an update to a Changelog to reflect the 0.6 release (there weren't really any changes of note). The commit he's hoping will go out actually introduced a bug (one that never got resolved), which is why it is not yet released and a 0.7 has not been issued. To me this all sounds like rational and responsible authorship.

As a consumer I don't want to wait month for author to fix their bower.json or create build for my module system.

That's fine. If an author doesn't participate the current behavior still occurs.

[locks]

For what is worth, I also care about this as both a library author that wants to provide a good user experience, as well as a library author that has to consume other libraries as dependencies.

[desandro]

Hi Trek! Lemme say thanks for digging hard into this and spending plenty of brain-time with it.

The a-ha moments of your Medium article & this proposal is discussing that consumers just want it to work. They don't want to deal with build processes or compilation rigamarole.

I share sheerun's concern about authors. They don't want to do extra work either.

What I was hoping to see was a proposal that puts all the responsibility on the package manager. Something like...

• Authors code how they want
• They identify their format in manifest.json, and list what hooks they expose
• Consumers request the format they want in their consumption manifest.json
• The package manager does the heavy lifting between the two formats

This is what I'd like to shoot for. Reading the proposal, the part I get hung up is how the author has to specify the build assets. Is this something the package manager can facilitate? Maybe I'm being idealistic.

---

@sheerun @trek I think we got into a straw man fallacy rat hole there. Authors care. Consumers care. Let's talk about package management.

[EndangeredMassa]

I believe @trek addresses the issue of where the transformation logic goes here.

If cli-tool can’t transform from one format to the other (e.g. circular dependencies that ES6 can handle, but CJS cannot or expressions in require statements that make resolution non-deterministic) warn the author and allow them to either fix or choose to skip publishing to a particular consumption format.

If there is an error when moving from one format to another, letting the author know that so they can fix it before they even publish sounds like the best approach.

[cvrebert]

@desandro

They identify their format in manifest.json, and list what hooks they expose

That sounds basically the same as bower/spec#10 , which never got fully resolved due to Bower's in(activity|decisiveness).