Collaboration with BioJS (same module system and package manager)

[bmpvieira]

BioJS was initially a registry of browser components for biological visualization. Bionode is more oriented for data manipulation (finding, parsing, analyzing, etc) and is more similar to other Bio* libraries like BioPython, BioRuby, etc. When possible, Bionode modules work client and server side while BioJS worked only on the Browser. Consequently, there was no overlap between the two projects. Now, BioJS no longer wants to be just a repository, and also wants to work server side. This could be an opportunity for both projects to work together to avoid duplicated efforts.

However, there's one major point where we're not agreeing, Bionode uses Node.js CommonJS with Browserify and BioJS team wants to move their modules to AMD. BioJS argues that AMD is the only system that allows for live module loading while others require building. Bionode went for Browserify because it allows using Node.js core features (like Streams) on the browser. Browserify supports live reload with tools like watchify, gulp, beefy, etc.

The BioJS team suggests discussing the following possibilities for integration between both projects:

1. use AMD on the server
   e.g. use RequireJS as Node.js module
2. use a CommonJS bundler for the client
   load compiled modules in the browser (Browserify, RequireJS,...)
3. define modules two-way: UMD (universal module definiton) [4]
   specify them as AMD and CommonJS module (and global browser constants) in parallel
   e.g. commonjsStrictGlobal.js
4. Bionode ideas
5. Stop talking

We hope this issue gets some feedback from the bioinformaticians, Node.js and JavaScript communities.

[max-mapper]

There are lots of tools that do bundling for you, e.g. http://requirebin.com, http://wzrd.in/, http://jspm.io/, https://normalize.github.io/. I'd love to see the bio.js folks make critiques of these, as it sounds like they have strong opinions on what "building" means and I'm not sure what their actual requirements are.

It would be pretty cool if the bio.js modules were written as small modules rather than one monolithic repo and used the npm package format so components could be easily shared between node/browser.

[wilzbach]

Hi thanks for opening this issue :)
I am Seb from the BioJS guys and very happy that we are having this discussion.

One of the best examples is that we have at least three different implementations of a FASTA parser - so I really want to modernize the BioJS codebase and introduce modules. My dream is that the Bionode project can also profit from our achievements.

We don't have a strong opinion on "building" - it's more that our current build process is totally annoying (it takes about one minute for bundling) and AMD seemed to be the easiest to use for client development as it requires no installation and therefore is (1) independent from the dev platform, (2) easy to use for starters.

During the last two months I made a lot of experience with CoffeeScript via an AMD loader and my overall summary is that it works most of the time fine, but has some pitfalls (shimming of third-party libs, debugging only works nicely in Chromium, relative module path don't work that smoothly for bundling with CS) - hence my opinion isn't fixed here and heading for an overall standard like the upcoming ES6 modules would be fantastic.

BTW there is another large JavaScript component with bioinformatics context (the JBrowse genome browser) which uses AMD.

I'd love to see the bio.js folks make critiques of these

They are all great projects - I tried to group your suggestions and added points that could be problematic.

IDE in the browser

http://requirebin.com:

• what if you want to edit n > 3 files
• does it work offline?
• how ugly is the debugging here? like in JSFiddle?

CDN bundler

http://wzrd.in/

• is it possible to use this as an development setup? (as they bundle existing requireJS modules on-the-fly)

---

below this line the real discussion (possible alternatives) begins

Reload with watchify, gulp, beefy

• we would loose the aim of having a really simple setup

ES 6 module loaders

http://jspm.io/ (universal), https://normalize.github.io/ (minimal), https://github.com/ModuleLoader/es6-module-loader (just ES6 modules), https://github.com/systemjs/systemjs (universal)

• I haven't yet found time to play around with the ES 6 modules
• If this downcompiling to AMD works fine - that would be an awesome solution :)
• Has anyone already experience with any of those loaders (or ES 6 modules in general)?

[alanrice]

Now, BioJS no longer wants to be just a repository, and also wants to work server side.

Is there any discussion on biojs's side about the desire to work server-side? I might have missed it but I didn't see it mentioned from a quick scan of the mailing list or issue tracker. Is this community driven or an organisation decision? Are there particular project goals outlined already?

[max-mapper]

@greenify hi, nice to meet you!

we would loose the aim of having a really simple setup

the steps for an app that uses beefy or watchify are:

1. clone repo
2. install node.js
3. run npm install
4. run npm start and/or npm test

e.g. here's an example of a start script using beefy: https://github.com/maxogden/dat-editor/blob/master/package.json#L7

note that beefy does not need to be installed globally, npm will use the node from the node_modules directory

to run a test suite you can use npm test, e.g. try the above steps on a module like https://github.com/maxogden/multiplex

browserify-cdn can be run locally, but beefy is a more sane setup when it comes to debugging modules

[wilzbach]

Is there any discussion on biojs's side about the desire to work server-side?

Our goal is to have reusable JavaScript modules (e.g. I/O, REST APIs, algorithms, ..) which would be a waste of effort if one can't use them for server-side JavaScript. However the main priority of BioJS "being a framework for visualizing biological data" did not change, it's more that a few other devs and I want to focus more on the framework part and want to make "BioJS fun to use".

We want to achieve the first big step towards a modular BioJS in the first digital "BioJS hackathon" in the begin of August and (hopefully migrate a lot of the existing components).

for more info:
https://biojs.github.io/code/2014/07/04/announcing-hackathon.html
https://groups.google.com/forum/#!topic/biojs/h19BgjDGdBo

[ariutta]

Hello,

I am with the wikipathways.org team and have spent much of the last two
years immersed in applying JS to the bio domain. My experience leads me to
support making libraries dual-use for browser and server whenever practical.

We have been using CommonJS and Browserify, but I've been hearing good
things here in the Bay Area about Webpack: http://webpack.github.io/. It
supports both CommonJS and AMD.
On Jul 19, 2014 8:31 AM, "greenify" notifications@github.com wrote:

Is there any discussion on biojs's side about the desire to work
server-side?

Our goal is to have reusable JavaScript modules (e.g. I/O, REST APIs,
algorithms, ..) which would be a waste of effort if one can't use them for
server-side JavaScript. However the main priority of BioJS "being a
framework for visualizing biological data" did not change, it's more that a
few other devs and I want to focus more on the framework part and want to
make "BioJS fun to use".

We want to achieve the first big step towards a modular BioJS in the first
digital "BioJS hackathon"
https://biojs.github.io/code/2014/07/04/announcing-hackathon.html in
the begin of August and (hopefully migrate a lot of the existing
components).

for more info:
https://biojs.github.io/code/2014/07/04/announcing-hackathon.html
https://groups.google.com/forum/#!topic/biojs/h19BgjDGdBo

—
Reply to this email directly or view it on GitHub
#9 (comment).

[bmpvieira]

Hi @ariutta,
I actually went for Webpack first since it looked like to most flexible option, but then started switching to Browserify because of the Node.js core support.

[ariutta]

That's our current strategy as well. Having streams both browser- and server-side opens up some exciting possibilities, as used in the Highland library.

For the AMD vs. CommonJS issue, the lodash library handles it like this, and underscore does this. Would following one of these patterns solve this issue?

[wilzbach]

Greetings back to everyone,

1. @maxogden: You are right - my argument about having an uncomplicated setup is very weak - it takes me about 60 sec to clone and run. Probably I am just having too much contact with "Windows"-persons. From all what I read so far I understand why you picked CommonJS ;-)

2. AMD: The idea is that everyone who wants to use the BioWeb modules should be able to do so.
   So yeah (@ariutta) sth. like UMD fragments could be used to wrap the lib for AMD loaders (one can also use UMD to support CJS in AMD modules). However using two different technologies like AMD and CJS in parallel cries for troubles ...

3. Assume BioJS would also use CommonJS (for the shared modules)

   If I read Bionode.js correctly Bionode consists just of a sequence class. I saw that there is a separate repo for parsing FASTA files from @alanrice. So what is the advantage of splitting the code in tiny plugins? At least I am not sure whether it really makes fun to maintain n > 10 node modules.

4. I didn't know that the draft for the ECMAScript 6 was so far - what is your opinion about the upcoming standard (section 15.1)? Loaders like SystemJS seem to be very promising ...
   Even tough the ES 6 module spec is very young and it is unknown whether it is final nor how it's future will look like, they call it a "standard" for a reason..

[mikolalysenko]

I'm a little bit late to the game here, but I would like to chime in with my own thoughts on the CommonJS vs AMD debate, and how I see it within the broader context of evolving software ecosystems:

First of all, there just isn't much difference between AMD and CommonJS. AMD has a slightly clunkier syntax and is more cumbersome when you are building for production deployments (since you still need to bundle everything anyway), while CommonJS requires you to use some tooling for live reloads during development. On the whole, these variations amount to bike-shed levels of significance with neither side having enough differences to give it any real leverage.

However, CommonJS has one towering advantage over AMD that so far has not been directly addressed, which is that it is default package format for node.js and npm. npm is the fastest growing software ecosystem in any language and for good reason - it makes it possible to safely use dependencies. Specifically, there are 3 core features in npm which solve this problem:

1. npm provides a system (semantic versioning) to specify version names and compatibility between different modules, and an interface/module system (CommonJS w/ node_module look up algorithm) which they can use to talk to each other.
2. Version conflicts in dependencies are handled by recursively installing multiple copies of modules.
3. The npm registry provides persistent and immutable naming service for modules at specific versions. eg. once you publish mymodule@1.0.0, it is impossible to ever change or modify the contents of that package, so that any users depending on that version can be guaranteed it will always work as in their test cases.

Any package manger which fails to address all 3 of these points fails to ensure the correctness of modules with dependencies:

• The first point is a bit trivial, but I bring it up since without some agreeing to some conventions then there is no way to proceed. In a sense, these choices are the most arbitrary ones in npm, but they are a logical necessity for what follows. If you don't take a stand, you will end up with something like bower, which is a "package manager" in the same sense that wget is (ie it downloads stuff for you, and that's about it).
• "Recursive dependency copying" is the most widely recognized innovation in npm. If a package manager installs all dependencies within a single flat namespace, then you can't install multiple versions of the same package (the "diamond" dependency problem). For example, if you use modules X and Y in your project, and X depends on Z@1.0.0 and Y depends on Z@2.0.0, where the versions of Z are incompatible, then it is impossible to use both X and Y in your project at the same time. This sort of issue shows up all over the place taking Z=jQuery/Angular/express/whatever framework you like, and X/Y being various flavors of plugins. While this might seem obvious, it is surprising that many "mainstream" package managers like go-get still fail to solve or even address this issue.
• Persistent naming is a more subtle issue, but it is necessary to ensure that projects do not fail after remote repositories or services go down. In general, any module system needs to provide some naming service to ensure that previously published modules do not silently change. Even otherwise decent package managers, like component or Rust's cargo, fail to address this problem. As an illustrative example of how inconsistent naming breaks software, consider the ill-fated Kickstarter for Haunts:
  • Go screwed over the inde game haunts

In that situation, "not even the original programmer could compile the original code on his original machine" due to dependencies silently and remotely changing. Needless to say the project was not able to deliver on their promises to their backers.

But more broadly, failure to take dependencies and interoperability seriously contributes to a hostile environment that discourages code reuse. Fear of dependencies pushes developers to build closed fiefdoms with layers of redundant and excruciatingly boring functionality. This is the sort of "framework hell" that many in the node community complain about. Working with a system like npm makes it not only possible, but even preferable to split code into as many small dependencies as possible. In my opinion, this makes it more fun to write code using CommonJS and npm, since you can skip all the boring details with simple canned modules and cut right to the most interesting parts of whatever you are trying to do.

[wilzbach]

Wow great final speech :)

even preferable to split code into as many small dependencies as possible.

I was questioning whether it really makes sense to ship 50 lines as a separate package.
At least for the BioJS core our idea was to have it as a single package (and maybe if it grows extensively split it into I/O, Algorithms etc.)

Yeah, I do grasp the advantage of tiny packages for better modularity, but

• what happens if you want to change your build chain (e.g. different testing, code analysis)?
• you loose control over your own core (because it is split into 50 packages, maintained by 20 "dead" people)
• how do you generate a global documentation or achieve bundling (if they are in separate repos - well there are git submodules, but then having one repo would be a lot simpler)

[dasmoth]

For what it's worth, Biodalliance agonised over this for a while, and eventually settled on CommonJS, largely because, like @mikolalysenko, I thought it was the technology with most momentum. Also, at the time Browserify seemed like the best bundling option (although I realize that's an area where things are moving quite rapidly).

[bmpvieira]

I prefer having lots of small modules (even if only 50 lines) instead of a bigger core that no one can read.
Each module should do one thing right, have their one tests and issues tracker. Looking at the tests should make obvious how the module is used and works. In addition, docs and examples should be provided in the README.

Having many independent modules would allow any other project (even outside biojs/bionode) to pick the modules they need for their specific goal. Some interesting integrations with other JS projects could come from this.

Small modules are easier to maintain, understand and fork if needed. They facilitate more pull requests.
We should try to agree on testing tools, but each module maintainer could use whatever they want as long as doing "npm test" just works and the README has badges.

I don't think you loose control. If no one wants to maintain a small module that has issues, then that functionality wasn't so needed after all. I'm more afraid of big modules that really only have one dev that knows what's going on overall and then that dev abandons the project.

You can later on have more higher level modules that require a group of small modules. For example, after we have stable parsers (bionode-fasta, bionode-sam, etc.) we can have a bionode-io that requires all those parsers and does some fancy things like auto detecting file formats. Those parsers could also be reused by a project like transformer
We could also have meta modules that simply require groups of modules, for example bionode-phylogenetics would require modules popular for phylogenetics, or bionode-all would require all bionode-* modules. So you could just do "npm install bionode-all" on a university cluster.

I haven't found the right approach for documentation, but if we agree on some common structure it shouldn't be too difficult to concatenate all *js comments or READMEs and build a fancy html with global docs. I would like to have something similar to underscorejs.org, but in that case that's a hand curated html.

[bmpvieira]

If you haven't yet, I think everybody here should see Max's talks about his views of open source, modularity and the NPM ecosystem (among other things).

https://vimeo.com/77376239
https://webrebels.23video.com/the-lebron-stack-its-a-slam-dunk-by-max
https://www.youtube.com/watch?v=8gM3xMObEz4

https://twitter.com/maxogden/status/494231617590673408