Skip to content

merge with fs.extra and fs-extra #17

coolaj86 opened this Issue May 22, 2012 · 44 comments

No. These pieces are separable so they should be separate. The bigger a piece of software gets, the more neglected and hard to work on it becomes. If you want to have a module that just does exports.mkdirp = require('mkdirp') that's perfectly fine but this module should be its own thing.

@substack substack closed this May 22, 2012

Obviously, a small, fifteen-functions fs library that will be used as a de-facto standard wouldn't be hard to support. It isn't a 100k loc framework, after all.

It's pretty shitty that you need to search for new module for every single function. There is A LOT of wheel reinventions in node community and most of the time you can't tell what module is better than other unless you'll test them.

@coolaj86 would be awesome to use great lib for this.


@paulmillr, well as I find these little tidbits I'll try to throw them into fs.extra. I already put this in there.

I agree with @substack in that that if you put every function into a single file it becomes difficult to maintain.

However, I think that it is easy enough to keep the functions in separate files, but in the same repository - easy to maintain, easy to find.

At the very least it would be good to have honorable mention (links) to the other fs libraries that provide similar functionality.


@paulmillr: I really don't see how it's better from a discoverability standpoint to have people use something called fs-extra when they just want to mkdirp.

Suppose I want to mkdir -p and I don't know what packages to use. So I type npm search mkdir.

$ npm search mkdir
NAME       DESCRIPTION                                                   AUTHOR      DATE              KEYWORDS
ensureDir  Ensure a directory exists, create it recursively if not       =samxxu     2012-03-17 17:00  ensureDir mkdir -p directory
fs-tools   fs helper utilities (walk, copy, mkdir -p)                    =vitaly     2012-02-09 16:59  fs file utils
meta-fs    Higher level utilities for filesystem                         =dvv        2011-12-21 13:00  file async mkdir walk find copy tree
mkdir      Directory creation utilities                                  =joehewitt  2012-04-17 01:16  fs
mkdirp     Recursively mkdir, like `mkdir -p`                            =substack   2012-04-30 08:52  mkdir directory
ndir       The lost dir util tools for Nodejs. Handle dir and file in Event =fengmk2 2012-03-09 02:27  dir mkdir walk file rm filesystem event readfile linereader
zk-mkdirp  mkdir -p for ZooKeeper, using the `zookeeper` node module     =dtrejo     2012-04-09 22:52

There are a few things that would seem to do what I want from the descriptions here.

  • ensureDir - clearly does what I need
  • fs-tools - does what I need but it was only obvious because mkdir -p was listed in the description. If there were more functions in the lib it couldn't put them all in the description and I would have skipped over it.
  • meta-fs - not clear that it does what I need, skipping it
  • mkdir - probably does what I need but it's not obvious that it has mkdir -p functionality, skipping because there are modules that specifically mention my use case
  • mkdirp - clearly does what I need
  • ndir - not clear that it does what I need, skipping it
  • zk-mkdirp - clearly not what I need in this case because I'm not using zookeeper

That leaves us with ensureDir, fs-tools, and mkdirp. fs-tools only made the cut because it was lucky and mentioned my use case in its description but that doesn't scale as a library gets more "little tidbits".

Adding more features to a library just makes it less obvious which features a library actually has. If there is an argument about re-inventing the wheel to be made here, I maintain that "extras" libraries are far more guilty of this than specialized libraries.


@substack don't know about you, but npm is probably the last place i'll search for module.

I will google / github at first. To reach highest rank in google / github search results, the package'll need links from other packages (e.g. README in node-mkdirp etc pointing to fs utils).

chilts commented May 23, 2012

Just as a different view, the first place I'd search would be npm, and the thing I would search for would be mkdir. In fact, I did this a long time ago and found mkdirp which I use in different projects. This issue should stay closed.

jed commented May 23, 2012

this seems like a promising way to kill two birds with one stone:


discrete is the only way I sleep at night.

luk- commented May 23, 2012

@paulmillr what's the problem with testing/experimenting with modules to find the right one for your library? Is wheel reinvention really causing THAT much of a problem for you? Just stick to well-maintained and tested libraries and you have less options already.

luk- commented May 23, 2012

Anyway, +1 to discrete.


@st-luke usually I just want to get my stuff done and not "experiment" with a lot of libs instead, implying they're all doing the same function and some of them are just shittier / more untested than others. They're not web frameworks with many features, so there's no fun of playing with them. You won't learn something new etc.

Modularity is cool. Too much modularity suck. See Python for proper libraries example.


You guys are all crazy. We're not talking about flavors of cake here or the works of Picasso.

I realize that Programming is 1 part mechanics, 1 part art, and 2 parts magic... but... how many hammers does a man need? One with semicolons, one without, two in coffeescript, and three that should have been patches to one of the other four?

I usually just google, but either google + github + npm aren't getting people (like myself) to the right library or the JS community is way too full of people with a terrible case of not-written-by-me syndrome. Or both.

But for goodness sake lets figure some way to fight it out and get a king-of-the-hill so that there's a clear-win somewhere for something.

luk- commented May 23, 2012



I prefer a blended approach. When you really only need require('mkdirp') that should be available for you decoupled and stand-alone. Often, however, having groups of functionality are useful. This is exactly the motivation behind utile where we depend upon common dependencies (including mkdirp)

So ... be discrete and build larger packages when you need to.

Marak commented May 23, 2012

But for goodness sake lets figure some way to fight it out and get a king-of-the-hill so that there's a clear-win somewhere for something.

Sounds dreadfully counter-productive to me. The new npm site will have a lot more metrics for helping find packages. That should help a bit.

You really shouldn't be forging together libraries when you can just as easily create a composite library ( as @substack and @indexzero pointed out ). If you feel a unified API would be better, you should create the composite library for it and use whatever you want for sub-dependencies.


I think I'm with indiezero on this one. I can have my mkdirp and utile too.


Also, to be fair, it can be tough to find a good hammer. Mine's an Estwing.


My way is the only right way and everyone should do it! Seriously guys, if it works for you do it, but don't be surprised if others have different opinions and ways to approach problems. certainly don't expect people to be receptive if it means more work for them :)


This isn't about the Right Way (TM) or finding a Good Hammer (TM).

I don't care if you write it CoffeeScript or IcedCoffeeScript or with semicolons or without semicolons. It's the same 15 lines. It's not like one is better than the other (unless one is actually broken).

  function rangeCheck(val, min, max) {
      if (val >= min && val <= max) {
         return true;

Does anyone care to write that the other 5 ways and publish them all to npm? Maybe we could have range-check and range_check and RangeCheck and rangecheck and checkRange and boppo (with a good description about how it checks ranges). And I'll work up the benchmarks on it while you guys contact Oracle and get a license for us to use it.

I've got some good ideas for another 10 functions or so that handle common validations on numbers. Can we each split up, write them out (some with obscure names for good measure), not tell each other about them, and make sure not to link to them?

I think it would be best if googling for it would take you just to the rangeCheck page and no reference to the other functions.


Dawg if my hammer's not American made I'll be pissed.

(My hammer is American made.)


I like the unix philosophy of on thing well, and not sure what the real issue is here. Seems like when a lib is used by so many projects, does one thing well, and is well done, maybe the question should really be whether or not to integrate into node core instead of with a more general library...


I'm going back to my first statement: Let's meet up at TexasJS and fight it out.
Whoever is left standing can figure out which side their on and who won.


@jgoodall, YES!!!! It should be integrated into core -- with (mostly) clear documentation and grouped with similar functionality.

But from what I've experienced in the past, many of the core NodeJS team have the opinion of this kind of stuff that "It's only 15 lines. People can just write it themselves.... over and over and over and over again"

Marak commented May 23, 2012

@coolaj86 - You should check out again.

If you really want to make a defacto extras / utility library that might get considered merging to core, you should join forces with the Flatiron team and help us get utile fully functional.


@coolaj86 when I do an npm update and mkdirp or lrucache or some othe tiny module that everyone uses is updated 12 times, I gotta wonder, is it so common that it should just be there without npm getting involved? There aren't many of those, but this may be one.


Adding more modules to core ends in a gigantic bloated neglected core that nobody can change because fixing it would break everything and sending a patch requires signing a contributors agreement and running all the tests and waiting until the next stable goes live before anybody else can use it and shimming out work-arounds for people not using the latest version of node. It's the module equivalent of state-sponsored industry or COMMUNISM. Let the modules fend for themselves in the free market chaos of npm. WHY DO YOU HATE FREEDOM?

Marak commented May 23, 2012

@substack - Exactly.

The one shining light, is that if we actually make a separate utility / extras / more module outside of core. maybe in time, it will mature to the point where inclusion would be possible.

I don't see it happening any other way.


@Marak that was pretty much my point. @substack is no novice, and everyone uses this. I don't think all modules should go into core, but this might be one that would be useful without resorting to the dreaded module COMMUNISM.

luk- commented May 23, 2012

+1 to free market chaos

@coolaj86 I'll be at TexasJS. I'd love to hear a convincing argument on your position.


guys, I have a great idea: why we don't just extract all these modules and create a package for every 1-3 functions of them? this would be so freeee market!


Hey man, even America has public schools. But that shit's done when you're 18.


@Marak I do think that utile is a good idea, but the grouping is a little unclear. Would be nice to have a utile (fs) and utile (flow-control) and such to group them.


function checkRange(val, min, max) {
    return (val >= min && val <= max)

Oh no, it's missing a semicolon! Good thing we didn't move it to core... that would take ages to get fixed... especially being so hefty and 5 lines long and all.

Sarcasm aside, I agree. Last time I submitted a pull request the process was quite a bit more involved than the first time I submitted a pull request. But for something this simple it's a win.

It seems that the general direction of the benevolent dictatorship is to push us to anarchy. It would be nice to have a good balance where things group together and either gain traction or die, not having so many things that are 'good enough' with no clear advantage.


After all these strong words and opinions...

A little love and an apple to say "thanks for everyone's hard work"

dstrek commented May 23, 2012

If the problem is module discovery how about we fix that instead of turning node into Java / Rails. It looks like the new NPM site is going to address this issue and we can only improve from there.

Javascript is so powerful and flexible because it's simple!

Modules need to do one thing or else be a framework. No one wants to be including frameworks when all they want is one function. I can look at a module like mkdirp, read and understand the entire code in one minute and feel ok deploying it into production.

The day node app.js feels like rails, ABANDON SHIP

"It seems that the general direction of the benevolent dictatorship is to push us to anarchy."

This is exactly what is going on, and it is a good thing! The present benevolent dictator of both node and npm @isaacs stated this explicitly in the talk he gave at tacoconf:

taco anarchy

Having a small core is informed by these anarchist principles. The node/npm ethos is that we should push most all of the decisions about how apis and modules work into userspace so that core can focus on doing a small set of things well.


@dstrek puts it well.

If the problem is module discovery how about we fix that instead of turning node into Java / Rails

It starts so innocently with promises of a little extra functionality here, 'cool' feature there, and an extra 5 lines somewhere. Then, before you know it, Merb has turned into bloatware called Rails 3.2. ...yeah, still bummed about that one :)

Don't let Node go this route. Keep up the anarchy.


@substack I took the time to read that post at 5am for some reason; exceptionally well-written, but I think it really shed some light on what @marak was trying to suggest to @coolaj86.

For example, if one were to consider @isaacs as the center of this directed graph:

Then by corollary many of the people on this discussion thread would be (near) top-level peer nodes from the center. The existence of a sibling edge between nodes in the graph is by definition agreement on some approach to filling the need for additional functionality in user-land.

To me, this really represents what we already have: oligarchy; not anarchy: users center around a set of modules created by one or more individuals. This creates healthy competition and without strict adherence to decoupling at some level this oligarchy would quickly turn into monarchy.

Marak commented May 23, 2012

isaacs commented May 23, 2012

As @Marak so eloquently points out, this whole discussion is a little bit silly.

It's fine to have packages that bundle a bunch of related functionality. However, the success of those depends on that functionality being developed in the first place, and keeping mkdirp separate means that it can be more easily iterated on.

Asking @substack to make it a part of fs-extra is fine. Him saying "no" is also fine, and actually, somewhat expected, given that he is very much on the "smaller is better" side of module size preference.

If you want to bundle mkdirp into fs-extra, just make it a dependency, and expose it somehow. mkdirp is MIT-X11 licensed, so you can use it in fs-extra or wherever, without even asking for permission, as long as you comply with the license.

Freedom must include the freedom to make mistakes, to disagree, to make requests and hear "no" as an answer, or to say "no" to requests, or else we're doomed.

donwb commented May 23, 2012

@isaacs actually, I don't think this conversation is silly; rather I think there's a fantastic underlying subtext about the philosophy around node, modularity, and the battle against complexity.

I've worked with plenty of bloated, overgrown frameworks; still do. What's worse, is I've recently seen several that start small (and awesome), only become bloated over time due to the irrational desire to continue adding "features". This whole thing is a slippery slope. You start consolidating/adding stuff and the next thing you know you're the .Net Framework.

Node is doing it the right way.. stay small and let userland fill in the holes. Userland, battle against the "conventional wisdom" in software; that bigger is better.

A richer npm catalog will go a long way to help this..

dscape commented May 23, 2012



Comming from, I had the pleasure of reading your opinions, and I feel I must comment to notify you it was really fun. Thank you guys, I both learned a bit, and laughed a bit, so I'm happy now.

It seems that we have a so flexible system that a package can be a module, and a module can be a package, so we have a system structure-free, where words like module and package can mean what we want, where the limit is imagination.

That's good news.

Happy coding for all, and thank you all.

stevemao commented Apr 5, 2016

sindresorhus/ama#10 sindre also wrote some good summary of small focused modules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.