Modularization #996

Merged
merged 39 commits into from Feb 24, 2016

Projects

None yet
@megawac
Collaborator
megawac commented Jan 3, 2016

A work in progress to bring modular support and lodash implementations of certain methods.

Previous discussion be found here #984.

Resolves #984

See https://github.com/caolan/async/tree/modularization/build for the generated bundles


TODOS

  • script releases:
    • generate modular builds/files
    • copy ES modules to build/es
    • add git commit
    • npm test
    • smoke test build files
    • cd build/
    • publish npm release
    • revert modular build changes, returning async to normal folder structure
    • git tag, git push
  • possibly? create npm release for each module
  • #984 (comment)
  • use baseOrderBy in the implementation of internal/flatten?
  • move documentation to jsdoc format (bonus?) save for later
  • test package with npm pack before first 2.0 publish
Kikobeats and others added some commits Dec 15, 2015
@Kikobeats @megawac Kikobeats Initial work in progress - modularization
Add .DS_Store

Extract util methods

Fix createTester callback

Fix notid exports

Refactor settimediate and nexttick

Fix dep path

Create bundle using browserify

Rename main file

Rename main file

Rename main file

Use browserify standalone mode

Modular interface for main methods 📦

Deleted unnecessary test

Add script to generate modules package.json

Delete noconflict module

Improve how to generat browser bundle

Update util modules references

Add a way to generate module scaffold

Fix version

Remove unnecessary dependencies

Require the dependency

Add missing methods

Add dependencies for each module

Bumped 0.4.0

Add useful scripts

Add .npmignore

Updated

Add npmignore files

Refactor

Fix little issues

Fix library name

Fix isarray module

Update script

Fix typo

Fix template links

Update deps

Revert "Fix template links"

This reverts commit 112a382.

Bump 0.5.1

Fix mapseries reference

Bump 0.5.2
7127b67
@megawac megawac [WIP] modularization (#984) 18e61d4
@megawac megawac Fix implementations (test green lights) 1448f24
@megawac megawac Set up async bundles 90656cf
@megawac megawac Adjust commonjs builds and use umd format 08af90e
@megawac
Collaborator
megawac commented Jan 4, 2016

I have managed to remove most of the string parsing logic included by lodash -- the only string parsing code remaining is toNumber included by baseRange. @jdalton may consider moving the toNumber logic to range and having baseRange assume it is provided numeric start, end and step?

Also @jdalton it'd be nice if there was a stable ref I could link for a lodash-es submodule -- it's pretty frustrating to update the submodule ref every couple days

@jdalton
Contributor
jdalton commented Jan 4, 2016

may consider moving the toNumber logic to range and having baseRange assume it is provided numeric start, end, and step?

Good idea!

it'd be nice if there was a stable ref I could link for a lodash-es submodule -- it's pretty frustrating to update the submodule ref every couple days

Lodash v4 will be released on the 12th.

@megawac
Collaborator
megawac commented Jan 4, 2016

Cool, so there will be a stable es-4.0.0 ref in v4?
On Jan 4, 2016 9:32 AM, "John-David Dalton" notifications@github.com
wrote:

may consider moving the toNumber logic to range and having baseRange
assume it is provided numeric start, end, and step?

Good idea!

it'd be nice if there was a stable ref I could link for a lodash-es
submodule -- it's pretty frustrating to update the submodule ref every
couple days

Lodash v4 will be released on the 12th.


Reply to this email directly or view it on GitHub
#996 (comment).

@jdalton
Contributor
jdalton commented Jan 4, 2016

Cool, so there will be a stable es-4.0.0 ref in v4?

Yep, just as there's a stable 3.10.1 ref.

@megawac
Collaborator

@Rich-Harris is there an option to disable this default check?

@megawac not presently. We should probably replace these with something similar to Babel's interopRequire

have opened an issue for it rollup/rollup#419

@aearly
Collaborator
aearly commented Jan 5, 2016

What does the testing situation look like? Do we just test the bundle? It might be handy to test the es6 modules directly somehow as well, just to verify that Async works in both forms.

@jdalton
Contributor
jdalton commented Jan 7, 2016

Updated the es branch with toNumber removed from baseRange.
The gzipped minified size is now 6.17 kB so a 1.9 kB difference which I think is fine.

@megawac
Collaborator
megawac commented Jan 7, 2016

Nice! 👍 that is totally reasonable in my opinion, thoughts @aearly?

@aearly aearly added this to the 2.0 milestone Jan 7, 2016
@jdalton
Contributor
jdalton commented Jan 7, 2016

Found another cheap win which will bring the size down to 5.73 kB; the diff to 1.4 kB
5.87 kB; the diff of 1.6 kB :P

@aearly
Collaborator
aearly commented Jan 7, 2016

Really excellent 👍 . Lodash is totally worth 1.4kb 😛 .

The only danger I see is relying on an internal lodash function, but those are pretty stable, right?

@jdalton
Contributor
jdalton commented Jan 7, 2016

The only danger I see is relying on an internal lodash function, but those are pretty stable, right?

It depends. When relying on internals I usually avoid the ^ range and go with the ~.
I can review the deps and report back if I see anything of concern. Offhand I think the internals used here are stable.

@megawac

can be even simpler and import lodash-es/

Collaborator

never mind, the reason I was using submodules was so rollup would properly bundle lodash

Collaborator

Yeah, I tried getting rollup to automatically bundle lodash-es from npm, but couldn't get it working. I need to familiarize myself with all the plugins and options more.

@megawac
Collaborator
megawac commented on 1f79497 Jan 8, 2016

Nice thanks

Collaborator

This is really odd, I don't know how @Kikobeats showed up as the commit author here. I'm the one working on this. 😄

Collaborator

Oh wait, I accidentally cherry-picked a merge commit (the .editorconfig). That caused some weirdness.

Collaborator

Haha, I was wondering where this commit came from.

@jdalton
Contributor
jdalton commented Jan 8, 2016

Just checked and the base methods used are totally stable. You should have no problems with them.
I also simplified the deps of toNumber.

@aearly
Collaborator
aearly commented Jan 16, 2016

I'm thinking of a clean way to publish the individual method files without polluting the base directory.

My initial thought is to copy the package.json/readme/changelog and whatever else we need to the build/ directory and run npm publish from there. We wouldn't have to use the files property of the package.json because we would just publish everything in that folder. This has some drawbacks:

  1. We can't use xyz any more since it relies on npm test, which needs to be run from the root directory. We'd have to use a custom script to run the tests, bump the versions, and publish.
  2. The package.json will be out of sync with its location. The scripts will be assuming it's still in the root, but the actual publish artifacts will be assumed to be in build/. For example, main would be just async-cjs.js rather than build/async-cjs.js, which could be confusing to people submitting PRs.
  3. We need to make sure that bower still pulls the correct file. The bower.json will need to have its main remain as dist/async.js, and we'll still have to commit that build artifact. The package.json and bower.json main fields will now be different.
  4. I'd like to publish the ES modules as well as the CJS modules. I assume we'll copy lib to build and include "esnext:main": "es/index.js" or something similar.

Anyone have better ideas?

@jdalton
Contributor
jdalton commented Feb 6, 2016

Anything I can help with. Lodash v4 is out and has stabilized.

@aearly
Collaborator
aearly commented Feb 6, 2016

Nope, not really now that v4 is on npm and we've migrated away from deprecated methods.

Any opinion on the proposed publish process in my last comment?

@jokeyrhyme

Now that https://github.com/rollup/rollup exists, is it worth putting much effort into this at a package level? It would seem that migrating to ES2015 modules here and using an ES2015-aware toolchain is a much easier way to offer consuming projects this level of optimisation.

@jdalton
Contributor
jdalton commented Feb 6, 2016

@aearly

Any opinion on the proposed publish process in my last comment?

Why not just copy files to a staging directory during the prepublish. Not sure anything needs to be moved permanently from root (package.json, etc.).

@aearly
Collaborator
aearly commented Feb 7, 2016

@jokeyrhyme We're trying to support both a monolithic CJS or UMD-style import, as well as the individual ES modules, in the same package. We're authoring them as ES modules, and then using babel and rollup to compile them to other formats. The goal is to support ES2015 users, as well as existing projects using CJS in node natively.

@aearly
Collaborator
aearly commented Feb 11, 2016

Nearing completion! I think i have the build and publish process where I want them. I'd love to have your feedback.

I think the next step is to start cherry picking in changes from master.

@TrySound
Contributor

I think you should add built files to .gitignore. This will reduce PR a bit ;)

@megawac
Collaborator
megawac commented Feb 12, 2016

Agreed, but lets do it as a final step so people can sanity check the output

On Fri, Feb 12, 2016 at 10:54 AM, Bogdan Chadkin notifications@github.com
wrote:

I think you should add built files to .gitignore. This will reduce PR a
bit ;)


Reply to this email directly or view it on GitHub
#996 (comment).

@aearly
Collaborator
aearly commented Feb 14, 2016

I think we'll need to keep build/async.js checked in in at least, for Bower support. Or we could just go full lodash and drop Bower support... 🌚

@jdalton
Contributor
jdalton commented Feb 14, 2016

Or we could just go full lodash and drop Bower support... 🌚

See lodash/lodash#1733 (comment).

@aearly
Collaborator
aearly commented Feb 14, 2016

Ah, I see you dropped explicit support for it, but it still works due to how lodash is authored. That wouldn't work here, unless we checked in build/async.js.

@megawac
Collaborator
megawac commented Feb 14, 2016

I think we should just check in the /dist folder with the built async.js
and async.min.js files

On Sun, Feb 14, 2016 at 5:38 PM, Alex Early notifications@github.com
wrote:

Ah, I see you dropped explicit support for it, but it still works due to
how lodash is authored. That wouldn't work here, unless we checked in
build/async.js.


Reply to this email directly or view it on GitHub
#996 (comment).

aearly added some commits Feb 14, 2016
@aearly aearly check in dist for bower support cfbf7ae
@aearly aearly pull in readme changes from master 0d09f43
@aearly aearly add note about in-progress docs for 2.0 db9acac
@aearly aearly merge from master e46c8a6
@aearly aearly update build
636908b
@aearly
Collaborator
aearly commented Feb 14, 2016

I...think we're ready to merge. I just merged from master, and resolved the conflicts. Luckily, they were pretty minor. I think after this we can finish the rest of the work for the 2.0 milestone on master.

@megawac
Collaborator
megawac commented Feb 15, 2016

I'll review tomorrow @aearly

On Sun, Feb 14, 2016 at 6:59 PM, Alex Early notifications@github.com
wrote:

I...think we're ready to merge. I just merged from master, and resolved
the conflicts. Luckily, they were pretty minor. I think after this we can
finish the rest of the work for the 2.0 milestone on master.


Reply to this email directly or view it on GitHub
#996 (comment).

@megawac megawac and 1 other commented on an outdated diff Feb 15, 2016
+build-es: $(ES_MODULES)
+
+$(BUILDDIR)/es/%.js: lib/%.js
+ mkdir -p "$(@D)"
+ sed -r "s/(import.+)lodash/\1lodash-es/g" $< > $@
@megawac
megawac Feb 15, 2016 Collaborator

Nice. Really liking this makefile @aearly

@aearly
aearly Feb 16, 2016 Collaborator

😄 I'm totally a Make fanboy.

This does force windows maintainers to use something like MinGW, though.

@megawac
megawac Feb 16, 2016 Collaborator

Thats fine, as long as they can write and run tests its okay for me

@megawac megawac and 1 other commented on an outdated diff Feb 15, 2016
build/async.js
@@ -0,0 +1,1225 @@
+'use strict';
@megawac
megawac Feb 15, 2016 Collaborator

Do we want this file to be called build/async or build/index. Also on publish are we going to move the /build files to / so people can do import each from 'async/each?

@megawac
megawac Feb 15, 2016 Collaborator

I'd prefer index.js thoughts @aearly

@megawac
megawac Feb 15, 2016 Collaborator

Ah nevermind, I get it -- we publish the build/ folder.

My comments remain - I'd rather have a dist/ folder under build/ with the built async.js and async.min.js files.

I'm not sure if I want es/ to be published to npm. I think I would prefer it to be published in a second repo (async-es or similar)

@aearly
aearly Feb 16, 2016 Collaborator

This is the UMD build. I called it async.js because it is the most analagous to the lib/async.js in v1.x and prior. The CJS build is called index.js.

I'm leaving the dist/ folder in the root so it can be easily targeted by bower users. Moving it to build/dist/ is no problem.

The es/ folder is an experiment. I'm copying what lodash 4.x does with lodash/fp. You can require('async/waterfall') or import waterfall from 'async/es/waterfall' with the same package.

@megawac
megawac Feb 16, 2016 Collaborator

Yeah, I think I'd prefer to keep them in seperate repos for build size concerns.

I was suggesting /dist/async.js for organizational reasons (for instance if we decided to add an async.async function in the future it wouldn't be possible due to the organization of built files.

@megawac megawac and 1 other commented on an outdated diff Feb 15, 2016
build/async.min.js
@@ -0,0 +1,2 @@
+"use strict";!function(){function n(){}function t(n){return n}function e(n){return!!n}function r(n){return!n}function u(n){return function(){if(null===n)throw new Error("Callback was already called.");n.apply(this,arguments),n=null}}function i(n){return function(){null!==n&&(n.apply(this,arguments),n=null)}}function o(n){return M(n)||"number"==typeof n.length&&n.length>=0&&n.length%1===0}function c(n,t){for(var e=-1,r=n.length;++e<r;)t(n[e],e,n)}function a(n,t){for(var e=-1,r=n.length,u=Array(r);++e<r;)u[e]=t(n[e],e,n);return u}function f(n){return a(Array(n),function(n,t){return t})}function l(n,t,e){return c(n,function(n,r,u){e=t(e,n,r,u)}),e}function s(n,t){c(W(n),function(e){t(n[e],e)})}function p(n,t){for(var e=0;e<n.length;e++)if(n[e]===t)return e;return-1}function h(n){var t,e,r=-1;return o(n)?(t=n.length,function(){return r++,t>r?r:null}):(e=W(n),t=e.length,function(){return r++,t>r?e[r]:null})}function m(n,t){return t=null==t?n.length-1:+t,function(){for(var e=Math.max(arguments.length-t,0),r=Array(e),u=0;e>u;u++)r[u]=arguments[u+t];switch(t){case 0:return n.call(this,r);case 1:return n.call(this,arguments[0],r)}}}function y(n){return function(t,e,r){return n(t,r)}}function v(t){return function(e,r,o){o=i(o||n),e=e||[];var c=h(e);if(0>=t)return o(null);var a=!1,f=0,l=!1;!function s(){if(a&&0>=f)return o(null);for(;t>f&&!l;){var n=c();if(null===n)return a=!0,void(0>=f&&o(null));f+=1,r(e[n],n,u(function(n){f-=1,n?(o(n),l=!0):s()}))}}()}}function d(n){return function(t,e,r){return n(P.eachOf,t,e,r)}}function g(n){return function(t,e,r,u){return n(v(e),t,r,u)}}function k(n){return function(t,e,r){return n(P.eachOfSeries,t,e,r)}}function b(t,e,r,u){u=i(u||n),e=e||[];var c=o(e)?[]:{};t(e,function(n,t,e){r(n,function(n,r){c[t]=r,e(n)})},function(n){u(n,c)})}function w(n,t,e,r){var u=[];n(t,function(n,t,r){e(n,function(e){e&&u.push({index:t,value:n}),r()})},function(){r(a(u.sort(function(n,t){return n.index-t.index}),function(n){return n.value}))})}function O(n,t,e,r){w(n,t,function(n,t){e(n,function(n){t(!n)})},r)}function S(n,t,e){return function(r,u,i,o){function c(){o&&o(e(!1,void 0))}function a(n,r,u){return o?void i(n,function(r){o&&t(r)&&(o(e(!0,n)),o=i=!1),u()}):u()}arguments.length>3?n(r,u,a,c):(o=i,i=u,n(r,a,c))}}function E(n,t){return t}function L(t,e,r){r=r||n;var u=o(e)?[]:{};t(e,function(n,t,e){n(m(function(n,r){r.length<=1&&(r=r[0]),u[t]=r,e(n)}))},function(n){r(n,u)})}function j(n,t,e,r){var u=[];n(t,function(n,t,r){e(n,function(n,t){u=u.concat(t||[]),r(n)})},function(n){r(n,u)})}function I(t,e,r){function i(t,e,r,u){if(null!=u&&"function"!=typeof u)throw new Error("task callback must be a function");return t.started=!0,M(e)||(e=[e]),0===e.length&&t.idle()?P.setImmediate(function(){t.drain()}):(c(e,function(e){var i={data:e,callback:u||n};r?t.tasks.unshift(i):t.tasks.push(i),t.tasks.length===t.concurrency&&t.saturated()}),void P.setImmediate(t.process))}function o(n,t){return function(){f-=1;var e=!1,r=arguments;c(t,function(n){c(l,function(t,r){t!==n||e||(l.splice(r,1),e=!0)}),n.callback.apply(n,r)}),n.tasks.length+f===0&&n.drain(),n.process()}}if(null==e)e=1;else if(0===e)throw new Error("Concurrency must not be zero");var f=0,l=[],s={tasks:[],concurrency:e,payload:r,saturated:n,empty:n,drain:n,started:!1,paused:!1,push:function(n,t){i(s,n,!1,t)},kill:function(){s.drain=n,s.tasks=[]},unshift:function(n,t){i(s,n,!0,t)},process:function(){for(;!s.paused&&f<s.concurrency&&s.tasks.length;){var n=s.payload?s.tasks.splice(0,s.payload):s.tasks.splice(0,s.tasks.length),e=a(n,function(n){return n.data});0===s.tasks.length&&s.empty(),f+=1,l.push(n[0]);var r=u(o(s,n));t(e,r)}},length:function(){return s.tasks.length},running:function(){return f},workersList:function(){return l},idle:function(){return s.tasks.length+f===0},pause:function(){s.paused=!0},resume:function(){if(s.paused!==!1){s.paused=!1;for(var n=Math.min(s.concurrency,s.tasks.length),t=1;n>=t;t++)P.setImmediate(s.process)}}};return s}function x(n){return m(function(t,e){t.apply(null,e.concat([m(function(t,e){"object"==typeof console&&(t?console.error&&console.error(t):console[n]&&c(e,function(t){console[n](t)}))})]))})}function A(n){return function(t,e,r){n(f(t),e,r)}}function T(n){return m(function(t,e){var r=m(function(e){var r=this,u=e.pop();return n(t,function(n,t,u){n.apply(r,e.concat([u]))},u)});return e.length?r.apply(this,e):r})}function z(n){return m(function(t){var e=t.pop();t.push(function(){var n=arguments;r?P.setImmediate(function(){e.apply(null,n)}):e.apply(null,n)});var r=!0;n.apply(this,t),r=!1})}var q,P={},C="object"==typeof self&&self.self===self&&self||"object"==typeof global&&global.global===global&&global||this;null!=C&&(q=C.async),P.noConflict=function(){return C.async=q,P};var H=Object.prototype.toString,M=Array.isArray||function(n){return"[object Array]"===H.call(n)},U=function(n){var t=typeof n;return"function"===t||"object"===t&&!!n},W=Object.keys||function(n){var t=[];for(var e in n)n.hasOwnProperty(e)&&t.push(e);return t},B="function"==typeof setImmediate&&setImmediate,D=B?function(n){B(n)}:function(n){setTimeout(n,0)};"object"==typeof process&&"function"==typeof process.nextTick?P.nextTick=process.nextTick:P.nextTick=D,P.setImmediate=B?D:P.nextTick,P.forEach=P.each=function(n,t,e){return P.eachOf(n,y(t),e)},P.forEachSeries=P.eachSeries=function(n,t,e){return P.eachOfSeries(n,y(t),e)},P.forEachLimit=P.eachLimit=function(n,t,e,r){return v(t)(n,y(e),r)},P.forEachOf=P.eachOf=function(t,e,r){function o(n){f--,n?r(n):null===c&&0>=f&&r(null)}r=i(r||n),t=t||[];for(var c,a=h(t),f=0;null!=(c=a());)f+=1,e(t[c],c,u(o));0===f&&r(null)},P.forEachOfSeries=P.eachOfSeries=function(t,e,r){function o(){var n=!0;return null===a?r(null):(e(t[a],a,u(function(t){if(t)r(t);else{if(a=c(),null===a)return r(null);n?P.setImmediate(o):o()}})),void(n=!1))}r=i(r||n),t=t||[];var c=h(t),a=c();o()},P.forEachOfLimit=P.eachOfLimit=function(n,t,e,r){v(t)(n,e,r)},P.map=d(b),P.mapSeries=k(b),P.mapLimit=g(b),P.inject=P.foldl=P.reduce=function(n,t,e,r){P.eachOfSeries(n,function(n,r,u){e(t,n,function(n,e){t=e,u(n)})},function(n){r(n,t)})},P.foldr=P.reduceRight=function(n,e,r,u){var i=a(n,t).reverse();P.reduce(i,e,r,u)},P.transform=function(n,t,e,r){3===arguments.length&&(r=e,e=t,t=M(n)?[]:{}),P.eachOf(n,function(n,r,u){e(t,n,r,u)},function(n){r(n,t)})},P.select=P.filter=d(w),P.selectLimit=P.filterLimit=g(w),P.selectSeries=P.filterSeries=k(w),P.reject=d(O),P.rejectLimit=g(O),P.rejectSeries=k(O),P.any=P.some=S(P.eachOf,e,t),P.someLimit=S(P.eachOfLimit,e,t),P.all=P.every=S(P.eachOf,r,r),P.everyLimit=S(P.eachOfLimit,r,r),P.detect=S(P.eachOf,t,E),P.detectSeries=S(P.eachOfSeries,t,E),P.detectLimit=S(P.eachOfLimit,t,E),P.sortBy=function(n,t,e){function r(n,t){var e=n.criteria,r=t.criteria;return r>e?-1:e>r?1:0}P.map(n,function(n,e){t(n,function(t,r){t?e(t):e(null,{value:n,criteria:r})})},function(n,t){return n?e(n):void e(null,a(t.sort(r),function(n){return n.value}))})},P.auto=function(t,e,r){function u(n){g.unshift(n)}function o(n){var t=p(g,n);t>=0&&g.splice(t,1)}function a(){h--,c(g.slice(0),function(n){n()})}"function"==typeof arguments[1]&&(r=e,e=null),r=i(r||n);var f=W(t),h=f.length;if(!h)return r(null);e||(e=h);var y={},v=0,d=!1,g=[];u(function(){h||r(null,y)}),c(f,function(n){function i(){return e>v&&l(k,function(n,t){return n&&y.hasOwnProperty(t)},!0)&&!y.hasOwnProperty(n)}function c(){i()&&(v++,o(c),h[h.length-1](g,y))}if(!d){for(var f,h=M(t[n])?t[n]:[t[n]],g=m(function(t,e){if(v--,e.length<=1&&(e=e[0]),t){var u={};s(y,function(n,t){u[t]=n}),u[n]=e,d=!0,r(t,u)}else y[n]=e,P.setImmediate(a)}),k=h.slice(0,h.length-1),b=k.length;b--;){if(!(f=t[k[b]]))throw new Error("Has nonexistent dependency in "+k.join(", "));if(M(f)&&p(f,n)>=0)throw new Error("Has cyclic dependencies")}i()?(v++,h[h.length-1](g,y)):u(c)}})},P.retry=function(n,t,e){function r(n,t){if("number"==typeof t)n.times=parseInt(t,10)||i;else{if("object"!=typeof t)throw new Error("Unsupported argument type for 'times': "+typeof t);n.times=parseInt(t.times,10)||i,n.interval=parseInt(t.interval,10)||o}}function u(n,t){function e(n,e){return function(r){n(function(n,t){r(!n||e,{err:n,result:t})},t)}}function r(n){return function(t){setTimeout(function(){t(null)},n)}}for(;a.times;){var u=!(a.times-=1);c.push(e(a.task,u)),!u&&a.interval>0&&c.push(r(a.interval))}P.series(c,function(t,e){e=e[e.length-1],(n||a.callback)(e.err,e.result)})}var i=5,o=0,c=[],a={times:i,interval:o},f=arguments.length;if(1>f||f>3)throw new Error("Invalid arguments - must be either (task), (task, callback), (times, task) or (times, task, callback)");return 2>=f&&"function"==typeof n&&(e=t,t=n),"function"!=typeof n&&r(a,n),a.callback=e,a.task=t,a.callback?u():u},P.waterfall=function(t,e){function r(n){return m(function(t,u){if(t)e.apply(null,[t].concat(u));else{var i=n.next();i?u.push(r(i)):u.push(e),z(n).apply(null,u)}})}if(e=i(e||n),!M(t)){var u=new Error("First argument to waterfall must be an array of functions");return e(u)}return t.length?void r(P.iterator(t))():e()},P.parallel=function(n,t){L(P.eachOf,n,t)},P.parallelLimit=function(n,t,e){L(v(t),n,e)},P.series=function(n,t){L(P.eachOfSeries,n,t)},P.iterator=function(n){function t(e){function r(){return n.length&&n[e].apply(null,arguments),r.next()}return r.next=function(){return e<n.length-1?t(e+1):null},r}return t(0)},P.apply=m(function(n,t){return m(function(e){return n.apply(null,t.concat(e))})}),P.concat=d(j),P.concatSeries=k(j),P.whilst=function(t,e,r){if(r=r||n,t()){var u=m(function(n,i){n?r(n):t.apply(this,i)?e(u):r.apply(null,[null].concat(i))});e(u)}else r(null)},P.doWhilst=function(n,t,e){var r=0;return P.whilst(function(){return++r<=1||t.apply(this,arguments)},n,e)},P.until=function(n,t,e){return P.whilst(function(){return!n.apply(this,arguments)},t,e)},P.doUntil=function(n,t,e){return P.doWhilst(n,function(){return!t.apply(this,arguments)},e)},P.during=function(t,e,r){r=r||n;var u=m(function(n,e){n?r(n):(e.push(i),t.apply(this,e))}),i=function(n,t){n?r(n):t?e(u):r(null)};t(i)},P.doDuring=function(n,t,e){var r=0;P.during(function(n){r++<1?n(null,!0):t.apply(this,arguments)},n,e)},P.queue=function(n,t){var e=I(function(t,e){n(t[0],e)},t,1);return e},P.priorityQueue=function(t,e){function r(n,t){return n.priority-t.priority}function u(n,t,e){for(var r=-1,u=n.length-1;u>r;){var i=r+(u-r+1>>>1);e(t,n[i])>=0?r=i:u=i-1}return r}function i(t,e,i,o){if(null!=o&&"function"!=typeof o)throw new Error("task callback must be a function");return t.started=!0,M(e)||(e=[e]),0===e.length?P.setImmediate(function(){t.drain()}):void c(e,function(e){var c={data:e,priority:i,callback:"function"==typeof o?o:n};t.tasks.splice(u(t.tasks,c,r)+1,0,c),t.tasks.length===t.concurrency&&t.saturated(),P.setImmediate(t.process)})}var o=P.queue(t,e);return o.push=function(n,t,e){i(o,n,t,e)},delete o.unshift,o},P.cargo=function(n,t){return I(n,1,t)},P.log=x("log"),P.dir=x("dir"),P.memoize=function(n,e){var r={},u={},i=Object.prototype.hasOwnProperty;e=e||t;var o=m(function(t){var o=t.pop(),c=e.apply(null,t);i.call(r,c)?P.setImmediate(function(){o.apply(null,r[c])}):i.call(u,c)?u[c].push(o):(u[c]=[o],n.apply(null,t.concat([m(function(n){r[c]=n;var t=u[c];delete u[c];for(var e=0,i=t.length;i>e;e++)t[e].apply(null,n)})])))});return o.memo=r,o.unmemoized=n,o},P.unmemoize=function(n){return function(){return(n.unmemoized||n).apply(null,arguments)}},P.times=A(P.map),P.timesSeries=A(P.mapSeries),P.timesLimit=function(n,t,e,r){return P.mapLimit(f(n),t,e,r)},P.seq=function(){var t=arguments;return m(function(e){var r=this,u=e[e.length-1];"function"==typeof u?e.pop():u=n,P.reduce(t,e,function(n,t,e){t.apply(r,n.concat([m(function(n,t){e(n,t)})]))},function(n,t){u.apply(r,[n].concat(t))})})},P.compose=function(){return P.seq.apply(null,Array.prototype.reverse.call(arguments))},P.applyEach=T(P.eachOf),P.applyEachSeries=T(P.eachOfSeries),P.forever=function(t,e){function r(n){return n?i(n):void o(r)}var i=u(e||n),o=z(t);r()},P.ensureAsync=z,P.constant=m(function(n){var t=[null].concat(n);return function(n){return n.apply(this,t)}}),P.wrapSync=P.asyncify=function(n){return m(function(t){var e,r=t.pop();try{e=n.apply(this,t)}catch(u){return r(u)}U(e)&&"function"==typeof e.then?e.then(function(n){r(null,n)})["catch"](function(n){r(n.message?n:new Error(n))}):r(null,e)})},"object"==typeof module&&module.exports?module.exports=P:"function"==typeof define&&define.amd?define([],function(){return P}):C.async=P}();
@megawac
megawac Feb 15, 2016 Collaborator

This file should be deleted and moved to /lib

@aearly
aearly Feb 16, 2016 Collaborator

lib/ or dist/? I do wonder if there is value in having the minified build on npm.

@megawac
megawac Feb 16, 2016 Collaborator

Whoops dist/* and there might be if people do

<script src="./node_modules/dist/async.min.js"></script>
@jokeyrhyme

Travis CI tests are failing for Node.js 0.10.x:

FAILURES: Undone tests (or their setups/teardowns):

  • memoize - avoid proto key return undefined

Is it time to discuss ending support for Node.js 0.10.x?

@megawac megawac commented on the diff Feb 15, 2016
support/xyz.sh
@@ -0,0 +1,171 @@
+#!/usr/bin/env bash
@megawac megawac commented on the diff Feb 15, 2016
support/build/plugin-lodash-import-rename.js
@@ -0,0 +1,24 @@
+import _ from 'lodash';
@megawac
megawac Feb 15, 2016 Collaborator

do we still need this?

@aearly
aearly Feb 16, 2016 Collaborator

Yes, it changes our import foo from 'lodash' to use the correct ES version when performing the rollup.

@aearly
aearly Feb 16, 2016 Collaborator

Maybe I should write a comment.

@megawac
Collaborator

@aearly do we want to have a make file test which copies the test folder, renames the async.js import and tests the built file?

Collaborator

I added support/build.test.js in a later commit that does a simple smoke test of the build artifacts.

@megawac
Collaborator
megawac commented Feb 15, 2016

Is it time to discuss ending support for Node.js 0.10.x?

I don''t think so, lets work around it.

@aearly left some comments

@megawac megawac Skip failing test case
9688db3
@jdalton
Contributor
jdalton commented Feb 16, 2016

@jokeyrhyme

Is it time to discuss ending support for Node.js 0.10.x?

See stats from npm
https://twitter.com/seldo/status/672569831565078528
and Semaphore CI
https://twitter.com/JavaScriptDaily/status/677741377745240064

@jokeyrhyme

@jdalton woah. Depressing, but good to know!

@aearly
Collaborator
aearly commented Feb 16, 2016

Yep, lots of organizations haven't had time to migrate. And don't be sad, be glad they're not on 0.8!

@aearly
Collaborator
aearly commented Feb 16, 2016

Remaining questions:

  • Should we publish ES modules as 'async/es/foo' or in a separate package 'async-es/foo'?
  • Where should async.js and async.min.js(UMD builds) go?
  • build/dist/ or dist/ (for bower UMD support)?
@megawac
Collaborator
megawac commented Feb 16, 2016

@jdalton was 0.8 not included in that list?

@megawac
Collaborator
megawac commented Feb 16, 2016

Should we publish ES modules as 'async/es/foo' or in a separate package 'async-es/foo'?

I'd say separate modules for build size concerns -- few people will likely be using our /es stuff

Where should async.js and async.min.js(UMD builds) go? build/dist/ or dist/ (for bower UMD support)?

I would say both. Add a /dist folder so people can download directly off github and then publish a /build/dist folder as well so people can use the compiled file on node.

Question is should we make the mainfile for the repo dist/async.js or index.js?

Also we only publish the build folder to npm correct @aearly?

@charlierudolph
Contributor

lib/async.js needs to be removed.

charlierudolph added some commits Feb 16, 2016
@charlierudolph charlierudolph filter, reject, detect, some, every with error 10ac9e9
@charlierudolph charlierudolph fix readme
d611a47
@jdalton
Contributor
jdalton commented Feb 16, 2016

@megawac

was 0.8 not included in that list?

Not in semaphore's report but in npm's report Node v0.8 had 0.1%.

@aearly aearly Merge pull request #1028 from charlierudolph/cr-withErrorModular
filter, reject, detect, some, every with error
b73cedc
@aearly
Collaborator
aearly commented Feb 16, 2016

@megawac correct, we only publish what is in build. I'm going to separate out the ES modules into a separate folder so it can be a separate package.

aearly added some commits Feb 16, 2016
@aearly aearly removed old lib/async.js 430e42f
@aearly aearly move es build to separate folder/package
73d57d2
@aearly aearly remove esnext:main from package json 943dbed
@aearly
Collaborator
aearly commented Feb 16, 2016

https://www.npmjs.com/package/async-es

May as well claim it now. :P

@aearly aearly move build/async.js to build/dist/async.js
54411fc
@aearly
Collaborator
aearly commented Feb 16, 2016

Okay, I cleaned up the UMD dist/ and separated out the ES build. finger moves closer to merge button

@megawac
Collaborator
megawac commented Feb 16, 2016

I'll review again tonight

@aearly aearly referenced this pull request Feb 16, 2016
Closed

leverage lodash #914

@megawac
Collaborator
megawac commented Feb 16, 2016

Havent checked yet but do we test the compiled builds Alex?
On Feb 16, 2016 5:20 PM, "Alex Early" notifications@github.com wrote:

Okay, I cleaned up the UMD dist/ and separated out the ES build. finger
moves closer to merge button


Reply to this email directly or view it on GitHub
#996 (comment).

@aearly
Collaborator
aearly commented Feb 17, 2016

We test lib/ with nodeunit and mocha using babel to compile the ES modules. We have a simple smoke-test for the compiled output (CJS, UMD, and ES formats).

@megawac megawac commented on the diff Feb 17, 2016
support/module_template.md
@@ -0,0 +1,13 @@
+# async.<%= name %>
@megawac
megawac Feb 17, 2016 Collaborator

Do we want to keep this?

@aearly
aearly Feb 17, 2016 Collaborator

I don't think so. It's only there if we were to publish each function as its own package. (same thing with the Gulpfile)

@aearly
Collaborator
aearly commented Feb 18, 2016

Anymore comments? I suppose anything else could be fixed on master.

@jtwebman
Contributor

Lets merge it, it is big change :)

@megawac
Collaborator
megawac commented Feb 18, 2016

Do we want to add the build directories to gitignore first?

On Thu, Feb 18, 2016 at 1:45 PM, JT Turner notifications@github.com wrote:

Lets merge it, it is big change :)


Reply to this email directly or view it on GitHub
#996 (comment).

@aearly
Collaborator
aearly commented Feb 18, 2016

I guess now that the code review is complete we can ignore them.

@megawac
Collaborator
megawac commented Feb 18, 2016

The last question in my mind is what do we want our main file to be? The
built file (to avoid imports) or the CJS file (current)

On Thu, Feb 18, 2016 at 2:06 PM, Alex Early notifications@github.com
wrote:

I guess now that the code review is complete we can ignore them.


Reply to this email directly or view it on GitHub
#996 (comment).

@aearly
Collaborator
aearly commented Feb 18, 2016

The CJS index seems more "correct", but it would only be better in node-land. If you're browserify-ing the UMD build will be smaller, and it will still wok in node. I say we make dist/async.js the main.

$ browserify --standalone async build/index.js | uglifyjs -mc | gzip -9 | wc -c
11089

$ cat dist/async.min.js |  gzip -9 | wc -c
6680
@megawac
Collaborator
megawac commented Feb 18, 2016

For the browserify case, it also depends on whether the user is importing
modules which also depend on lodash.

On Thu, Feb 18, 2016 at 2:38 PM, Alex Early notifications@github.com
wrote:

The CJS index seems more "correct", but it would only be better in
node-land. If you're browserify-ing the UMD build will be smaller, and it
will still wok in node. I say we make dist/async.js the main.

$ browserify --standalone async build/index.js | uglifyjs -mc | gzip -9 | wc -c
11089

$ cat dist/async.min.js | gzip -9 | wc -c
6680


Reply to this email directly or view it on GitHub
#996 (comment).

@aearly
Collaborator
aearly commented Feb 18, 2016

If the browserify user was cherry-picking async methods, they'd get some overlap with lodash and some possible de-deuping, if they were also cherry-picking lodash methods. But if you require both async and lodash wholesale, you'll get no module overlap, and some code duplication. It's a tradeoff between code duplication and CJS module overhead.

@aearly aearly make the package json main dist/async.js
5da468f
@jdalton
Contributor
jdalton commented Feb 18, 2016

Just popping in, if you use browserify, are you using bundler-collapser? I know webpack generally produces smaller builds and has more flexibility in terms of module path hackery.

@aearly
Collaborator
aearly commented Feb 18, 2016
$ browserify --standalone async build/index.js | bundle-collapser | uglifyjs -mc | gzip -9 | wc -c
9393

Helps a little. It's also up to the browserify user to use bundle-collapser.

@jtwebman
Contributor

I thought browserify was going away. It seems more and more people only are using npm. Maybe that is just from what I have seen. Has anyone else noticed it?

@aearly
Collaborator
aearly commented Feb 19, 2016

Browserify is still alive and well, but whatever the case, here it's just a quick way to get a feel for what the module overhead is, regardless of what bundler you're using.

ecasilla and others added some commits Feb 20, 2016
@ecasilla ecasilla added a base implementation for unsaturation event #868
updating the README

fix readme
e642157
@aearly aearly Merge pull request #1030 from ecasilla/modularization
async.queue.unsaturated #868
3cf46d0
@aearly aearly changed the title from [WIP] Modularization to Modularization Feb 24, 2016
@aearly aearly added the enhancement label Feb 24, 2016
@aearly aearly gitignore build directories
070ed49
@aearly aearly merged commit 3d1781c into master Feb 24, 2016

3 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
coverage/coveralls Coverage increased (+2.8%) to 100.0%
Details
@megawac
Collaborator
megawac commented Feb 24, 2016

When do you want to release v2? and do we want to do a pre-release?

@martinheidegger

I am jumping on this bandwaggon waaay too late but: Why all this effort of splitting it out to own packages? This will make npm just slower. npm supports require('async/something') for files within the async repo. Async only needed to distribute the modules as.js` files in the tree:

- package.json
- parallel.js
- each.js
- serial.js
- ....

Then you could do require('async/parallel') would automatically just load the code required for parallel. This would significantly faster in every respect, wouldn't it?

@megawac
Collaborator
megawac commented Feb 24, 2016

Why all this effort of splitting it out to own packages?

We're not doing that at the moment, but the benefit is the download for an individual method is smaller than the entire async package

@jdalton
Contributor
jdalton commented Feb 24, 2016

Though with npm's cache, packages of a given version are only downloaded once.
Subsequent requests hit the local npm cache on your system.

@aearly
Collaborator
aearly commented Feb 24, 2016

I want to release 2.0 when all the issues in the milestone are complete. (All the breaking changes). I'd also like to get the docs migrated to jsdoc and the tests ported to Mocha, but those aren't required.

@martinheidegger

@megawac I do understand the need for splitting out the modules in own files/modules as this will reduce the size if used by packages such as a browserify but when downloading the files through npm this method should slow down things (considering the package.json and other overhead). Browserify (and babel afaik) do support deep binding require("<package>/<module>") so the bundled files would already become a lot smaller.

@megawac
Collaborator
megawac commented Feb 24, 2016

rowserify (and babel afaik) do support deep binding require("/") so the bundled files would already become a lot smaller.

Sure, assuming that none of the user's depencies require async and do require("async").

Also, at the moment babel does nothing in regards to import <package>/<module> besides transpile it to require("<package>/<module>"). Bundlers such as webpack, browserify, and rollup do preform this action

@aearly I think it makes sense to set a package.json field for "browser": "index.js", the reason being is webpack and rollup support tree shaking. This would allow the bundler to exclude unused modules (as if the user did require('async/some-module')

@martinheidegger

@megawac Users likely will always be able to do require("async") and blow up their dependencies.

@megawac megawac deleted the modularization branch Apr 12, 2016
@megawac
Collaborator
megawac commented Jul 13, 2016 edited

For reference, I measured the require time of using index.js vs dist/async in v2.

Env dist/async.js async/index.js
v0.12 8 ms 69ms
v4.4.7 9 ms 58ms
v6.2.2 11 ms 46ms

Methodology of test

I used time-require to make these measurements. I ran a tiny script

  require('time-require');

  var async = require('async');
  var async1 = require('async/index');

I ran this in v0.12, v4, and v6 installed via nvm 5 times each and took the average (tended not to deviate much) on my laptop

Computer

Costs involved loading async/index (node 6)

Note: I snipped a couple irrelevant parts of this log

Start time: (2016-07-13 22:37:29 UTC) [treshold=1%]
 #  module                          time  %
 5  ./isFunction (...sFunction.js)   1ms  ▇ 2%
 6  ./toNumber (no.../toNumber.js)   1ms  ▇ 2%
 7  ./toFinite (no.../toFinite.js)   1ms  ▇ 2%
 8  ./toInteger (n...toInteger.js)   1ms  ▇ 2%
 9  lodash/rest (n...dash/rest.js)   2ms  ▇▇ 4%
10  ./internal/app...applyEach.js)   3ms  ▇▇ 6%
11  ./_getLength (...getLength.js)   1ms  ▇ 2%
12  lodash/isArray...ArrayLike.js)   1ms  ▇ 2%
13  ./getIterator...tIterator.js)    1ms  ▇ 2%
14  ./isArray (nod...h/isArray.js)   1ms  ▇ 2%
15  ./_indexKeys (...indexKeys.js)   2ms  ▇▇ 4%
16  lodash/keys (n...dash/keys.js)   2ms  ▇▇ 4%
17  ./iterator (no.../iterator.js)   4ms  ▇▇▇ 8%
18  ./eachOfLimit...chOfLimit.js)    5ms  ▇▇▇ 10%
19  ./internal/doP...llelLimit.js)   6ms  ▇▇▇▇ 12%
20  ./mapLimit (no.../mapLimit.js)   6ms  ▇▇▇▇ 12%
21  ./map (node_mo...async/map.js)   6ms  ▇▇▇▇ 12%
22  ./applyEach (n...applyEach.js)   9ms  ▇▇▇▇▇▇ 18%
23  ./applyEachSer...achSeries.js)   1ms  ▇ 2%
24  ./asyncify (no.../asyncify.js)   1ms  ▇ 2%
25  ./_baseFor (no.../_baseFor.js)   1ms  ▇ 2%
26  lodash/_baseFo...aseForOwn.js)   1ms  ▇ 2%
27  ./auto (node_m...sync/auto.js)   1ms  ▇ 2%
28  lodash/_arrayM..._arrayMap.js)   1ms  ▇ 2%
29  ./_checkGlobal...eckGlobal.js)   1ms  ▇ 2%
30  ./_root (node_...ash/_root.js)   1ms  ▇ 2%
31  ./_Symbol (nod...h/_Symbol.js)   1ms  ▇ 2%
32  ./_baseToStrin...eToString.js)   1ms  ▇ 2%
33  ./_stringToArr...ngToArray.js)   1ms  ▇ 2%
34  lodash/trim (n...dash/trim.js)   2ms  ▇▇ 4%
35  ./autoInject (...utoInject.js)   4ms  ▇▇▇ 8%
36  ./internal/que...nal/queue.js)   1ms  ▇ 2%
37  ./cargo (node_...ync/cargo.js)   2ms  ▇▇ 4%
38  ./eachOfLimit...chOfLimit.js)    1ms  ▇ 2%
39  ./eachOfSeries...hOfSeries.js)   1ms  ▇ 2%
40  ./reduce (node...nc/reduce.js)   1ms  ▇ 2%
41  ./seq (node_mo...async/seq.js)   1ms  ▇ 2%
42  ./compose (nod...c/compose.js)   1ms  ▇ 2%
43  ./internal/doP...oParallel.js)   1ms  ▇ 2%
44  ./concat (node...nc/concat.js)   1ms  ▇ 2%
45  ./internal/doS.../doSeries.js)   1ms  ▇ 2%
46  ./concatSeries...catSeries.js)   1ms  ▇ 2%
47  ./constant (no.../constant.js)   1ms  ▇ 2%
48  lodash/identit.../identity.js)   1ms  ▇ 2%
49  ./internal/cre...ateTester.js)   1ms  ▇ 2%
50  ./detect (node...nc/detect.js)   2ms  ▇▇ 4%
51  ./doDuring (no.../doDuring.js)   1ms  ▇ 2%
52  ./eachLimit (n...eachLimit.js)   1ms  ▇ 2%
53  ./each (node_m...sync/each.js)   1ms  ▇ 2%
54  ./ensureAsync...sureAsync.js)    1ms  ▇ 2%
55  ./filter (node...nc/filter.js)   1ms  ▇ 2%
56  ./forever (nod...c/forever.js)   1ms  ▇ 2%
57  ./mapValues (n...mapValues.js)   1ms  ▇ 2%
58  ./memoize (nod...c/memoize.js)   1ms  ▇ 2%
59  ./internal/par.../parallel.js)   1ms  ▇ 2%
60  ./parallelLimi...llelLimit.js)   1ms  ▇ 2%
61  ./parallel (no.../parallel.js)   1ms  ▇ 2%
62  ./queue (node_...ync/queue.js)   1ms  ▇ 2%
63  ./priorityQueu...rityQueue.js)   1ms  ▇ 2%
64  ./reduceRight...duceRight.js)    2ms  ▇▇ 4%
65  ./reject (node...nc/reject.js)   1ms  ▇ 2%
66  ./rejectSeries...ectSeries.js)   1ms  ▇ 2%
67  ./retryable (n...retryable.js)   1ms  ▇ 2%
68  ./some (node_m...sync/some.js)   1ms  ▇ 2%
69  ./timeout (nod...c/timeout.js)   1ms  ▇ 2%
70  ./timesSeries...mesSeries.js)    1ms  ▇ 2%
71  ./internal/onl...nal/onlyOnce)   1ms  ▇ 2%
72  ./whilst (node...nc/whilst.js)   1ms  ▇ 2%
73  ./until (node_...ync/until.js)   1ms  ▇ 2%
74  async/index (n...ync/index.js)  44ms  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 86%
Total require(): 334
Total time: 51ms

P.s. I hope all your browsers support detail tags :)

Summary
Computer
Processor4x Intel(R) Core(TM) i5-4300M CPU @ 2.60GHz
Memory11988MB (7240MB used)
Operating SystemUbuntu 14.04.4 LTS
@aearly
Collaborator
aearly commented Jul 13, 2016

Cool! Thanks for backing up my gut assertions with some actual data. 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment