file transforms

[matthewmueller]

i've been thinking a lot about file transforms lately and I think I've come up with a decent solution. Here's what the API would look like:

duo
  .use(jade())
  .use(styl())

Where styl and jade are one of the following:

• a synchronous function that returns a string function(...) { return compiled }
• an asynchronous function: function(..., fn) { return fn(null, compiled) }
• a gulp stream

The reason I'm pushing for gulp streams is that it will allow us to tap into a huge repository of plugins for things like: transpiling ES6 => ES5 (adding import, etc), coffeescript support, and many other things I haven't thought of.

However, the problem I'm running into is the following. Maybe this is obvious to someone that's worked a lot with streams, but basically if you do the following, you will not get the intended results:

var fs = require('fs');
var map = require('map-stream');

function prepend(str, fn) {
  return fn(null, 'hi ' + str);
};

var t = map(prepend);

var a = fs.createReadStream('./a.txt', 'utf8');
var b = fs.createReadStream('./b.txt', 'utf8');
var c = fs.createReadStream('./c.txt', 'utf8');

a.pipe(t).pipe(process.stdout);
b.pipe(t).pipe(process.stdout);
c.pipe(t).pipe(process.stdout);

Basically, the transform stream pulls from all 3 sources at the same time. This is a problem because gulp streams are already initialized by the time they're passed through (ie. styl()).

What needs to happen is the files need to be queued and wait until the transform pipeline is free. I'm not sure if this is going to be a bottleneck yet, or if there is a better way, but that's what I'm facing right now.

I guess to get duo out there maybe we can just support the first 2 options (sync and async fn), but I think the gulp streams will be a huge plus for people.

---

How we'll walk the files is the following:

$ duo main.css

main.css:

@import "./one.styl"
@import "./two.less"
@import "./three"

To resolve three we will base it off the root entry (extname("main.css") => ".css" => "./three.css"). We will do the same for JS and eventually HTML.

/cc @yields @juliangruber @ianstormtaylor

[juliangruber]

in the example the transform only pulls from a

[matthewmueller]

woops, sorry, incomplete example. updated. working on it right now. I think I can get it working with an internal queue.

[juliangruber]

The api for streams we'd need is:

function styl(opts){
  return function(){
    return transformStream();
  }
}

[juliangruber]

I think in the example you'll also get each file content written to stdout 3 times, as there are 3 pipes from t to stdout

[matthewmueller]

then you'd need to wrap every gulp stream though

[matthewmueller]

haha i think that's my current bug :-P

[juliangruber]

var fs = require('fs');
var map = require('map-stream');

function prepend(str, fn) {
  return fn(null, 'hi ' + str);
};

function t(){
  return map(prepend);
}

var a = fs.createReadStream('./a.txt', 'utf8');
var b = fs.createReadStream('./b.txt', 'utf8');
var c = fs.createReadStream('./c.txt', 'utf8');

a.pipe(t()).pipe(process.stdout);
b.pipe(t()).pipe(process.stdout);
c.pipe(t()).pipe(process.stdout);

[matthewmueller]

i guess we could do something like...

duo
  .use(wrap(styl)(opts))

pretty ugly though

[yields]

To resolve three we will base it off the root entry (extname("main.css") => ".css" => "./three.css"). We will do the same for JS and eventually HTML.

this is already implemented in #duo-build branch.

a gulp stream

no clue on how gulp works, didn't have a chance to look into it, but if async will work how come streams won't work?

function plugin(opts){
  return function(duo, done){
    var stream = stuff(duo);
    stream.on('error', done);
    stream.on('close', done);
  };
}

no clue why streams will be useful, do people really want to transform coffee to js using a stream ?

[matthewmueller]

this is already implemented in #duo-build branch.

<3

no clue on how gulp works, didn't have a chance to look into it, but if async will work how come streams won't work?

the problem is that we're using the same instance of the transform over again. that's totally f-ing things up haha.

no clue why streams will be useful, do people really want to transform coffee to js using a stream ?

it allows you to do stuff like:

entry.js

var a = require('./a.coffee');

or

import a from 'yields:shortcuts'

which would get transformed into:

var a = require('yields:shortcuts').a;

which would then get picked up by file-deps as ["yields:shortcuts"].

That's the idea anyway, haha.

[matthewmueller]

the reason i want gulp is so we don't have to write all this crap ourselves.

[yields]

the reason i want gulp is so we don't have to write all this crap ourselves.

aha, fair enough +1 for streams then, don't want to write those things too ;)

[yields]

i have an idea that might make things easier:

go with .pipe(stream) && .use(async || sync).
do an ugly try catch to require gulp, wrap streams internally, when duo is ready call gulp with all the streams.

[matthewmueller]

hmm, could you show me in some code? i'm not sure i follow.

[matthewmueller]

ahh interesting, the files are read in levels, so I'm not sure that will work:

entry.js:

require('yields:shortcuts')
require('./a.coffee')
require('./b.coffee')

First recursion: parse yields:shortcuts, ./a.coffee, and ./b.coffee in parallel.

Which would result in:

parsing ./a.coffee:

  gulp('a.coffee').pipe(t)

parsing ./b.coffee:

  gulp('./a.coffee').pipe(t) // same instance of t

[matthewmueller]

i think i have a solution, so don't think too much about it ;-)

[matthewmueller]

okay this isn't going to happen haha. the main module people use for gulp-plugins map-stream doesn't allow you to call the same instance a second time. we'll have to wrap. I think getting gulp plugins working is lower priority now, at least for release. maybe just support sync and async functions based on argument length.

[yields]

sounds good, i don't mind releasing without gulp support mainly because wrapping them will be easy, almost like thunkify haha

hmm, could you show me in some code? i'm not sure i follow.

i was thinking we could just .push(stream) and stream.pause() and use them later on with require('gulp'), but last time i used .pause() node exploded, so not sure if this will work haha ;)

[matthewmueller]

haha yah streams right now is a total clusterfuck, with like half the modules still implementing streams1.

almost like thunkify haha

yep that's what i was thinking. i'll come up with an example of using gulp in duo, you want to take the duo#use(fn) implementation ?

[yields]

you want to take the duo#use(fn) implementation ?

yup :)

[yields]

add duo.use()