Proposal for lifted pipelines
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information. Added missing syntax highlighting to code block Aug 4, 2018

Lifted Pipeline Proposal

  1. TL;DR: Just cut to the chase - I don't have time for a long explanation
  2. Introduction
  3. Pipeline lifting
  4. Pipeline combining
  5. Pipeline chaining
  6. Why operators, not functions?
  7. Why this? We already have .map/.filter/etc...
  8. Possible expansions
  9. Inspiration
  10. Related strawmen/proposals

TL;DR: Just cut to the chase ()

  1. Pipeline lifting for simple .map/.then-like stuff, using coll :> func + @@lift.
  2. Pipeline combining for simple .merge/.combine-like stuff, using Object.combine(...colls, func) + @@combine.
  3. Pipeline chaining for things like .filter/.takeWhile/.flatten, using coll >:> func + @@chain.
  4. Async variants exist for each, via coll :> async func/Object.asyncCombine/coll >:> async func, with matching @@async{Lift,Combine,Chain} symbols for each.
  5. coll :> await funcawait (coll :> async func), coll >:> await funcawait (coll >:> async func).

If you have a little more time... ()

The proposal is in three parts:

  1. Pipeline lifting:

    • New operator coll :> func, which calls coll[Symbol.lift](func).
    • New symbol @@lift to control the above.
    • For arrays, this is like .map.
    • For promises, this is like .then.
    • For functions, this is like composition.
  2. Pipeline combining:

  3. Pipeline chaining:

Async variants exist for each:

  • coll :> async func calls Symbol.asyncLift instead of Symbol.lift.
  • Object.asyncCombine calls Symbol.asyncCombine instead of Symbol.combine.
  • coll >:> async func calls Symbol.asyncChain instead of Symbol.chain.
  • Each of these wrap their callback to return a promise unconditionally, and they all themselves return a promise to the result.

The two operator variants can use await instead of async in async functions as sugar:

  • coll :> await funcawait (coll :> async func).
  • coll >:> await funcawait (coll >:> async func).

Introduction ()

Original es-discuss thread (previously, this was specific to function composition, but I've since generalized it.)

Before I continue, if you came here wondering what the heck this is, or what the point of it is, I invite you to read this blog post about composition and this one on monads, and I encourage you to google both concepts. Long story short, yes, it's a thing, and yes, it's pretty useful for a variety of reasons.

Also, note that this is meant to work as a complementary extension of the existing pipeline operator proposal. I use the F# variant here for simplicity, but there are still multiple competing syntaxes.

Function composition has been used for years, even in JS applications. It's one thing people have been continually reinventing as well. Many utility belts I've found have this function – in particular, most common ones have it:

There's also the numerous npm modules and manual implementations (it's trivial to write a basic implementation). Conceptually, it's pretty basic:

function compose(f, g) {
    return (...xs) => f(g(...xs))

It lets you do turn code like this:

function toSlug(input) {
    return encodeURIComponent(
        input.split(" ")
            .map(str => str.toLowerCase())

to this:

const toSlug = [
    _ => _.split(" "),
    _ => => str.toLowerCase()),
    _ => _.join("-"),

Or, using this proposal:

const toSlug =
    _ => _.split(" ")
    :> _ => => str.toLowerCase())
    :> _ => _.join("-")
    :> encodeURIComponent

Another scenario is when you just want to define a long stream:

// RxJS Observables
import {Observable} from "rxjs"

function randInt(range) {
    return Math.random() * range | 0

function getSuggestions(selector) {
    const refreshElem = document.querySelector(".refresh")
    const baseElem = document.querySelector(selector)

    const refreshClickStream = Observable.fromEvent(refreshElem, "click")

    const responseStream = refreshClickStream.startWith()
        .map(() => `${randInt(500)}`)
        .flatMap(url => Observable.fromPromise(
            window.fetch(url).then(response => response.json())
        .map(() => listUsers[randInt(listUsers.length)])

    return Observable.fromEvent(baseElem, "click")
    .combineLatest(responseStream, (_, listUsers) => listUsers)
    .merge( => undefined).startWith(undefined))
    .map(suggestion => ({selector, suggestion}))

Observable.of(".close1", ".close2", ".close3")
.flatMap(selector => getSuggestions(selector))
.forEach(({selector, suggestion}) => {
    if (suggestion == null) {
        // hide the selector's suggestion DOM element
    } else {
        // show the selector's suggestion DOM element and render the data

Problem is, there's this massive boilerplate, complexity, and jQuery-like tendencies inherent with nearly every reactive library out there. RxJS has attempted to compromise with a .do(func)/.let(func) that's the moral equivalent of a |> operator, but even then, using custom operators doesn't feel as natural as built-in ones. (jQuery and Underscore/Lodash have similar issues here, especially jQuery.) Using this proposal (all three parts) + the pipeline operator proposal + the observable proposal, this could turn out a bit easier and lighter:

// This proposal (35 SLoC)
function randInt(range) {
    return Math.random() * range | 0

function eachEvent(elem, event) {
    return new Observable(observer => {
        const listener = e =>
        elem.addEventListener(event, listener, false)
        return () => elem.removeEventListener(event, listener, false)

const refreshElem = document.querySelector(".refresh")
const refreshClickStream = fromEvent(refreshElem, "click")
const listUsers = (async () => refreshClickStream
    :> `${randInt(500)}`
    :> await () => window.fetch(url)
    :> await async response => response.json()
    :> listUsers => listUsers[randInt(listUsers.length)])

Observable.of([".close1", ".close2", ".close3"])
:> selector => document.querySelector(selector)
>:> (async elem => Object.combine(
    refreshClickStream :> () => ({elem})
        fromEvent(baseElem, "click"), await listUsers,
        (_, suggestion) => ({elem, suggestion})
|> o => o.subscribe({elem, suggestion}) => {
    if (suggestion != null) {
        // show the selector's suggestion DOM element and render the data
    } else {
        // hide the selector's suggestion DOM element

For comparison, here's the RxJS and callback equivalents:

// RxJS (29 SLoC + dependency)
import {Observable} from "rxjs"

function randInt(range) {
    return Math.random() * range | 0

function getSuggestions(selector) {
    const refreshElem = document.querySelector(".refresh")
    const baseElem = document.querySelector(selector)

    const refreshClickStream = Rx.Observable.fromEvent(refreshElem, "click")

    const responseStream = refreshClickStream.startWith()
        .map(() => `${randInt(500)}`)
        .flatMap(url => Rx.Observable.fromPromise(
            window.fetch(url).then(response => response.json())
        .map(listUsers => listUsers[randInt(listUsers.length)])

    return Rx.Observable.fromEvent(baseElem, "click")
    .combineLatest(responseStream, (_, listUsers) => listUsers)
    .merge( => undefined).startWith(undefined))
    .map(suggestion => ({selector, suggestion}))

Rx.Observable.of(".close1", ".close2", ".close3")
.flatMap(selector => getSuggestions(selector))
.subscribe(({selector, suggestion}) => {
    if (suggestion == null) {
        // hide the selector's suggestion DOM element
    } else {
        // show the selector's suggestion DOM element and render the data

// Callbacks (31 SLoC)
function randInt(range) {
    return Math.random() * range | 0

const refresh = document.querySelector(".refresh")

function getSuggestions(selector, send) {
    const elem = document.querySelector(selector)
    send(elem, undefined)

    async function getUsers() {
        const url = `${randInt(500)}`
        const response = await window.fetch(url)
        const listUsers = await response.json()
        return listUsers[randInt(listUsers.length)]

    let current = await getUsers()
    send(elem, current)

    refresh.addEventListener("click", () {
        current = undefined
        send(elem, undefined)
        send(elem, current = await getUsers())
    }, false)

    elem.addEventListener("click", () => send(elem, current), false)

for (const selector of [".close1", ".close2", ".close3"]) {
    getSuggestions(selector, (elem, suggestion) => {
        if (suggestion != null) {
            // show the first suggestion DOM element and render the data
        } else {
            // hide the first suggestion DOM element

It's not much longer than the RxJS variant, but critically, it has zero dependencies beyond polyfills

Pipeline lifting ( | )

So, we've got several ways of transforming values within things:

  • => f(x)) - Transform the entries of an array.
  • promise.then(x => f(x)) - Transform the value of a promise.
  • => f(x)) - Transform the values emitted from an observable.
  • stream.pipe(map(x => f(x))) - Transform the values in a Node stream (where map is through2-map).
  • func.compose(x => f(x)) - Transform the return value of a function (where .compose is a theoretical Function.prototype.compose).

If you squint hard enough, they are all variations of this same theme: object.transform(x => f(x)). What does that bring us?

  • A way to generically map over something without having to care so much about what's in it.

My proposal for this is to add a new syntax with an associated symbol:

// What you write:
x :> f

// What it does:
function pipe(x, f) {
    if (typeof func !== "function") throw new TypeError()
    return x[Symbol.lift](x => f(x))

It doesn't look like much, but it's incredibly useful and freeing with the right method implementations.

  • Want to get all the names out of an array of records? Use array :> r =>
  • Want to get a stream of input values from an event stream? Use stream :> e =>
  • Is the function returning an object you only want the contents of? Use func :> r => r.contents.
  • Want to shoehorn a function that takes a value and make it take events instead? Use e => :> setValue.
  • Have a Set of numbers and strings, and you only want numbers? Use set :> Number

If you want to dig deeper into what this really does and what all it entails, this contains more details on the proposal itself.

Pipeline combining ( | )

Sometimes, you might have a couple collections, promises, or whatever things you have that hold data, and you want to combine them. You want to join them. This .combineLatest looks like your sweet spot. Or maybe Bluebird's Promise.join is that missing piece you were looking for. Or maybe, you just wanted to run through a couple lists without pulling your hair out. That's what this is for. It takes all those nice and helpful things, and lifts them up to where the language understands it itself. Fewer nested loops, easier awaiting, and easier zipping iterables (which is harder than it looks to do correctly).

When you squint hard enough, these start to run together, and it's why I have this:

My proposal is to add a couple new builtins with related symbols:

// Combines each of `...args` using their related `Symbol.combine` method
Object.combine(...args, (...values) => ...)

// Combines each of `...args` using their related `Symbol.asyncCombine` method,
// returning a promise resolved with the return value
Object.asyncCombine(...args, (...values) => ...)

These are pretty straightforward, and their comments explain the gist of what they do. If you want more details about this proposal, or just want to read a little deeper into what the implementation might look like, take a look here.

Pipeline chaining ( | )

Of course, mapping and combining things is nice, but they're weak sauce. They do nothing to go "no more", and they offer no facility to go "nope, not passing that along". They also don't let you go "hey, add this into the mix, too". .map isn't enough; you want more. You want to not simply combine, but also flatten, but also filter. That's where this comes in.

After doing a bit of research to see what they really build off of, I managed to narrow it down to a single operation. Here's how I formulated that into a proposal:

// What you write:
x >:> f

// What this does (roughly):
function chain(x, f) {
    if (typeof func !== "function") throw new TypeError()
    return x[Symbol.chain](value => {
        const result = f(value)
        // break
        if (result == null) return undefined
        // emit values (optimization)
        if (Array.isArray(result)) return result
        // flatten value
        if (typeof result[Symbol.chain] === "function") return result
        throw new TypeError("invalid value")

It's not as simple and foolproof to implement as the first two, but here's how you use it:

  • If you want to break, you return null/undefined.
  • If you want to emit raw values, you return an array of them.
  • If you want to emit values from a chained object (usually same type as the collection), you return it.

This helper makes it possible to filter, flatten, and truncate things generically. For example, the common .takeWhile you find for collections and observables could be generically translated into a very simple helper:

// Use like so: `coll >:> takeWhile(cond)`
function takeWhile(cond) {
    return x => cond(x) ? [x] : undefined

This isn't the only one, there's several other helpers that become trivial to write, which may change how you find yourself manipulating collections in some cases.

Also, there is an async variant that awaits both the result and its callbacks before resolving, coming in two flavors: x >:> async func (returns promise) and x >:> await func (for async/await, awaits result). This variant is itself non-trivial, not because the basic common functionality is complex, but due to various edge cases, and it's the only non-trivial facet of this entire proposal.

Why this? We already have .map/.filter/etc... ()

It's much more general and open. I wanted to seek the lowest denominator for working with collections, one that keeps the user's cognitive overhead at a minimum and one that didn't make the implementation of various operators difficult. Yeah, it's nice to have high-level operators as methods, but I wanted to provide the means for people to define what looping is like for their types, so they can be looped over similarly to iterables and friends, with all the same set of control flow possibilities. Specifically:

  • I wanted to keep it flexible and meaningful semantically.

    • Some things are easily mapped over, but not everything is meaningfully combined. A good example of this is with functions. (You could implement such a method for them, but it's not really useful in practice, just in theory.)
    • Some things are easily combined, but not everything is meaningfully chained or even manipulated. A good example of this is with a validation object, where you either have a single value or a list of errors. (In fact, you can't chain this one like you can with promises - chained values can depend on previously chained results, while failed validations don't have a result you can chain.)
    • Some things can be not only combined, but even filtered and finitely extended, but not flattened. One example of this is a named, managed, observable slot. If you don't control it, you can't chain it without loss of information. (In this case, it'd be appropriate to just throw if you receive a non-array, non-undefined value.)
  • I wanted to decouple the object from the action, and more so, the type from the action.

    • The reason why the iterable interface is so attractive to implement is because people can then just use it natively in for loops and the like. You don't need to make assumptions about the object you have apart from that one clear interface. (If they have a weird .forEach or a non-traditional .filter, you don't need to care about that.)
    • Picking an interface for people to implement makes it much easier to avoid the long monstrosity of overly-specific methods that plague jQuery, Lodash, Underscore, RxJS, BaconJS, Bluebird, and nearly every other utility belt out there.
    • Having the type decoupled from the action helps make it easier to write such utility methods yourself and share them elsewhere.
  • I wanted to pick the minimal pragmatic solution that solved the problem as a whole.

    • There's a key difference between bolting on interface after interface to add missing features and actually thinking through your programming model before implementing it.
    • The minimal solution does not keep people from implementing utility functions themselves. In fact, the ability to recreate those methods is an explicit goal of this entire proposal. (If you can reimplement Array.prototype.filter's basic functionality with this proposal, that's a good thing.)
    • The "minimal" part shouldn't be considered relative to each part, but to the sum of them. Adding 15 relatively simple interfaces is worse than 3 slightly more complicated interfaces that can cover the same ground.
    • The minimal solution should consider ease of use and implementation, not just ease of modeling - if a marginal increase in a feature's complexity makes the user's code complexity substantially lesser, I would consider that a reduction towards a minimal solution.
    • The minimal solution should consider ease of use and implementation over elegance of the surrounding model. Even if something looks like a hack, if it makes 99% of cases simple, it doesn't need to be the perfect work of art just to cover that 1% of remaining cases. Lambda calculus and its "elegant"-yet-counterintuitive model isn't the ideal here, and it doesn't make much sense in a dynamically typed programming language to try to emulate System F<: or CoC just because they mathematically look cool. (Sorry, Fantasy Land fans.)

Of course, partial userland solutions have existed for a while for several of these issues with compatibility and extensibility (for observables + variant, many basic data structures, thenables, iterables + async variant), but this is an attempt to unify most of these under a single umbrella in a way that feels like JS, something that just fits right in without being too out there and unusual (especially in light of other recent proposals). Furthermore, even though it is possible to implement this in userland, it's not ideal, hence the need for a new standard:

  1. Most in-language implementations of function composition involve a .reduce or equivalent out of necessity since they are almost always variadic. In this scenario, engines commonly end up seeing the value as megamorphic, whether in for or the native .reduce, because there's only two independent IC feedback points, and because of this, it ends up hitting the slow path every single time. A native assist would be invaluable for this.

  2. Engines have had so much trouble with optimizing Array builtins in the past, and userland implementations are even slower than that. With this proposal, the intermediate values are inaccessible unless the symbols are overridden, making optimization opportunities easier.

    • V8 only just recently managed to crack the nut of inlining array methods like .map and .filter while eliding the intermediate function allocation and has-property check, because they can't just naïvely do a for loop - they also have to skip missing properties. And given the fact loop bodies can delete elements mid-loop, they had to first provide the ability to bail out from within existing optimized code into a slow path corresponding that same segment of optimized code, which is easier said than done. (No other engine does this IIUC.)
    • No engine AFAIK currently reuses existing intermediate arrays, although I've mentioned in an aside in this V8 bug that it's possible given the right set of ICs and static analysis checks.
    • In fact, for builtins (like arrays and iterables), it can frequently just merge two pipeline chains into a single callback internally. This is part of why I designed the proposal the way I did - engines don't need massive amounts of static analysis for massive gains.
  3. Userland standards tend to be much better at working us into this ugly problem. We need fewer of those.

    • For one, as a library writer, figuring out what the heck to implement for things that are kind of iterable-ish, but not in a way I can just implement Symbol.iterator, leads me to write the same exact methods about 5 times over for repeated variants.
    • Most existing "standards" for streams are so special-cased to a single type of library (monadic streams) that it occludes creating another form entirely (arrow-like streams).
    • The only real "standard" for non-stream collection-like constructs that aren't necessarily iterable is with Fantasy Land, and it doesn't always pick the most efficient way of specifying the various constructs. (Church-encoding the results over just using {done, value}, really?)
    • If the system is broken and impossible to fix, it's best to just throw it all away and start over.

Why operators, not functions? ()

I know it's a common criticism that function composition, and even this proposal as a whole, doesn't need new syntax. There are in fact tradeoffs involved. But here's why I elected to go with syntax:

  • This is meant to mirror the pipeline operator in appearance. It's not an exact one-to-one correspondence, but I specifically want to encourage people to view it as not dissimilar to a pipeline.

  • There's fewer parentheses and tokens in general involved, especially if the operator is lower-precedence. Instead of a pair of parentheses for each chain + commas for each call, it's a single infix token. Also, there's fewer cases of nested parentheses, something that tends to plague functional JS.

    • I know this is subjective, but I'm not alone in viewing this as a benefit for proposals. It's also no accident that a lot of functional JS fans end up making larger use of functional composition than even Haskell users - it reduces the sheer number of nested parentheses they frequently run into.
  • There's less to polyfill, since you only need a statically analyzable runtime helper. The binary nature of the operators make it possible to not also have to account for a variadic application. (The Object.combine and Object.asyncCombine implementations are good examples of why this is the case.)

  • Operators are in their nature less verbose than functions, and in general, this proposal aims to keep things simple without getting too verbose. It also tries to keep from becoming unreadable, and line noise is something I wish to avoid. (In fact, the proposal tries to avoid being too tacit, requiring you to be explicit what you do at each step.)

And of course, there are downfalls to using syntax to express this:

  • It's not possible to use with polyfills alone. This alone will draw people away from this proposal, because they either strongly resent transpiling in general, or they just don't want to have to add yet another Babel plugin just to use it. Trust me, I get the pain, too. I've written entire 50K+ SLoC projects solo targeting ES5, and I've have historically had very little need for the ES6+ syntax additions. (About the only things I've really found myself wanting are generators, arrow functions, and async/await.) And yes, transpilers are usually a pain to set up, especially Babel and TypeScript and especially with existing projects with complex build systems.

  • The operator looks a bit foreign and/or weird. I'm fully aware the operator looks pretty arcane if you're just looking at it without added context, and doesn't obviously imply any sort of substantially modified pipeline. I'm not wanting to design a Haskell or Perl extension, so if you have better ideas, please tell me. I really want to hear it.

    • My base requirements for the operator are that a) it doesn't conflict with existing syntax, b) it implies some sort of piping, and c) it implies some sort of obvious, clear direction.
    • Keep in mind, I've already gone through a few ideas:
      • >=> is too close to Haskell's Kleisli composition operator visually, which is equivalent to composing Symbol.chain callbacks to make a new such callback. (I initially proposed this in the mailing list, and it confused functional people.)
      • >:> was initially used for the base pipeline operator here, but it started to look a little too Perl-like, and was a little too verbose to merit being "better" than a simple utility function.
      • ->> was once proposed, but the reverse conflicts with unary negation (think: f <<- g vs f << -g). It also doesn't visually imply piping, even though it does imply a direction.
      • >> and >>> may seem incredibly obvious, but you can't use them without potentially breaking a lot of existing code (they are the bit-wise arithmetic and logical right shifts, respectively). | also falls in a similar wheelhouse as the bitwise exclusive or operator (it also happens to be a major asm.js dependency).
    • I previously had reverse equivalents for each, but I decided against it since it's not that hard to just flip the application, and it doesn't seem to fit well with the existing base pipeline operator proposal.
  • It adds to an already-complicated language, and can be fully implemented in userland without the assistance of operators. Most things that are userland-implementable don't really need to become language constructs, and very few things would actually benefit from being a core extension.

    • If you've been active or watching es-discuss for a while, you may also have had me bring up this particular email more than once. I really don't like the idea of adding substantial syntax or even new major builtins unless there are equal or greater amounts of opportunity to be gained from it. In fact, this is why I have been very cautious in how I formulated this proposal.
    • If you come from an object-oriented or procedural background and don't find yourself doing a lot of transforming on lists and/or working on data in the abstract, I can understand how this wouldn't affect you as much. This is especially true if you do mostly computationally-intensive stuff like numerical computation, games, and front-end view libraries/frameworks, where every allocation is very costly, or highly inherently stateful stuff like CRUD apps, where object-oriented programming fits your class-based domain model like a glove. (Sometimes, Rails + Backbone is the perfect combo for your app, since it's pretty much a giant interactive multi-user database with little else short extra little features.)

Possible expansions ()

These are just ideas; none of them really have to make it. ()

An to provide as an escape hatch which also facilitates optional propagation through the various operators. Here's an example with help from the optional chaining proposal:

// Old
function getUserBanner(banners, user) {
    if (user && user.accountDetails && user.accountDetails.address) {
        return banners[user.accountDetails.address.province];

// New
function getUserBanner(banners, user) {
        :> p => banners[p]
  • This gets uncomfortably verbose...

    • It is a little more flexible and not quite as common, which helps offset this some.
  • null values are censored to undefined for consistency.

    • The DOM already just censors undefined to null, so there's nothing to account for there.
  • The returned object's prototype would implement the following:

    • .value: get the underlying value
    • Symbol.iterator: yield the underlying value if not undefined, then return
    • Symbol.lift:
      • If this' underlying value is undefined, return this.
      • Else, call the callback with the value, box the result, and return it.
    • Symbol.asyncLift:
      • If this' underlying value is undefined, return this.
      • Else, call the callback with the value, await and box the result, and return the promise to it.
    • Symbol.combine:
      • If this' or other's underlying value is undefined, return this.
      • Else, call the callback with both values, box the result, and return it.
    • Symbol.asyncCombine:
      • If this' or other's underlying value is undefined, return Promise.resolve(this).
      • Else, call the callback with both values, await and box the result, and return the promise to it.
    • Symbol.chain:
      • If this' underlying value is undefined, return this.
      • Else, call the callback with the value, then:
        • If the result is a boxed value, return it directly.
        • Else, box the result and then return it.
    • Symbol.asyncChain:
      • If this' underlying value is undefined, return Promise.resolve(this).
      • Else, call the callback with the value, await the result, then:
        • If the result is a boxed value, return a promise to it directly.
        • Else, box the result and then return a promise to it.
  • Engines could with type feedback elide the entire pipeline and generate optimal assembly (think: zero-cost abstraction in optimized code) with little effort provided ICs assert and the relevant symbols aren't touched.

    • JS could use a few more zero-cost abstractions...

Cancellation proxying ()

Depending on whether cancellation turns out to include sugar syntax, this could hook into and integrate with that, adding an extra optional argument to all symbol hooks (like Symbol.lift, etc.) to allow handling cancellation (if they support it). This could allow much better cleanup in the face of cancellation, like closing sockets or aborting long polling loops.

Inspiration ()

Related strawmen/proposals ()

This is most certainly not on its own little island - even the introduction shows this. Here's several other existing proposals that could potentially benefit, or in some cases, be truly amplified, from this proposal, whether via being able to integrate with this well to its benefit, enhancing and complementing this proposal itself, or just being generally useful alongside it: