Skip to content
🚰 binary data "streams+" via data producers, data consumers, and pull flow.
JavaScript C++ C Python
Branch: master
Clone or download
Latest commit 163cc17 Jan 8, 2020
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
addons
diagrams Add diagrams, feature the full diagram in the readme Oct 12, 2018
experiments add experiments/ Oct 5, 2018
helpers buffer-source: fix again Jan 8, 2020
profiles add zlib profiles Feb 16, 2018
tests deps: do not rely on bob-status Jan 8, 2020
.gitignore git: ignore upcomming test artefacts Jul 11, 2019
.travis.yml ci: add travis Jul 11, 2019
contributing.md work thus far Oct 23, 2017
index.js exports: re-export status_type as Status Jan 8, 2020
license work thus far Oct 23, 2017
package-lock.json 0.4.2 Jan 8, 2020
package.json 0.4.2 Jan 8, 2020
performance.md general project cleanup Feb 20, 2018
readme.md docs: update readme Jul 22, 2019
reference-buffered-transform.js lib: move to Status over status_type Jan 8, 2020
reference-extension-stop.js lib: move to Status over status_type Jan 8, 2020
reference-passthrough.js tests: automated tests & linting Jul 15, 2019
reference-sink.js lib: move to Status over status_type Jan 8, 2020
reference-source.js
reference-status-enum.js exports: re-export status_type as Status Jan 8, 2020
reference-verify.js lib: move to Status over status_type Jan 8, 2020

readme.md

BOB

A binary data "streams+" API & implementations via data producers, data consumers, and pull flow.

The name? BLOB β€” Matteo Collina.

Bytes Over Buffers β€” Thomas Watson

This is a Node.js strategic initiative aiming to improve Node.js streaming data interfaces, both within Node.js core internally, and hopefully also as future public APIs.

Flow of data & errors though BOB sinks & sources

Published Modules

The following modules contain usable components (sources, sinks, or transforms) and are published to npm.

The following modules are not published but are 'functional'.

API Reference

The following files serve as the API's reference:

Examples

The composition of the classes looks like this:

const { Stream } = require('bob-streams')

const source = new Source(/* args */)
const xform = new Transform(/* args */)
const sink = new Sink(/* args */)

const stream = new Stream(source, xform, sink)
stream.start(error => {
  // The stream is finished when this is called.
})

An entire passthrough could look like this:

class PassThrough {
  bindSource (source) {
    source.bindSink(this)
    this.source = source
    return this
  }

  bindSink (sink) {
    this.sink = sink
  }

  next (status, error, buffer, bytes) {
    this.sink.next(status, error, buffer, bytes)
  }

  pull (error, buffer) {
    this.source.pull(error, buffer)
  }
}

API Extension Reference

The following files serve as API extension references:

  • extension-stop - Tell a source to stop.
    • Useful for dealing with timeouts on network APIs.

Project Approach

High-level timeline:

  • Prototype separate from core entirely.
  • Move into nodejs org once JS & C++ APIs are significantly prototyped.
  • Begin transitioning Node.js internals once the APIs and perf are proved.
  • If an internal transition works out well, begin planning public APIs.

All of these steps necessitate the buy-in of many stakeholders, both in Node.js core and the greater Node.js ecosystem. This is a long-term project by necessity and design.

Goals

Some collective goals for this initiative.

  • Both performance and ease-of-use are key.
  • Implementable in a performant and usable way for both JS and C++.
  • Browser portability is preferable.

Protocol

As a preface, "protocol" refers to a system with "producer / source" and "consumer / sink" endpoints.

The Protocol itself must be simple:

  • Pull-based: The consumer requests ("pulls") data from the producer.
  • Binary-only: Data is binary buffers only, "object mode" and string encodings are not supported at the protocol level.
  • Stateless: The protocol must not require state to be maintained out-of-band.
    • Non-normative: While the protocol itself does not require out-of-band state, actual operations almost always do.
    • Minimize state assumed between calls.
  • One-to-one: The protocol assumes a one-to-one relationship between producer and consumer.
  • Timing agnostic: The protocol makes no timing (sync or async) assumptions.
  • No buffering: The protocol must not require buffering (although specific implementations might).
    • Non-normative: While the protocol itself does not require buffering, starting sources almost always do (including transforms).
  • In-line errors and EOF: Errors, data, and EOF ("end") should flow through the same call path.

Consumer

  • Should make no assumption on the timing of when data will be received (sync or async).
  • Should own any preallocated memory (the buffer).
  • Must never make more than one data request upstream at the same time.

Performance

Please see performance.md for profiling results & information.

Current results estimate a 30% decrease of CPU time in bad cases, and up to 8x decrease in good cases. This should correlate to overall throughput but may not be exact.

Project Layout

API reference examples sit in the top-level directory and are prefixed by reference-. These are functional and tested when practical, notably reference-verify, reference-passthrough, and verify-buffered-transform.

Other helpers, such as Stream(), reside in the /helpers/ and /tests/helpers directories. All useful and usable components in this repo are exported from index.js with the bob-streams npm module.

Functional sources, sinks, and so on can be found in their own npm modules. See [Published Modules](#Published Modules).

Development

Tests

npm install && npm test

Building the addons

The addons are presently very out-of-date.

You must have a local install of Node master @ ~ 694ac6de5ba2591c8d3d56017b2423bd3e39f769

npm i node-gyp
node-gyp rebuild --nodedir=your/local/node/dir -C ./addons/passthrough
node-gyp rebuild --nodedir=your/local/node/dir -C ./addons/fs-sink
node-gyp rebuild --nodedir=your/local/node/dir -C ./addons/fs-source

License

MIT Licensed β€” Contributions via DCO 1.1

You can’t perform that action at this time.