Skip to content

Commit

Permalink
Add batteries-included package
Browse files Browse the repository at this point in the history
  • Loading branch information
mtth committed Jun 23, 2019
1 parent 9767b99 commit b5e61ea
Show file tree
Hide file tree
Showing 7 changed files with 308 additions and 23 deletions.
134 changes: 113 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,37 +3,129 @@
Pure JavaScript implementation of the [Avro
specification](https://avro.apache.org/docs/current/spec.html).

## Avro types [![NPM version](https://img.shields.io/npm/v/@avro/types.svg)](https://www.npmjs.com/package/@avro/types) [![Download count](https://img.shields.io/npm/dm/@avro/types.svg)](https://www.npmjs.com/package/@avro/types)
## Features

+ Blazingly [fast and compact][benchmarks] serialization! Typically faster than
JSON with much smaller encodings.
+ All the Avro goodness and more: [type inference][type-inference], [schema
evolution][schema-evolution], [arbitrary JavaScript object
serialization][logical-types]...
evolution][schema-evolution], and [remote procedure calls][rpc].
+ Support for [serializing arbitrary JavaScript objects][logical-types].
+ Unopinionated [64-bit integer compatibility][custom-long].

## Installation

For the entire distribution:

```bash
$ npm install avsc
```

The functionality is also available via individual modules:

+ Avro types [![NPM version](https://img.shields.io/npm/v/@avro/types.svg)](https://www.npmjs.com/package/@avro/types) [![Download count](https://img.shields.io/npm/dm/@avro/types.svg)](https://www.npmjs.com/package/@avro/types)
+ Avro services [![NPM version](https://img.shields.io/npm/v/@avro/services.svg)](https://www.npmjs.com/package/@avro/services) [![Download count](https://img.shields.io/npm/dm/@avro/services.svg)](https://www.npmjs.com/package/@avro/services)
+ Avro streams [![NPM version](https://img.shields.io/npm/v/@avro/streams.svg)](https://www.npmjs.com/package/@avro/streams) [![Download count](https://img.shields.io/npm/dm/@avro/streams.svg)](https://www.npmjs.com/package/@avro/streams)
+ Avro IDL [![NPM version](https://img.shields.io/npm/v/@avro/idl.svg)](https://www.npmjs.com/package/@avro/idl) [![Download count](https://img.shields.io/npm/dm/@avro/idl.svg)](https://www.npmjs.com/package/@avro/idl)

## Documentation

+ [Home][home]
+ [API](https://github.com/mtth/avsc/wiki/API)
+ [Quickstart](https://github.com/mtth/avsc/wiki/Quickstart)
+ [Advanced usage](https://github.com/mtth/avsc/wiki/Advanced-usage)

## Examples

Inside a node.js module, or using browserify:

```javascript
const {Type} = require('@avro/types');

const type = Type.forSchema({
type: 'record',
fields: [
{name: 'kind', type: {type: 'enum', symbols: ['CAT', 'DOG']}},
{name: 'name', type: 'string'}
]
});

const buf = type.toBuffer({kind: 'CAT', name: 'Albert'}); // Encoded buffer.
const val = type.fromBuffer(buf); // = {kind: 'CAT', name: 'Albert'}
const avro = require('avsc');
```

## Avro services [![NPM version](https://img.shields.io/npm/v/@avro/services.svg)](https://www.npmjs.com/package/@avro/services) [![Download count](https://img.shields.io/npm/dm/@avro/services.svg)](https://www.npmjs.com/package/@avro/services)
+ Encode and decode values from a known schema:

```javascript
const type = avro.Type.forSchema({
type: 'record',
fields: [
{name: 'kind', type: {type: 'enum', symbols: ['CAT', 'DOG']}},
{name: 'name', type: 'string'}
]
});

const buf = type.toBuffer({kind: 'CAT', name: 'Albert'}); // Encoded buffer.
const val = type.fromBuffer(buf); // = {kind: 'CAT', name: 'Albert'}
```

+ Infer a value's schema and encode similar values:

```javascript
const type = avro.Type.forValue({
city: 'Cambridge',
zipCodes: ['02138', '02139'],
visits: 2
});

// We can use `type` to encode any values with the same structure:
const bufs = [
type.toBuffer({city: 'Seattle', zipCodes: ['98101'], visits: 3}),
type.toBuffer({city: 'NYC', zipCodes: [], visits: 0})
];
```

+ Get a [readable stream][readable-stream] of decoded values from an Avro
container file compressed using [Snappy][snappy] (see the [`BlockDecoder`
API][decoder-api] for an example including checksum validation):

```javascript
const snappy = require('snappy'); // Or your favorite Snappy library.
const codecs = {
snappy: function (buf, cb) {
// Avro appends checksums to compressed blocks, which we skip here.
return snappy.uncompress(buf.slice(0, buf.length - 4), cb);
}
};

avro.createFileDecoder('./values.avro', {codecs})
.on('metadata', function (type) { /* `type` is the writer's type. */ })
.on('data', function (val) { /* Do something with the decoded value. */ });
```

+ Implement a TCP server for an [IDL-defined][idl] protocol:

TODO
```javascript
// We first define a service.
const service = new avro.Service({
protocol: 'LengthService',
messages: {
stringLength: {
request: [{name: 'str', type: 'string'}],
response: 'int'
}
}
});

## Avro streams [![NPM version](https://img.shields.io/npm/v/@avro/streams.svg)](https://www.npmjs.com/package/@avro/streams) [![Download count](https://img.shields.io/npm/dm/@avro/streams.svg)](https://www.npmjs.com/package/@avro/streams)
// We then create a corresponding server, implementing our endpoint.
const server = new avro.Server(service)
.onMessage().stringLength((str) => str.length);

TODO
// Finally, we use our server to respond to incoming TCP connections!
const gateway = new avro.NettyGateway(server.channel());
require('net').createServer()
.on('connection', (con) => { gateway.accept(con); })
.listen(24950);
```

## Avro IDL [![NPM version](https://img.shields.io/npm/v/@avro/idl.svg)](https://www.npmjs.com/package/@avro/idl) [![Download count](https://img.shields.io/npm/dm/@avro/idl.svg)](https://www.npmjs.com/package/@avro/idl)

TODO
[custom-long]: https://github.com/mtth/avsc/wiki/Advanced-usage#custom-long-types
[decoder-api]: https://github.com/mtth/avsc/wiki/API#class-blockdecoderopts
[home]: https://github.com/mtth/avsc/wiki
[idl]: https://avro.apache.org/docs/current/idl.html
[logical-types]: https://github.com/mtth/avsc/wiki/Advanced-usage#logical-types
[node.js]: https://nodejs.org/en/
[readable-stream]: https://nodejs.org/api/stream.html#stream_class_stream_readable
[releases]: https://github.com/mtth/avsc/releases
[rpc]: https://github.com/mtth/avsc/wiki/Quickstart#services
[schema-evolution]: https://github.com/mtth/avsc/wiki/Advanced-usage#schema-evolution
[snappy]: https://avro.apache.org/docs/current/spec.html#snappy
[type-inference]: https://github.com/mtth/avsc/wiki/Advanced-usage#type-inference
47 changes: 47 additions & 0 deletions packages/avsc/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
/* jshint esversion: 6, node: true */

'use strict';

const idl = require('@avro/idl');
const types = require('@avro/types');
const streams = require('@avro/streams');
const fs = require('fs');
const path = require('path');

function parse(s, opts) {
let schema;
if (typeof s == 'string' && ~s.indexOf(path.sep) && fs.existsSync(s)) {
// Try interpreting `s` as path to a file contain a JSON schema or an IDL
// protocol. Note that we add the second check to skip primitive references
// (e.g. `"int"`, the most common use-case for `avro.parse`).
const contents = fs.readFileSync(s, {encoding: 'utf8'});
try {
schema = JSON.parse(contents);
} catch (err) {
schema = idl.assembleProtocolSync(s, opts);
}
} else {
schema = s;
}
if (typeof schema == 'string' && schema !== 'null') {
// This last predicate is to allow `read('null')` to work similarly to
// `read('int')` and other primitives (null needs to be handled separately
// since it is also a valid JSON identifier).
schema = JSON.parse(schema);
}
return types.Type.forSchema(schema);
}

module.exports = Object.assign(
{
parse,
Type: types.Type,
types,
createFileDecoder: streams.createFileDecoder,
createFileEncoder: streams.createFileEncoder,
extractFileHeader: streams.extractFileHeader,
streams
},
require('@avro/services'),
idl
);
20 changes: 20 additions & 0 deletions packages/avsc/lib/files.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
/* jshint node: true */

'use strict';

/**
* Filesystem specifics.
*
* This module contains functions only used by node.js. It is shimmed by
* another module when `avsc` is required from `browserify`.
*/

var fs = require('fs'),
path = require('path');


module.exports = {
// Proxy a few methods to better shim them for browserify.
existsSync: fs.existsSync,
readFileSync: fs.readFileSync
};
71 changes: 71 additions & 0 deletions packages/avsc/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

52 changes: 52 additions & 0 deletions packages/avsc/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
{
"name": "avsc",
"version": "6.0.0",
"description": "Avro for JavaScript",
"homepage": "https://github.com/mtth/avsc",
"keywords": [
"avro",
"avsc",
"binary",
"buffer",
"data",
"decoding",
"encoding",
"interface",
"json",
"marshalling",
"schema",
"serialization",
"type"
],
"files": [
"lib",
"types"
],
"main": "./lib",
"types": "./types",
"engines": {
"node": ">=7"
},
"scripts": {
"clean": "rm -rf coverage dist node_modules",
"cover": "nyc mocha",
"coverAndPublish": "nyc npm test && nyc report --reporter=text-lcov | coveralls ../..",
"prepublishOnly": "npm test",
"test": "mocha"
},
"dependencies": {
"@avro/idl": "^0.1.1",
"@avro/services": "^0.10.0",
"@avro/streams": "^0.2.0",
"@avro/types": "^1.0.2"
},
"author": {
"name": "Matthieu Monsch",
"email": "mtth@apache.org"
},
"license": "MIT",
"repository": {
"type": "git",
"url": "git://github.com/mtth/avsc.git"
}
}
3 changes: 3 additions & 0 deletions packages/services/lib/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
'use strict';

const {Channel, RoutingChannel, SelfRefreshingChannel} = require('./channel');
const {JsonChannel, JsonGateway} = require('./codecs/json');
const {NettyChannel, NettyGateway} = require('./codecs/netty');
const {Service} = require('./service');
const {SystemError} = require('./utils');
Expand All @@ -24,6 +25,8 @@ module.exports = {
Channel,
Client,
Deadline,
JsonChannel,
JsonGateway,
NettyChannel,
NettyGateway,
RoutingChannel,
Expand Down
4 changes: 2 additions & 2 deletions packages/services/lib/utils.js
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,12 @@ class SystemError extends Error {
return stringType.wrap(this);
}

get _isSystemError() {
get _isAvroServicesSystemError() {
return true;
}

static isSystemError(any) {
return any && any._isSystemError;
return !!(any && any._isAvroServicesSystemError);
}

static orCode(code, err) {
Expand Down

0 comments on commit b5e61ea

Please sign in to comment.