[JS] Implement JavaScript-compatible implementation

[asfimport]

I would really like to see an implementation of Arrow that is compatible with NodeJS and modern web browsers. This would enable streaming columnar data to the browser without an intermediate conversion to JSON – I could imagine web-based data visualization systems would benefit substantially from this functionality.

Because Google's Flatbuffers library has first-class JS support, this should be achievable. Carlos Scheidegger created a prototype implementation of the Feather file format (basically: mini-Arrow) last year: https://github.com/wesm/feather/tree/master/js

This JIRA can be an umbrella issue for the JS implementation. As soon as we find interested contributors to work on this, we can create JIRAs and attach them to this issue until we reach a certain level of maturity.

Without starting a bike shed discussion, I humbly request that the implementation be done in TypeScript https://www.typescriptlang.org/

Reporter: Wes McKinney / @wesm
Assignee: Paul Taylor / @trxcllnt

Related issues:

• [JS] Implement random-access file format (contains)
• [JS] Generate multiple sets of artifacts (contains)
• [JS] Use flatc to generate TypeScript (contains)
• [JS] Read streaming format (contains)
• [JS] Read dictionary-encoded vectors (contains)
• [JS] Implement fixed width list type (contains)
• [JS] Support custom_metadata (contains)
• [JS] Implement JSON writer for Integration tests (contains)
• [JS] Add entry to Travis CI matrix (contains)
• [JS] Add unit test suite (contains)
• [JS] Rename directory to js/ (contains)
• [JS] Fix generated API documentation (contains)
• [JS] Move io operations to arrow.io namespace (contains)

_{Note: This issue was originally created as ARROW-541. Please see the migration documentation for further details.}

[asfimport]

Dan Robinson / @danrobinson:
I'm interested in contributing to a JS/Node implementation. Here's one I worked on last year: https://github.com/danrobinson/jsarrow. It is probably out of date with the spec and certainly the C++ implementation (with which it was fairly tightly coupled), so I'd be fine starting from scratch if you or others have a roadmap in mind, and maybe cannibalizing it for parts (like the C++ bindings).

+1 to TypeScript (another reason I would scrap most of that old repo).

[asfimport]

Brian Hulette:
I'm also interested in contributing. I worked on a (very basic) library last fall focused on reading arrow files in the browser/node for data visualization. Unfortunately its a private repo and I can't link it here but I've got the go-ahead to cannibalize parts of it for an official implementation.

TypeScript does seem like the way to go for this.

[asfimport]

Rick Fanta:
I'm also interested. Have worked on a lot of demanding projects using Java + Python (sadly for closed-source investment banks), and my code quality is generally pretty good. Happy to adopt code styles from high quality Typescript projects and put in some good sweat equity to implement an implementation that the community can be happy with..

I'm esp. interested in getting cool projects like this to work with Arrow (https://github.com/nicolaskruchten/pivottable). [i.e. Super cool but begging for higher quality I/O that Arrow would provide.]

[asfimport]

Wes McKinney / @wesm:
Sounds great to me. Once the project is spun up and we have the build and test machinery in place, I can help with a bit of implementation / testing and code review. Good excuse to learn TypeScript also

[asfimport]

Brian Hulette:
I ported my original implementation over to TypeScript and opened a PR that includes it: #358

Its still pretty rough around the edges, but it does add the ability to read the arrow file format in the browser and with Node.

[asfimport]

Wes McKinney / @wesm:
I'll add JIRAs to this issue as things develop, once we reach a certain level of feature completeness we can close this JIRA

[asfimport]

Brian Hulette / @TheNeuralBit:
@wesm should I go through you to create tickets under this one, or just go ahead and write them up myself? I'd like to add some structural things like rename javascript/ to js/, add an arrow.io namespace, and also implementations for the streaming format as well as the fixed length list type.

[asfimport]

Wes McKinney / @wesm:
Sure, go ahead and create new JIRAs and attach them to this one

[asfimport]

Wes McKinney / @wesm:
While integration tests are still in the works, I think we can call this one closed. Thanks to all!