Bit Voyage

Jump to the working demo.

Vision: Allow anyone with a modern browser to drag, drop, and stream a 1GB, 10GB, 100GB, or 1TB file over the Internet to a happy home.

Requirements: Files should never be read entirely into memory at any stage. File progress should be real-time and clear. Uploads should be resumable and pause-able at any time.

Implementation:

• Use the Drag and Drop API, available in modern desktop browsers, to let anyone drag a file into their browser
• Use the File API, available in modern browsers, to chunk and stream file objects in-browser
• Use the FileReader object, available in modern browsers, to read the contents of chunks into memory

Goals:

• Short-term: Use the S3 Multipart Upload API to allow users to upload files of any size directly to S3. (Update: this is working, for up to ~100GB.)
• Long-term: Allow upload to servers which support the Amazon S3 API but provide their own layer of authentication. Example: The Internet Archive.
• Longest-term: Instead of S3, allow users to stream files directly to other users over WebRTC, in the style of Sharefest.me, as long as their browser tab is open.

Working demo

A working demo is available at bit.voyage. It lets you drag files of up to around 100GB into an S3 bucket. Files much larger than that will cause your browser to try to send very large parts, and files of around the 1TB range may crash your browser (like they did mine).

• Enter your S3 credentials into the URL hash (not the query string):

https://bit.voyage/#bucket=[your-bucket]&key=[your-key]&secret_key=[your-secret-key]

• Make sure your S3 bucket has CORS enabled. Use the following CORS configuration, changing https://bit.voyage to * if you want it to work with more domains.

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>https://bit.voyage</AllowedOrigin>
        <AllowedMethod>HEAD</AllowedMethod>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedMethod>POST</AllowedMethod>
        <AllowedMethod>DELETE</AllowedMethod>
        <ExposeHeader>ETag</ExposeHeader>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

Next steps include:

• Petitioning the Internet Archive to add CORS to their S3-like API.
• Making a new UI.
• Resuming very large downloads. I have yet to actually store a complete 100GB file, because of the time involved.

Getting to 1TB+

For Amazon S3, this will require making 100MB+ HTTP POST requests, which realistically can only be done by a server. So, this involves:

• Finding or making a websocket stream library that can receive file data.
• Finding or making a thin web server that can receive websocket data and make authorized multipart uploads to S3.
• Being able to handle backpressure from the server all the way down to reading the file.
• Sending progress/etc events down the wire as well.

One interesting project is @maxogden's abstract-blob-store, which aims to present a consistent interface to streaming blobs up and down anything that supports a stream. It's a little nascent, and doesn't yet have the semantics to support resumption, but it feels like the right direction.

Credits

Created by Eric Mill.

Builds on work by Brian Brennan, Max Ogden, Derrick Parkhurst, and Young Hahn.

Public domain

This project is dedicated to the public domain. As spelled out in CONTRIBUTING:

The project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.