New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blobs #11103

Closed
philikon opened this Issue Nov 24, 2016 · 22 comments

Comments

Projects
None yet
8 participants
@philikon
Contributor

philikon commented Nov 24, 2016

Here at Silk Labs we've had a Blob implementation for React Native for a while. We want to contribute it back upstream. This issue discusses the API and implementation and tracks progress.

Motivation

Like in the browser, Blobs are opaque lumps of (usually binary) data that can be read into TypedArrays or strings, but don't have to be. Concretely in React Native that means that the binary data lives in native and not in JS, unless you explicitly request it. Why is this useful? Say you're receiving some data via XHR, WebSocket, or WebRTC, or some custom method that you've implemented and you want to display it in an <Image> view. Or you want to send it along through some other means without looking paying the cost of transferring it to the JS thread and back.

We already have something somewhat similar to this for images on iOS (RCTImageStoreManager). This would be a generalization of that.

API

The web has an API for Blobs. It's not particularly pretty or anything, but it's kind of a standard, so I think we should use that, or at least a pretty good subset.

Creating

Blobs typically get created when JS is receiving data, e.g. via XHR, WebSocket, etc. I am planning on making

xhr.responseType = 'blob';

and

websocket.binaryType = 'blob';

work so that when you specify those settings, the XHR and WebSocket responses will return Blob objects and not transfer the data to the JS thread. This works exactly like on the web.

Mutating

A subset of mutation operations will be supported:

let combinedBlob = new Blob([existingBlob, anotherBlob]);
let halfBlob = blob.slice(0, blob.size / 2);

I'm not planning on supporting creating a blob in JS from JS data (e.g. a string or TypedArray), though it should be easily implementable.

Consuming

Like in the browser, you'll be able to map Blobs to URIs that you can then use in things that eat URIs, e.g. the view:

let url = URL.createObjectURL(blob);

Also like in the browser, you'll be able to read the blob into a TypedArray or string. I am planning on simply implementing the web's FileReader API.

Freeing

The following method is part of the Blob standard, but is currently not implemented by browsers. However, because we can't automatically garbage collect our blobs like browsers can, it will be necessary to provide a manual way of freeing blobs, so that consumers can make use of it when appropriate:

blob.close();

Here's a complete example of how consuming blobs would work: https://gist.github.com/philikon/4592efd153673f3e64dec353046af7a8

Implementation

Proposal

React Native already has a loose concept of a "blob", in other words, a lump of data that's not directly accessible by JS. The iOS (RCTImageStoreManager) mentioned above is an example.

XHR and most notably the <Image> tag already have a way to express a reference to a blob:

{uri: "some://uri/that/gets/resolved"}

How does some://uri/that/gets/resolved get resolved? On iOS, it gets resolved through a RCTURLRequestHandler. On Android, it gets resolved through a ContentProvider.

For instance, you can upload binary data provided by a system resource in the following way:

const xhr = new XMLHttpRequest();
xhr.open(...);
xhr.send({uri: "some://big/blob/of/binary/data"})

As long as some://big/blob/of/binary/data is resolvable by a RCTURLRequestHandler or ContentProvider, this will Just Work(tm) already.

On this basis, I'm proposing to implement Blobs as simply another way of URI-addressable binary data. Our implementation does exactly that.

Progress

Prep work:

  • Make XMLHttpRequest and XMLHttpRequest.upload proper EventTargets so that we can receive progress events even when we're not receiving data to JS, because we're receiving it into a blob (#7017)
  • Add responseType as a concept to RCTNetworking, so that we can add 'blob' later (#8324)

Implementation work, most likely by porting silk-react-native-blobs:

  • Provide native blob storage with NativeModule API, and JS implementation of Blob class
  • Provide URI integration (RCTURLRequestHandler on iOS, ContentProvider on Android)

Future implementation work:

  • XHR integration: allow xhr.responseType = 'blob' and xhr.send(blob)
  • WebSocket integration: allow websocket.binaryType = 'blob' and websocket.send(blob)
  • Implement FileReader API for reading blobs into JS e.g. as TypedArrays. This can just be a thin wrapper around XHR, yay!
@ide

This comment has been minimized.

Collaborator

ide commented Nov 24, 2016

Can you coordinate with @satya164, he has a Blob implementation almost ready I believe.

@philikon

This comment has been minimized.

Contributor

philikon commented Nov 24, 2016

Yep, that's the reason for this issue :), so that we can coordinate. I believe he's uplifting our Blob implementation.

@satya164

This comment has been minimized.

Collaborator

satya164 commented Nov 24, 2016

Hey! Yeah, so I've been working to port changes from @philikon's implementation to RN master.

I've only worked on Android part so far.

My focus was to first get it working with firebase so people could upload files to firebase. This has slightly different requirements since we need to create a blob locally from a file before uploading.

So I think in addition to the proposed APIs, we need two more non-standard methods,

Blob.fromURI(uri: string, options: { type: string }): Promise<Blob>
File.fromURI(uri: string, options: { type: string }): Promise<File>

I just copied @philikon's implementation for the blob module and with minor changes, and writing some code to support XHR I've been able to get the following working,

  • Create blobs when data is received via XHR
  • Create blobs from a local file URI
  • Upload blobs to a server (e.g. - firebase) via XHR

I couldn't get URL.createObjectURL working from the current implementation, looking into it right now.

I've not yet tested the WebSocket implementation yet, but since that was already implemented I guess shouldn't take much time to get it working.

So things I need to do are,

  • Get content provider working
  • Get the WebSocket implementation working

What I'd like to explore is, right now when you create a blob from URI, I read it to memory as bytes. But we could probably just store the uri and read the bytes when needed (we'd need get the bytes length though, so we'd need to read it twice!)

I also want to know if there's a full test suite available to test my implementation against.

@philikon

This comment has been minimized.

Contributor

philikon commented Nov 26, 2016

So I think in addition to the proposed APIs, we need two more non-standard methods,

Blob.fromURI(uri: string, options: { type: string }): Promise
File.fromURI(uri: string, options: { type: string }): Promise

  • I don't understand the need for File. What use cases does File solve that aren't covered by Blob?
  • What specific use cases does Blob.fromURI(...) solve? If I have a URI, why wouldn't I make an informal {uri: ...} blob object? If there's an RCTURLRequestHandler / ContentProvider for the URI in question, it'll already work with XHR! The only thing that we gain with Blob support is concatenation and slicing, but as you say, that's really hard to do if we don't at least know the size of each part. I'm not familiar with Firebase, but I'd like to hear more about why a generic URI-based Blob is a good idea. For now it seems pretty hard to implement Blob semantics around any lump of data that's not already stored in memory. Maybe a different API that also happens to expose uri for XHR-compatibility would be better suited for Firebase integration?
  • It also sounds like you might be thinking about this backwards. If people want to create blobs from some sort of datasource, the datasource should return Blob objects, e.g. someFirebaseAPI.get(options).then((blob) => ...). Or would Blob.fromURI() be an API that's meant only for React Native modules, much like the non-standard Blob.create() I introduced?
@satya164

This comment has been minimized.

Collaborator

satya164 commented Nov 26, 2016

I don't understand the need for File. What use cases does File solve that aren't covered by Blob

File implements the File API from the spec. It's just a thin wrapper around blob.

What specific use cases does Blob.fromURI(...) solve? If I have a URI, why wouldn't I make an informal {uri: ...} blob object? If there's an RCTURLRequestHandler / ContentProvider for the URI in question, it'll already work with XHR! The only thing that we gain with Blob support is concatenation and slicing, but as you say, that's really hard to do if we don't at least know the size of each part. I'm not familiar with Firebase, but I'd like to hear more about why a generic URI-based Blob is a good idea. For now it seems pretty hard to implement Blob semantics around any lump of data that's not already stored in memory. Maybe a different API that also happens to expose uri for XHR-compatibility would be better suited for Firebase integration?

Libraries like firebase expect a blob or file object for uploading. We cannot pass a { uri: string } object to firebase. So allowing to create a file/blob object from an URI is the only way people can use firebase for now. I don't like it much, but I don't think there are any options.

Firebase API looks like this,

const ref = firebase.storage().ref().child('image.jpg');
const snapshot = await ref.put(blob);

It also sounds like you might be thinking about this backwards. If people want to create blobs from some sort of datasource, the datasource should return Blob objects, e.g. someFirebaseAPI.get(options).then((blob) => ...). Or would Blob.fromURI() be an API that's meant only for React Native modules, much like the non-standard Blob.create() I introduced?

This is so that we'll be able to pass blob objects to libraries which don't accept uris. I think in general this should be discouraged though, and libraries can internally use this API if they need.

@philikon

This comment has been minimized.

Contributor

philikon commented Nov 26, 2016

File implements the File API from the spec. It's just a thin wrapper around blob.

Meh ok. Seems unnecessary, unless there's demand from 3rd party libraries out there. (Perhaps Firebase?)

Libraries like firebase expect a blob or file object for uploading.

Ok, so then just do

const response = await fetch(uri);
const blob = await response.blob();
someFirebaseAPI.consume(blob);

Or in XHR spelling:

const xhr = new XMLHttpRequest();
xhr.open('GET', uri);
xhr.responseType = 'blob';
xhr.onload = () => {
  const blob = xhr.response;
  someFirebaseAPI.consume(blob);
};
xhr.send();

Either variant will load the data from the URI into a Blob that can then be passed to Firebase. Though, I still don't quite understand what Firebase is going to do with it? Upload it to the server?

@satya164

This comment has been minimized.

Collaborator

satya164 commented Nov 26, 2016

Meh ok. Seems unnecessary

It's very tiny amount code to polyfill a web API, which makes sure libs which need File constructor will work.

Either variant will load the data from the URI into a Blob that can then be passed to Firebase.

I didn't know I can do fetch from local URI. Or are you proposing to implement this API?

Though, I still don't quite understand what Firebase is going to do with it? Upload it to the server?

Yes

@philikon

This comment has been minimized.

Contributor

philikon commented Nov 26, 2016

I didn't know I can do fetch from local URI.

Yep, should already work!

(* Definitely on iOS. No idea about Android. If XHR can't currently download from content:// URIs, then we should add that support.)

@philikon

This comment has been minimized.

Contributor

philikon commented Nov 26, 2016

It's very tiny amount code to polyfill a web API, which makes sure libs which need File constructor will work.

Sure, ok.

@brentvatne

This comment has been minimized.

Collaborator

brentvatne commented Nov 27, 2016

It also sounds like you might be thinking about this backwards. If people want to create blobs from some sort of datasource, the datasource should return Blob objects, e.g. someFirebaseAPI.get(options).then((blob) => ...). Or would Blob.fromURI() be an API that's meant only for React Native modules, much like the non-standard Blob.create() I introduced?

I had the same thought, the issue that @satya164 pointed out to me is that if the datasource API (let's say ImagePicker which lets you pick an image from your camera roll or take a photo) returns a Blob every time then people who are not interested in the Blob but rather in other other values like the URI might just forget to close the Blob. We could make it an optional parameter on those APIs:

let imageBlob = ImagePicker.launchImageLibraryAsync({
  allowsEditing: false,
  aspect: [1, 2],
  responseType: 'blob',
})

Thoughts @satya164 @philikon?

(the ImagePicker I refer to here is the one in Exponent - see docs if you're curious)

@ide

This comment has been minimized.

Collaborator

ide commented Nov 27, 2016

Another approach would be to write a separate module that creates a Blob from a URI -- BlobLoader.loadFromURIAsync(uri). This would keep the function out of the Blob/FileSystem namespace but still let you work with any API that gives you back a local URI.

That said, one thing I like about libraries that return Blobs (like Brent's ImagePicker example) is that you either get back a valid Blob (assuming it's eagerly loaded) or the call fails. There's no uncertain state halfway where you get back a valid URI but then the file on disk gets deleted before you load the Blob from the URI.

So we might want both -- Blob-aware APIs that give you back Blobs in a more atomic way, and a generic URI-to-Blob loader that works with any URI-aware API.

@satya164

This comment has been minimized.

Collaborator

satya164 commented Nov 28, 2016

Another approach would be to write a separate module that creates a Blob from a URI

Separate module for creating blobs from URI sounds good to me

We could make it an optional parameter on those APIs

That could work, it only saves one line of code of code though (not really since you've to specify responseType: 'blob' anyways).

While the blob-aware APIs seem nice to me in a general sense, in case of RN I'm not super happy about them since the user always has to manually free the blob. . One thing I wanted to explore is to lazy-load URI based blobs. But this negates the case @ide mentioned about not having to deal with uncertain state.

@hramos

This comment has been minimized.

Contributor

hramos commented May 25, 2017

Closing this issue because it has been inactive for a while. If you think it should still be opened let us know why.

@hramos hramos closed this May 25, 2017

@hramos hramos added the Icebox label May 26, 2017

facebook-github-bot added a commit that referenced this issue Jul 26, 2017

Add blob implementation with WebSocket integration
Summary:
This is the first PR from a series of PRs grabbou and me will make to add blob support to React Native. The next PR will include blob support for XMLHttpRequest.

I'd like to get this merged with minimal changes to preserve the attribution. My next PR can contain bigger changes.

Blobs are used to transfer binary data between server and client. Currently React Native lacks a way to deal with binary data. The only thing that comes close is uploading files through a URI.

Current workarounds to transfer binary data includes encoding and decoding them to base64 and and transferring them as string, which is not ideal, since it increases the payload size and the whole payload needs to be sent via the bridge every time changes are made.

The PR adds a way to deal with blobs via a new native module. The blob is constructed on the native side and the data never needs to pass through the bridge. Currently the only way to create a blob is to receive a blob from the server via websocket.

The PR is largely a direct port of https://github.com/silklabs/silk/tree/master/react-native-blobs by philikon into RN (with changes to integrate with RN), and attributed as such.

> **Note:** This is a breaking change for all people running iOS without CocoaPods. You will have to manually add `RCTBlob.xcodeproj` to your `Libraries` and then, add it to Build Phases. Just follow the process of manual linking. We'll also need to document this process in the release notes.

Related discussion - #11103

- `Image` can't show image when `URL.createObjectURL` is used with large images on Android

The websocket integration can be tested via a simple server,

```js
const fs = require('fs');
const http = require('http');

const WebSocketServer = require('ws').Server;

const wss = new WebSocketServer({
  server: http.createServer().listen(7232),
});

wss.on('connection', (ws) => {
  ws.on('message', (d) => {
    console.log(d);
  });

  ws.send(fs.readFileSync('./some-file'));
});
```

Then on the client,

```js
var ws = new WebSocket('ws://localhost:7232');

ws.binaryType = 'blob';

ws.onerror = (error) => {
  console.error(error);
};

ws.onmessage = (e) => {
  console.log(e.data);
  ws.send(e.data);
};
```

cc brentvatne ide
Closes #11417

Reviewed By: sahrens

Differential Revision: D5188484

Pulled By: javache

fbshipit-source-id: 6afcbc4d19aa7a27b0dc9d52701ba400e7d7e98f

rozele pushed a commit to Microsoft/react-native-windows that referenced this issue Sep 11, 2017

Add blob implementation with WebSocket integration
Summary:
This is the first PR from a series of PRs grabbou and me will make to add blob support to React Native. The next PR will include blob support for XMLHttpRequest.

I'd like to get this merged with minimal changes to preserve the attribution. My next PR can contain bigger changes.

Blobs are used to transfer binary data between server and client. Currently React Native lacks a way to deal with binary data. The only thing that comes close is uploading files through a URI.

Current workarounds to transfer binary data includes encoding and decoding them to base64 and and transferring them as string, which is not ideal, since it increases the payload size and the whole payload needs to be sent via the bridge every time changes are made.

The PR adds a way to deal with blobs via a new native module. The blob is constructed on the native side and the data never needs to pass through the bridge. Currently the only way to create a blob is to receive a blob from the server via websocket.

The PR is largely a direct port of https://github.com/silklabs/silk/tree/master/react-native-blobs by philikon into RN (with changes to integrate with RN), and attributed as such.

> **Note:** This is a breaking change for all people running iOS without CocoaPods. You will have to manually add `RCTBlob.xcodeproj` to your `Libraries` and then, add it to Build Phases. Just follow the process of manual linking. We'll also need to document this process in the release notes.

Related discussion - facebook/react-native#11103

- `Image` can't show image when `URL.createObjectURL` is used with large images on Android

The websocket integration can be tested via a simple server,

```js
const fs = require('fs');
const http = require('http');

const WebSocketServer = require('ws').Server;

const wss = new WebSocketServer({
  server: http.createServer().listen(7232),
});

wss.on('connection', (ws) => {
  ws.on('message', (d) => {
    console.log(d);
  });

  ws.send(fs.readFileSync('./some-file'));
});
```

Then on the client,

```js
var ws = new WebSocket('ws://localhost:7232');

ws.binaryType = 'blob';

ws.onerror = (error) => {
  console.error(error);
};

ws.onmessage = (e) => {
  console.log(e.data);
  ws.send(e.data);
};
```

cc brentvatne ide
Closes facebook/react-native#11417

Reviewed By: sahrens

Differential Revision: D5188484

Pulled By: javache

fbshipit-source-id: 6afcbc4d19aa7a27b0dc9d52701ba400e7d7e98f
@austinksmith

This comment has been minimized.

austinksmith commented Oct 3, 2017

@hramos Please reopen , blob support is very critical to my current issue. #16034 (comment)

@satya164

This comment has been minimized.

Collaborator

satya164 commented Oct 3, 2017

Theres no point in re-opening. There's already a WIP PR implementing it.

@austinksmith

This comment has been minimized.

austinksmith commented Oct 3, 2017

Can you post that here and on the ticket I've opened? ETA on completion etc.?

@satya164

This comment has been minimized.

Collaborator

satya164 commented Oct 3, 2017

There's no ETA. #11573

@austinksmith

This comment has been minimized.

austinksmith commented Oct 3, 2017

So how can we get this issue prioritized, it appears that PR has been in limbo for almost a year

@satya164

This comment has been minimized.

Collaborator

satya164 commented Oct 3, 2017

Only way is to help me with the iOS code and then wait for review.

@austinksmith

This comment has been minimized.

austinksmith commented Oct 6, 2017

@satya164 What is needed on IOS side?

@pradeep250677

This comment has been minimized.

pradeep250677 commented Jan 16, 2018

@satya164 could you share me with Andriod code? Binary data communication API between JSC and Native using RN bridge is very important. I am the only concern for Andriod.

@satya164

This comment has been minimized.

Collaborator

satya164 commented Jan 16, 2018

@pradeep250677 you can check the pr linked above.

@facebook facebook locked as resolved and limited conversation to collaborators Jul 19, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.