Track: react native support

[hugomrdias]

Tracking:

• ipfs-http-client Track: HTTP client support on React Native for Android and iOS #3404
• ipfs

Related

ipfs-http-client

https://github.com/ipfs/js-ipfs-http-client/issues/1117#issuecomment-589011968
ipfs-inactive/js-ipfs-http-client#1224 (comment) this needs to be patched in react-native not here
https://github.com/ipfs/js-ipfs-http-client/issues/1219#issuecomment-582846315 close after proper fix for arrayBuffer
ipfs-inactive/js-ipfs-http-client#1224

ipfs

#1254 nodejs-mobile, nodejs-mobile-react-native
#2768 nodejs-mobile-react-native

[pcowgill]

I appreciate you adding the label awesome endeavour 😄

[pcowgill]

The latest update on my efforts related to this:

Back to using the GitHub fetch polyfill which is bundled in React Native (rather than rn-fetch-blob or polyfilling fetch with streams using @stardazed/streams-polyfill) and locally handling the case when body is undefined and giving an async iterable behavior in a fork of ipfs-http-client
Latest upstream changes to ipfs-http-client allow requests to go through with a 2xx status code to the local daemon which wasn't the case before with older versions, but it's always the same empty file that is added in the local node.

• Did a lot of logging on the receiving side after switching over to js-ipfs for the local daemon (from go-ipfs before), and that helped determined the difference between what was actually reaching the node in React vs. React Native
• When doing an add from React Native the data we're adding to the multipart/form-data payload is missing (boundary, headers, etc. are still there), but it's there when sent from a React env. Next step is logging along the path to the http request that is initiated to see where the data is dropped. React Native's non-standard implementation of Blob seems likely to be the problem.
• We now have the tooling to see what is reaching the IPFS node, but other than continuing to log incoming payloads there the next debugging to be done is back on the React Native side as the data is transformed into the HTTP request.
• When using dag.put rather than add, it isn't working in either React or React Native at the moment, which is surprising. Issue here: dag.put does not work in the js ipfs daemon with cbor-style input, even though the same request works with go-ipfs #2825

[pcowgill]

ipfs-inactive/js-ipfs-http-client#1224 (comment) this needs to be patched in react-native not here

I think this is true on a longer timescale, but I'm not sure if we have a good chance of getting this through on a short timescale. So I think it's worth patching here in the meantime. @achingbrain what do you think?

[pcowgill]

I think this is true on a longer timescale, but I'm not sure if we have a good chance of getting this through on a short timescale. So I think it's worth patching here in the meantime. @achingbrain what do you think?

We could use an approach like you all did with ky-universal before:

// TODO remove this when upstream fix for ky-universal is merged
// https://github.com/sindresorhus/ky-universal/issues/9
// also this should only be necessary when nodeIntegration is false in electron renderer
if (isElectronRenderer) {
  exports.toFormData = require('./form-data.browser').toFormData
}

[pcowgill]

Here is the tracking issue on the React Native repo facebook/react-native#27741

[achingbrain]

Reading that and the related issues, it doesn't sound like something they'll take on unless it becomes a priority for them, but it does sound like they are open to other people contributing a fix. Want to take a stab at PRing support in?

[hugomrdias]

Exactly these issues about half baked spec implementations need to be patch directly in rn or in a external package.

[pcowgill]

Based on the latest comments on that issue, they're open to creating a fork of github/fetch under the react-native-community org, so we can move forward with that. I emailed @alloy and cc'ed @hugomrdias to get that started.

Here's the original discussion about the changes to be made to github/fetch github/fetch#746. That is a medium-term solution. The ideal long-term solution is a spec-compliant implementation of fetch at the native level https://react-native.canny.io/feature-requests/p/implement-fetch-using-native-implementations-instead-of-fetch-on-top-of-xhr

I added a related comment here about where these changes should land from the IPFS org's perspective #2847 (comment)

[RobertFischer]

https://www.npmjs.com/package/cross-fetch works in RN already...why not just use that?

[hugomrdias]

cross-fetch is just a wrapper problems the issue is much deeper in the stack

[RobertFischer]

@pcowgill @hugomrdias What's the status of this? I've got some spare cycles that I'd like to throw at it.

[x5engine]

@RobertFischer cycles? who wants to fund this? I'll take care of it.

[pcowgill]

@x5engine If you're asking what @RobertFischer means by cycles, he means time to work on this.

@RobertFischer @x5engine My comment on Mar 10 reflects the latest status on this as far as I know. My funding to work on this problem disappeared along with the pandemic, but I'd be happy to hop on a call to do a full debrief of the current state of things!

[x5engine]

@pcowgill I have no issues on this if you want to have a call and do a full debrief on how to get this working I would be very glad to be there and we can live stream it too! set a time

[ZTECH10]

I am currently working on using IPFS on react-native.

On React-native: When I add a buffered string(i.e. Buffer(str) ) to IPFS using ipfs.add, the buffered string is added successfully and IPFS returns the hash.

When I try to retrieve the buffered string via ipfs.cat and the hash, ipfs.cat returns [Error: XHR error]. So the returned variable is undefined.

On Node.jS and ReactJs: I do not have this problem. Both ipfs.add and ipfs.cat works.

Is this a problem with react native? is it related to fetching ? would changing the ipfs-http-client version in packag.json help?

btw, i am using ipfs-http-client 27.0.0

Any suggestions would be highly appreciated

[x5engine]

@ZTECH10 nice show us a repo or something on how you did that? did you use an ipfs node or just the http api?

[qalqi]

@x5engine

Here is react-native sandbox with working get requests https://github.com/qalqi/react-native-ipfs-http-client/tree/primary/rn-nodeify

[theabdulmateen]

@ZTECH10 Hi, I have started a project that is based on IPFS, have you found a fix or a workaround to this problem, any help would be appreciated, please provide an update on this.

Thanks.

[duxiaofeng-github]

actually, the easier way to do this is running ipfs in an invisible web view using react-native-webview and communicate with each other by using injectJavascript.

but this solution has a weakness. i couldn't read data anymore after the app had been killed. when you need to read some data on schedule task in background, it would failed.

[duxiaofeng-github]

also, i think the 'right' way to do this is implement the libp2p by swift and java. then ipfs, finally bring them into react native. but it's a difficult task😐. libp2p written in kotlin is in progress. no swift version at all.😂

[RobertFischer]

I've been working on the side to get IPFS on React Native. As a part of that, I've created an implementation of Node's dns module which is based on fetch and DNS-over-HTTPS: https://github.com/RobertFischer/fetch-dns Also, I needed a pure randomBytes implementation, so I made that: https://github.com/robertfischer/randombytes-pure The other library substitution that I needed to make was using fast-text-encoding to replace web-encoding. Both libp2p-mdns and mortice needed shimming: the libp2p-mdns shim just exports a function that explodes when you call it; the mortice shim exports a function that returns a new EventEmitter instance.

[chebyte]

I've been working on the side to get IPFS on React Native. As a part of that, I've created an implementation of Node's dns module which is based on fetch and DNS-over-HTTPS: https://github.com/RobertFischer/fetch-dns Also, I needed a pure randomBytes implementation, so I made that: https://github.com/robertfischer/randombytes-pure The other library substitution that I needed to make was using fast-text-encoding to replace web-encoding. Both libp2p-mdns and mortice needed shimming: the libp2p-mdns shim just exports a function that explodes when you call it; the mortice shim exports a function that returns a new EventEmitter instance.

can you share how you get ipfs working on react-native? I still cant

[achingbrain]

@RobertFischer thanks for doing the research into what needs patching for it to work.

If you could share a demo repo that runs an app under React Native with IPFS I'm sure we could fix up up those modules (e.g. web-encoding, mortice, etc) to pull in the relevant polyfills so you wouldn't need to shim them.

[chebyte]

@achingbrain +1

[acostalima]

I've been running some experiments with the HTTP client on RN (iOS simulator currently) for about two weeks now and I was already able to get ipfs.add working successfully, on both the request and response side of things. I haven't played with anything else besides ipfs.id and ipfs.add yet. I've been taking detailed notes of what I've been doing at https://github.com/ipfs-shipyard/react-native-ipfs-demo where you can also find a playground app. It's nothing fancy, just a bunch of buttons to test the client's API methods. My debugging sessions have been pretty much console.loging all over the place to keep track of the execution flow and understand what's going on. Yes, we can't use the debugger to remote debug the app otherwise it runs in the browser instead of JSC.
As you all may already know, getting the HTTP client to work properly in the most recent version of RN is quite problematic for several reasons. Below you can find a summary of the key issues I've came across so far:

• URL and URLSearchParams APIs are poorly supported. I had to switch out their implementations with https://github.com/charpeni/react-native-url-polyfill. While this polyfill works and it is light sized, it has no Unicode support. I tried to use whatwg-url instead, but I was not able to get it to work.
• BigInt API is not supported at all. It had to be polyfilled with https://github.com/peterolson/BigInteger.js
• Encoding APIs, TextDecoder and TextDecoder, are not supported. Had to be polyfilled with https://github.com/inexorabletash/text-encoding.
• Async generators are not supported, namely the for await (...) syntax (AsyncGenerator doesn't exist / work facebook/react-native#27432 (comment)). Transpiling is required with https://babeljs.io/docs/en/babel-plugin-proposal-async-generator-functions Babel plugin.
• FormData API is poorly supported. The HTTP client uses the FormData.set method which is not supported. I've patched the API to put together a working implementation.
• FormData cannot handle Blob objects. It can handle blob-like objects which, among others, it takes an uri property which accepts and URI to a native blob (e.g. a file stored in disk) with the following scheme blob:<blob UUID>?offset=<Blob.data.offset>&size=<Blob.data.size>. RN's FormData has a non-standard getParts() method which is used internally to construct HTTP request objects that the underlying iOS and Android implementations of the network layer accept. I've patched getParts() to handle Blobs as well by providing the missing uri property which can be created from URL.createObjectURL.
• ReadableStream is not supported at all. I've developed a brute force and naive implementation to get Response.body.getReader() to work. The read() method does not offer true streaming but instead all it does is take the whole response body as single chunk from Response.arrayBuffer() and wrap it with a Uint8Array as per spec (Implement response.body as a stream github/fetch#746 (comment)).
• FileReader does not support readAsArrayBuffer method. Considering that, apparently, all response bodies in RN are Blobs, to make Response.arrayBuffer() work we need to read them and store its contents in a buffer. RN fully manage blobs in the native side and only expose opaque references to JS land. Since we can't read the binary data of a blob into a buffer natively, all we can do at the moment is read it as a data URL, extract the base64, atob it to get a binary string and load the bytes into the buffer. This is just exactly what the implementation I've developed is doing.

I've tested ipfs.add with several types of input. Most of them appear to work just fine but when a Blob is passed as content, it fails when the Blob is constructed with an ArrayBuffer throwing a not implemented error. Just like FileReader.readAsArrayBuffer, this kind of suggests RN's native blob implementation does not know how to or cannot deal with ArrayBuffers for some reason or it could be maybe be a limitation of the bridge self (?). Thus, as far as I can tell from Blob's implementation, only strings are supported so far.

All of the above are just short-term hacks, of course. If we can get away with patching up RN's JS core here and there just to get things running quickly, we may have a short-term solution for app developers. At the same time, we are uncovering and documenting the issues within the platform. Later on, we must sit down with the RN team to discuss these issues and plan how they should be addressed, eventually generating contributions to up level the ecosystem. Next week, I'm going to have a look at other HTTP client's methods, such as .get(), .cat(), .swarm.* and .pubsub.* to check whether are there other issues we yet to be revealed.

A few RN issues I found related to this endeavour:

• [0.59.x] URLSearchParams 'Error: not implemented' facebook/react-native#23922
• AsyncGenerator doesn't exist / work facebook/react-native#27432
• ReferenceError: Can't find variable: BigInt facebook/react-native#28492
• Fetch hangs when trying to access HTTP streaming API facebook/react-native#12912
• Creating an instance of URL like: new URL('http://facebook.com') throws an exception. facebook/react-native#16434
• Warn about URL and URLSearchParams not being spec-complaint facebook/react-native#30188
• Implement URL standard and URLSearchParams spec compliant facebook/react-native#25719
• URL incorrectly adds trailing slash facebook/react-native#24428

[acostalima]

I've got .get, .cat and .pubsub working on both iOS and Android. 🎉

Subscriptions operate on the basis of a long-running HTTP responses, i.e., an endless stream. As React Native does not support returning a ReadableStream natively nor provide access to the underlying byte-stream (only base64 can be read through the bridge), so we have to fallback to XMLHttpRequest. React Native's XHR provides progress events which buffers text and allows us to concatenate a response by encoding it into its UTF-8 byte representation using the TextEncoder API. Although very inefficient, it's some of sort of pseudo-streaming that works. The problem, however, is that we're reading text and not raw binary data so this may be a shortcoming for some use cases. Pubsub subcriptions are currently base64 encoded, so we should be fine for now in that regard.

To make pubsub subscriptions work, I have have polyfilled ReadableStream and integrated the stream's controller with XHR's progress events in React Native's fetch implementation. It's important to note that progress events only work when XMLHttpRequest.responseType is set to text (https://github.com/facebook/react-native/blob/v0.63.3/Libraries/Network/RCTNetworking.mm#L544-L547). If you wish to process raw binary data, either blob or arraybuffer has to be used. In this case, the response is read as a whole, when the load event is fired, and enqueued to the stream's controller as single chunk. I'm currently using Content-Type header to selectively determine whether text or other response type should be used, but we probably should find another way to do this. 🤔

The patched fetch implementation still requires further work, but I will carry on with that effort over to https://github.com/react-native-community/fetch in next few days. I'm planning to rewrite the implementation in modern JS, add features that might be missing, remove code that is not relevant in the scope of React Native and setup a CI in GitHub Actions that runs the tests in an actual React Native app. Before proceeding with that, next week I'm quickly putting together a repo to compile down all these fixes I've been making to date into a library which app developers can install and import to patch the environment. 💪

[acostalima]

Patches and polyfills are now available as a module: https://github.com/acostalima/react-native-polyfill-globals.

[x5engine]

This is epic

[acostalima]

@autonome @hugomrdias

Roadmap of work ahead:

1. Fetch API for React Native - estimated to be completed until November 20th
   • Finish implementation (nearly concluded).
   • Take existing tests from GitHub's Fetch implementation and make them run in a React Native app (iOS and Android) with Mocha or other testing framework. Add tests for text streaming support (chunked encoding transfers).
   • Setup CI pipeline in GitHub Actions to run said tests automatically.

Technical discussion takes place at react-native-community/fetch#4.

2. Setup HTTP client test suite to run on React Native - November 23-27 (~1 week)
   • Counting on @hugomrdias' support for this task.
3. Open PRs in React Native repo to land minor fixes to a few APIs - December 2-4 (~3 days)
   • Known APIs to date required for the HTTP client to operate:
     • Implement FileReader.readAsArrayBuffer.
     • Implement missing FormData methods (at least .set).
     • Patch FormData.getParts to add support for Blob.
4. (?) To be discussed, if applicable, depending on existing priorities.

In general, there is evidence that the RN team is not interested in polyfilling or implementing standard web APIs and bundle them within RN's core due to bundle size concerns. Such concerns, as far as I know, are motivated by the Lean Core efforts.

Past discussions related to the above:

• Implement URL standard and URLSearchParams spec compliant facebook/react-native#25719
• Warn about URL and URLSearchParams not being spec-complaint facebook/react-native#30188
• Support for server-sent events (EventSource) react-native-community/discussions-and-proposals#99

Example APIs include: TextEncoder, TextDecoder, ReadableStream, URL and URLSearchParams.
Developers are encouraged to polyfill the environment as required to meet the requirements and use cases of their apps. IPFS can leverage https://github.com/acostalima/react-native-polyfill-globals for this purpose (which will be updated progressively).

[acostalima]

I've ran into a possible issue with the ReadableStream polyfill. Started a discussion at MattiasBuelens/web-streams-polyfill#68.

[acostalima]

Found an issue in RN about the usage of console.error and console.warn when consuming a Blob with FileReader.readAsDataURL. Started a discussion at facebook/react-native#30378.

[chrisdukakis]

Hi, has anyone managed to polyfill crypto? I use https://github.com/webview-crypto/webview-crypto but fails to serialize large files when I add().

EDIT: Managed to fix with webview-crypto/webview-crypto#8.
Now everything works. However, does someone know what this line does: https://github.com/libp2p/js-libp2p/blob/29597947966682b70e8751ddb1d022580557112a/src/keychain/index.js#L469
I had to comment it out, otherwise was getting Key 'self' already exists error.

[acostalima]

@chrisdukakis I'm sorry, I have no clue as to why you're getting that error in libp2p. I'm not familiar with libp2p's codebase.
I did polyfill window.crypto before, but I used react-native-get-random-values instead.

[RobertFischer]

I've also got a library that's a pure implementation of `randomBytes`: https://github.com/RobertFischer/randombytes-pure

…

On Sun, Jun 13, 2021 at 4:15 PM André Costa Lima ***@***.***> wrote: @chrisdukakis <https://github.com/chrisdukakis> I'm sorry, I have no clue as to why you're getting that error in libp2p. I'm not familiar with libp2p's codebase. I did polyfill window.crypto before, but I used react-native-get-random-values <https://github.com/LinusU/react-native-get-random-values>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2813 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAE5V7BSFBFR2MWQ2QBPL3TSUGWDANCNFSM4LASJE7A> .