ipfs.add async Iterable not truly asynchronous. Does not function as intended. #2838
Comments
|
When you see this behaviour are you running against a remote go or js daemon? |
|
@achingbrain go-ipfs. I have confirmed it works once async iterable is finished. |
|
Is the behaviour the same with a js-ipfs daemon? (I'm assuming when you said "if using js-ipfs ipfs.add will function as intended" you meant an in-process node rather than a remote one). |
|
The behavior is different (working as intended) with an in process js-ipfs daemon. Sorry I did not make that clear. |
|
Doing some further analysis reveals no request is actually sent to HTTP API Port 5001 for go-ipfs daemon. I believe the issue lies here https://github.com/ipfs/js-ipfs-http-client/blob/e9ce88d2523663b9730a0ffa28c6791f45976549/src/add/form-data.js#L15-L61 |
|
@vaultec81 are you running your code in the browser or in Node.js? If in the browser, ensure you've setup CORS correctly on your daemon. If in Node.js then what version are you running? There's an issue with |
|
@alanshaw I am running node v12. |
|
Ah, I just re-read your issue and yes this is a problem with how FormData works with the fetch API. File contents may be streamed but FormData needs to know the full list of files before it can be passed to the fetch API. |
|
yet there is no support for streaming Requests in browsers yet, only streaming Responses. |
|
@hugomrdias I don't see it being too hard to implement streamed uploads. it does require some changes to the ipfs http api specifications. I don't if its possible to do streamed multipart uploads. Potentially a websocket connection for
This isn't specifically a browser issue. |
|
browsers can not stream requests using HTTP, no way around it. |
|
@hugomrdias Can you please go into more detail what you mean by "stream requests" |
|
one can only send a blob/buffer from browsers using HTTP, there is no way to stream bytes from the browser. |
|
@hugomrdias I don't see it being a big deal implementing chunked request multipart uploads, Or better yet websocket which can upload data in a streamed fashion. Although websocket is not exactly HTTP, it is widely supported by the majority of web browsers, and common web infrastructure. |
|
its not that's why i changed this from bug to feature. |
|
@hugomrdias I believe it should still be treated as if its a bug as it causes functionality differences between js-ipfs in process daemon, and js-http-client. I am open to the possibly of submitting a PR. However, I believe it should be up to the IPFS team to develop a universal solution. |
|
The problem here is that in order for The documented way to do that is to pass a ReadbleStream as the request body, the problem is that no browser has implemented this - see the It doesn't help that the MDN docs talk about it like it's something you can use today. I think we have a few options:
My thoughts are not fully-formed on the matter but I think my preference would be for 3 though it'll have an affect on the bundle size. |
|
maybe some thing like this |
jacobheun
added
status/ready
Adds a server running a gRPC endpoint over websockets running on port 5003, a `ipfs-grpc-client` module to access the server and a `ipfs-client` module that uses the gRPC client with HTTP fallback. This is to solve shortcomings and limitations of the existing HTTP API and addresses the concerns raised in the 'Streaming HTTP APIs and errors, y u no work?' session we had at IPFS team week in NYC. ## Key points 1. Enables full duplex communication with a remote node When making an HTTP request in the browser, a [FormData][] object must be created. In order to add all the values to the FormData object, an incoming stream must be consumed in its entirety before the first byte is sent to the server. This means you cannot start processing a response before the request has been sent, so you cannot have full-duplex communication between client and server over HTTP. This seems unlikely to change in the near future. With a websocket transport for gRPC-web, individual messages can be sent backwards and forwards by the client or the server enabling full-duplex communication. This is essential for things like progress events from `ipfs.add` in the short term, and exposing the full stream capabilities of libp2p via remote client in the long term. 2. Enables streaming errors The existing HTTP API sends errors as HTTP trailers. No browser supports HTTP trailers so when a stream encounters an error, from the client's point of view the stream just stops with no possibility of finding out what happened. This can also mask intended behaviour cause users to incorrectly interpret the API. For example if you specify a timeout to a DHT query and that timeout is reached, in the browser the stream ends without an error and you take away the results you've received thinking all is well but on the CLI the same operation results in a non-zero exit code. A websocket transport has no restrictions here, since full-duplex communication is possible, errors can be received at any time. 3. Listens on websockets with no HTTP fallback gRPC-web exists and is a way of exposing a gRPC service over HTTP. Whereas gRPC supports four modes (unary, e.g. one request object and one response object, client streaming, server streaming and bidirectional streaming), gRPC-web only supports [unary and server streaming](https://github.com/grpc/grpc-web#wire-format-mode). This is due to limitations of the web platform mentioned above and doesn't give us anything over our existing HTTP API. The gRPC-web team are evaluating several options for client and bidirectional streaming, all of which require new capabilities to be added to browsers and none of which will be available in a reasonable time frame. Notably they have [no plans to use websockets](https://github.com/grpc/grpc-web/blob/master/doc/streaming-roadmap.md#issues-with-websockets) as a transport, even though it solves the problems we have today. The team from [improbable](https://improbable.io/) maintain a [gRPC-web-websockets bridge](https://github.com/improbable-eng/grpc-web) which the client added by this PR is compatible with. Their bridge also has a go implementation of a [reverse proxy](https://github.com/improbable-eng/grpc-web/tree/master/go/grpcwebproxy) for use with gRPC servers to turn them into gRPC-web servers with an optional websocket transport. My proposal is to embrace the use of websockets to solve our problems right now, then move to whatever streaming primitive the gRPC-web team settle on in the years to come. As implemented there's only websockets here and no HTTP fallback as the existing HTTP API works fine for unary operations so there's little to be gained by blocking this work on reimplementing the whole of the HTTP API in gRPC-web, and the client can pick and choose which API it'll use per-call. By running the websocket server on a different port to the existing HTTP API it gives us room to add gRPC-web fallback for the API if we find that useful. 4. Has protobuf definitions for all requests/responses See the [ipfs-grpc-protocol](https://github.com/ipfs/js-ipfs/tree/feat/add-grpc-server-and-client/packages/ipfs-grpc-protocol) module, which contains definitions for API requests/reponses. They've been ported from the existing API and will need some checking. The [ipfs-grpc-server/README.md](https://github.com/ipfs/js-ipfs/blob/feat/add-grpc-server-and-client/packages/ipfs-grpc-server/README.md) has a rundown of the websocket communication protocol that was ported from [improbable-eng/grpc-web](https://github.com/improbable-eng/grpc-web). 5. Options as metadata When making a request, metadata is sent during the preamble - these take the form of a string identical to HTTP headers as the initial websocket message - I've used this mechanism to send the options for a given invocation. Notably these are not defined as a protocol buffer, just an unspecified list of simple key/value pairs - maybe they should be to ensure compatibility between implementations? This will be trivial in the implementation in the PR as it contains a server implementation too but to do it in go will require patching or forking the improbable gRPC proxy. 6. Errors as metadata Similar to the existing HTTP API, message trailers are used to send errors. Four fields are used to re-construct the error on the client side: | Field | Notes | | ----- | ----- | | grpc-status | 0 for success, 1+ for error | | grpc-message | An error message | | grpc-stack | A stack trace with `\n` delimited lines | | grpc-code | A string code such as `'ERROR_BAD_INPUT'` that may be used for i18n translations to show a message to the user in their own language | Similar to options these fields are unspecified, if a convention is not enough, perhaps they should be specified as a protobuf and the trailer sent as binary? 7. Streams When sending data as part of an `ipfs.add`, we send repeated messages that contain a path, a content buffer and an index. The index is used to differentiate between streams - path cannot be used as it could be empty. Only the first supplied `path` is respected for a given index. On the server separate input streams are created for each file being added. A file stream is considered closed when an unset or empty content buffer is received. Ultimately this will allow us to apply backpressure on a per-file basis and read from different file streams in parallel and asymmetrically based on the available server capacity. 8. Performance Observed performance pegs gRPC-web over websockets as similar to the HTTP Client with pretty much zero optimisation work performed 9. Security Browsers require TLS for all use of websocket connections to localhost. They do not require it for the loopback address, however, which this PR uses, though loopback means the traffic will not leave the local machine. The incoming requests start as HTTP requests so have a referer header and user agent so would follow the same restrictions as the existing HTTP API. Fixes #2519 Fixes #2838 Fixes #2943 Fixes #2854 Fixes #2864 [FormData]: https://developer.mozilla.org/en-US/docs/Web/API/FormData
Hello
I have noticed after doing some experimentation that ipfs-http-client requires the input async iterable to be completed in order for the request to proceed. There is no output until the input async iterable is completed. Meaning the code functions no different then if an array was specified instead.
However, if using js-ipfs in process daemon ipfs.add will function as intended. Returning CIDs through the output async iterable as the input async iterable is processed.
In my use case I am reading the output, and continuing to pass input as it is processed.
Example code
The text was updated successfully, but these errors were encountered: