utility crate: fetch and making HTTP requests

[fitzgen]

We should wrap up the fetch API into a nice Rust-y utility crate!

[fitzgen]

Can we make the dual to https://crates.io/crates/http-service ?

[eminence]

maybe there are Bonus points to be had for developing an crate that can also work outside of the browser/JS context. For example, if someone were to develop a library for the github API or something, they might want their crate to be usable as both a standard crate linked into a CLI application, or as a wasm crate.

[jaynagpaul]

Even more bonus points for contributing wasm support to an already established library? seanmonstar/reqwest#288

[David-OConnor]

Can something like Request be modified to support wasm without reinventing it, or does it need to use something like web_sys::request_with_... ?

@Pauan wrote a general wasm fetch lib using Futures, with a nice API

[fitzgen]

@Pauan wrote a general wasm fetch lib using Futures, with a nice API

Sweet! Care to share a link?

[David-OConnor]

https://github.com/David-OConnor/seed/blob/master/src/fetch.rs

Guide

[David-OConnor]

@Pauan Want to submit a PR?

[Pauan]

@David-OConnor I'm not sure if that's a good idea. I think we should first figure out whether we're going to be creating something new or contributing to an existing project, and I also think we should discuss a bit about the API.

[tomhoule]

I would be interested in contributing "mid-level" bindings (as discussed in issue #27) for XHR. As it would be a straightforward wrapper for web_sys::XmlHttpRequest with more idiomatic Rust types, does it need to go through a design phase, or wait until #27 is resolved? (there seems to be consensus there).

[fitzgen]

@tomhoule great! I think the next thing we need to push this forward is an API design proposal (here is a template you can use: https://github.com/fitzgen/gloo/blob/master/.github/ISSUE_TEMPLATE/propose_design.md)

For the "mid-level" bindings I expect the design and consensus work to be fairly straightforward.

One question I do have right off the bat, that I think the proposal should address and/or leave a TODO hole for, is whether we want to build atop XHR or fetch or have two mid-level lilbraries for each of them, or one mid-level library that builds on both or...

[Pauan]

One question I do have right off the bat, that I think the proposal should address and/or leave a TODO hole for, is whether we want to build atop XHR or fetch or have two mid-level lilbraries for each of them, or one mid-level library that builds on both or...

That's a really good question! Fetch is the new high-level "blessed" API, but I don't think it fully supports all of the features of XHR, so maybe having both is a good idea?

Also, since fetch uses Promises, it matches very naturally to Rust Futures, so we could treat XHR as the "mid-level" API, and fetch as the "high-level" API.

[tomhoule]

My specific use-case with XHR at the moment is file uploads. To the best of my knowledge fetch doesn't support tracking upload progress, so it would make sense to have both. As the fetch API gets more features it will be the better option however (it is already, for many use-cases).

The two APIs are very different so a common wrapper (XHR and fetch) would definitely be in "high level" territory in my opinion. It could look something like axios, but I would be happy to just have the callback-based XHR API with nicer Rust types for now.

One more thought on @Pauan's last comment: if the crate structure gloo adopts ends up being futures and callback modules, the callback API would make little sense for fetch (as you noted, it's already promise-based so it maps bettter to futures), and the futures approach would be more opinionated for XHR, as it would look completely different from the original callback-based API.

[tomhoule]

API design proposal for XHR: #37

[yoshuawuyts]

In JavaScript / Node the xhr and xhr-request crates have been a joy to work with. In particular one thing I think they nailed is that they managed to evade the "double await" design that the newer fetch API introduced.

fetch problems
The "double await" design is the part where fetch forces each request to become a two-phase process, bridged by a temporary binding in the center:

// text
let req = await fetch('/endpoint')
let res = await req.text()
console.log(res)

//json
let req = await fetch('/endpoint')
let res = await req.json()
console.log(JSON.stringify(res))

The temporary binding exists so users can decide how to parse a result based on the headers. This is not common though, and forces everyone to name the temporary binding rather than being able to chain and fit it on one line.

xhr
A comparable xhr-style promise API doesn't need to name a temporary binding, and can skip directly to the result:

// text
let res = await xhr('/endpoint')
console.log(res)

// json
let res = await xhr('/endpoint', { json: true })
console.log(JSON.stringify(res))

bringing it to Rust
In Rust we can't quite do the arbitrary parsing that JS does, so we'll usually define our expected data in a struct (rather than a hashmap). If we then apply the same concepts as with the xhr version, we could get to something that roughly fits on one line, and evades awkward temporary bindings.

// text
let res = xhr::get("/endpoint").send().await?;
dbg!(res);

// json
struct User { name: String }
let user: User = xhr::get("/endpoint").send_json().await?;
dbg!(user);

This is not to say this should be the only API possible. But I think that if we manage to make most common interactions (JSON, forms, multipart, etc.) available through convenient APIs, then we'll be able to create a pretty compelling story for making HTTP requests in Rust

conclusion
I've tried to explain fetch's "double await" design, some of the problems it has, and a possible direction we could take for our libraries to avoid it.

[richard-uk1]

Having a good http client will be a key part of making gloo a success, but it is a very big job. I would propose that we first create nice zero-cost libraries around xhr and fetch, using the onion philosophy. We could break them out into 2 separate issues and explore their design separately.

Once we are happy with those, we could begin to specify how a client would look. The first piece of work would be to do a review of existing solutions (e.g. reqwest, hyper, node, haskell, whatever is considered useful) and write that up. That would be a starting point for designing the api for the request library.

[David-OConnor]

Thoughts on getting a not-perfect-but workable module released?

[yoshuawuyts]

Oh yeah, I forgot to update this issue but we released https://docs.rs/surf/1.0.2/surf/ a few weeks ago, which works in the browser out of the box (:

[CryZe]

What's the status of this? I think creating a low level library built on top of the http crate and JS's fetch API should be fairly simple.

[yoshuawuyts]

Can we make the dual to https://crates.io/crates/http-service ?

https://docs.rs/http-client/1.0.0/http_client/ has now been published!

[CryZe]

That seems like a decent start, but it seems to ignore all request headers. But I guess I might use this.

[David-OConnor]

Seed's fetch module API docs, for ref. Built on web_sys's fetch.

[CryZe]

Both of these seem unfitting for the job. The Seed implementation is not based on the http crate (and is a whole framework, not just a tiny wrapper around fetch) and http-client is too immature and stuck with futures 0.1.

I'll go with my own implementation in the meantime.

[yoshuawuyts]

http-client is too immature and stuck with futures 0.1

We're using futures-preview 0.3.0; with std futures + the current shape of the stream traits. The only reason why we haven't upgrade yet to futures 0.3.0 proper is because the curl library we rely on hasn't moved over yet, but that work is currently happening and should be done in a few days.

[richard-uk1]

I'm trying to push forward this issue by adding webidl iterator support to wasm-bindgen at rustwasm/wasm-bindgen#1913. Using iterators seems to be the only way to get at the full list of headers in the response.

[CryZe]

Nice, I'm also slowing preparing a gloo-http PR. I've been using my implementation for quite a bit and it seems pretty robust now.

[richard-uk1]

@CryZe nice I've had a look at your work on your github and it looks cool!

I haven't written this up yet, but i was thinking of proposing that the http stack in gloo looked something like

1. gloo-xhr - wraps web_sys making xmlhttprequest as 'rusty' as possible
2. gloo-fetch - wraps web_sys making fetch as 'rusty' as possible
3. gloo-http - provides a high-level interface for http using the http crate for all the types. Can delegate to gloo-xhr or gloo-fetch.

Of course the alternative is to wrap the whole thing into gloo-http. What are your thoughts?

[CryZe]

Yeah I'll look into making the design closer to this. However XHR is probably out of scope for that PR at least (and maybe in general? Pretty sure every wasm compatible browser has fetch).

[richard-uk1]

@CryZe yeah now you say that I realise you're right and I can't see any reason to support xhr. Maybe just gloo-http using fetch then. :)

[CryZe]

That's actually a good point. Actually my implementation was initially based on XHR and apparently you only get a very raw unparsed view of the response headers (unless I missed something). So we may need to bring the httparse crate for that implementation.

[richard-uk1]

@dakom if we were only supporting up-to-date browsers, do AbortControllers do enough to cover all use cases?

[ranile]

Why do we need that when reqwest has WASM support (with web-sys)?

[richard-uk1]

One reason is that reqwest is opinionated, and another is that it is a heavier dependency. gloo aims to provide minimal wrappers over web APIs that make them Rust-y whilst adding as little overhead as possible.

[vnermolaev]

has there been any progress on the issue?
if I may ask: what's the status quo: what is the best current approach?

[ranile]

has there been any progress on the issue?
if I may ask: what's the status quo: what is the best current approach?

@vnermolaev, I have written a crate, reqwasm, specifically for this. Please take a look.

[ranile]

We have two options to move forward here:

• Move reqwasm's code into gloo
• Re-export reqwasm from gloo

pub use reqwasm::http as fetch
pub use reqwasm::websocket as websocket

I personally prefer the latter approach as it allows reqwasm to grow on it own, e.g. adding framework support behind feature flags, etc while also allowing it to be used from gloo.

Thoughts?

[yoshuawuyts]

localghost::net is probably also worth mentioning in this thread. It exposes fetch, server-sent events, and Beacon APIs.