Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement window.fetch into core #19393

Open
thecodingdude opened this issue Mar 16, 2018 · 251 comments
Open

Implement window.fetch into core #19393

thecodingdude opened this issue Mar 16, 2018 · 251 comments

Comments

@thecodingdude
Copy link

@thecodingdude thecodingdude commented Mar 16, 2018

Edit: please note that this issue is pretty old and a lot of the information in the first few comments isn't up to date - for current status please see #19393 (comment) .


https://github.com/bitinn/node-fetch

It would make sense if window.fetch was implemented into core. it seems to be a stable enough API that would make a good candidate for inclusion. Not sure what the process is from here but thought I'd raise an issue :)

@jasnell
Copy link
Member

@jasnell jasnell commented Mar 16, 2018

this has come up from time to time but has not had much traction just yet. Let's see what folks think tho :-)

Loading

@devsnek
Copy link
Member

@devsnek devsnek commented Mar 16, 2018

bradley and i were discussing this in a roundabout way on the subject of importing from urls. if that feature was introduced (and i think in general we do want it) we would need to implement this: https://html.spec.whatwg.org/multipage/webappapis.html#fetch-a-single-module-script which uses the fetch spec. as another note if this was added in core i would want to pull in an existing c++ implementation from one of the browsers. however at a bare minimum node will definitely be adding Request and Response objects, it just might not add a function called fetch

Loading

@mscdex
Copy link
Contributor

@mscdex mscdex commented Mar 16, 2018

-1 this kind of higher-level functionality is best left to userland

Loading

@guybedford
Copy link
Contributor

@guybedford guybedford commented Mar 16, 2018

This would definitely be useful in simple cross-platform APIs and I think is what a lot of people use node-fetch for already. Also it would be nice if HTTP/1 v HTTP/2 negotiation can be handled automatically like in browsers as well.

Loading

@styfle
Copy link
Member

@styfle styfle commented Mar 17, 2018

I would love this! ❤️

Isomorphic JS is one of the big reasons people who start with JS on the front end, eventually pick up Node.js on the backend.

You get to run the exact same function in the browser and the server and the one place of contention I keep finding is window.fetch.

Some of the code that runs on the server and client needs to make HTTP requests to another server (think microservices with server side rendering and client side rendering).

One case for bringing it into core is that making HTTP requests (client) is closely tied to responding to HTTP requests (server).

And we now have isomorphic URL parsing so now all we need is fetch! Let’s make fetch happen!

Loading

@mikemaccana
Copy link
Contributor

@mikemaccana mikemaccana commented Mar 17, 2018

Fetch is 'low level' according to it's author and major supporter hence missing a bunch of features like support for content types, JSON not being default, no query string encoding. Apps that need a high level quality HTTP client available in all JavaScript environments can continue using superagent.

Loading

@devsnek
Copy link
Member

@devsnek devsnek commented Mar 17, 2018

@mikemaccana the fetch we are talking about is https://fetch.spec.whatwg.org/ and i don't think its appropriate to be plugging other http libraries

Loading

@mikemaccana
Copy link
Contributor

@mikemaccana mikemaccana commented Mar 17, 2018

@devsnek Yes I know, that's the one I was specifically referring to. I don't have any particular enjoyment of superagent asides from it being:

  • a full featured HTTP client
  • available in node and the browser
  • more popular than fetch
  • has JSON as a default
  • encodes query strings
  • uses content types to determine response body
  • uses HTTP verbs as method names, so you can happily .get() and .post() things rather than 'fetching with method POST' which is a somewhat odd mental model

If fetch supported these I'd suggest it be included in node. To repeat: I've asked fetch's author and main proponent, Jake Archibald, why it doesn't support these things and it's stated that fetch is designed to be a low level API and things like sensible defaults can/should be added by higher level APIs. As a result I see no reason to use fetch in node, or in the browser.

Loading

@joyeecheung
Copy link
Member

@joyeecheung joyeecheung commented Mar 18, 2018

@mikemaccana

These are actually desirable properties for the argument of including fetch in Node.js core, if we want to keep the core low-level and small. I believe we will need to eventually provide a promisified API for http/http2 anyway, and fetch as an existing spec for a similar set of low-level functionality as our http is something worth considering. In my experience the missing pieces in fetch feels pretty similar to the missing pieces in Node.js's http, the major difference is that you have a spec'ed API to count on.

Also I kind of doubt the "more popular than fetch" part, I don't think there are as many people using super-agent in the browser as in Node, given that fetch simply exists in the browser (modulo situations needing polyfills)?

Although, before we start implementing fetch, I believe we will need to introduce the stream API in the browser? Is introducing yet another stream in Node.js on the table?

The one time I've been forced to resort to XHR in the browser was when I needed progress, although I think with the browser's ReadableStreams it's possible to do the same thing with an API that's a bit awkward (res.body.read().byteLength)- if I'd implement progress in Node.js I think I'll need to use chunk.byteLength from the chunk being emitted in the data event, which is where the difference between the two streams start to matter.

Also, the fetch spec does not seem to include timeout or agents (yet?) at the moment, there might be more functionalities missing compared to our existing http API. For reference, node-fetch seems to implement these non-standard options. Again, not sure if our implementation should implement non-standard functionalities even if they supply the missing pieces compared to the old API.

Loading

@joyeecheung
Copy link
Member

@joyeecheung joyeecheung commented Mar 18, 2018

also cc @TimothyGu you might be interested?

Loading

@joyeecheung
Copy link
Member

@joyeecheung joyeecheung commented Mar 18, 2018

@thecodingdude

entirely disagree; Node uses v8, and by extension, should implement as many as v8 features as possible that make sense. fetch is one of those where developers wouldn't need to npm install request or node-fetch which are very popular libraries so this functionality warrants being in core.

Technically fetch is not a v8 feature though, it's an API spec'ed by WHATWG, whereas v8 implements ECMA-262, a spec by ECMA TC39 - if v8 implemented fetch then there would not be this feature request, because we basically just expose what v8 exposes.

BTW: I don't think we are talking about pulling npm packages into the core? Rather, we are talking about implementing the spec on our own and pulling in the WPT to test compliance, possibly with a bunch of code written in C++, much like what we did for WHATWG URL.

Loading

@mikemaccana
Copy link
Contributor

@mikemaccana mikemaccana commented Mar 18, 2018

@thecodingdude the continued popularity of non-fetch libraries isn't a personal opinion, it is a fact - superagent had 1.6 million downloads this week . Nor is the lack of reasonable high-level defaults in fetch: again (again) that is acknowledged by fetch's author. Please don't reframe verifiable objective technical facts as irrelevant subjective opinions because you do not like them.

please don't plug other libraries here, they are irrelevant to our discussion.

developers wouldn't need to npm install request or node-fetch

😂👍

Loading

@bnoordhuis
Copy link
Member

@bnoordhuis bnoordhuis commented Mar 19, 2018

I'm personally -1 for two reasons:

  1. Having two different APIs (fetch() and http.request()) for doing the same thing is uncohesive.

  2. fetch() is not flexible enough to support things like proxying, custom TLS certificates, etc. It's handled transparently by the browser but that won't work in node.js.

currently, 84 people agree

I wouldn't put too much stock in that. GH polls are easily gamed through Twitter brigading and whatnot. I'd take them more seriously if it cost a dollar to cast an upvote.

Loading

@mikemaccana
Copy link
Contributor

@mikemaccana mikemaccana commented Mar 19, 2018

thecodingdude I have commented with a bullet pointed list of technical reasons why fetch is unsuitable as a general purpose HTTP client. I am aware you "do not care how/good bad fetch is". I think this is wrong: node should pick and adopt quality APIs. That you're personally uninterested in fetch's quality does not make it irrelevant to the discussion. The browser vendors were comparing fetch to window.xhr, node is not.

Loading

@Jamesernator
Copy link

@Jamesernator Jamesernator commented Mar 19, 2018

@mikemaccana Why does fetch need to have high level features like "HTTP verbs as method names" for it to be included in core? You can trivially write your own methods that do this, in addition to the other high level features you've mentioned.

The nice thing about having standard APIs shared across both the browser and Node is that by writing the high-level API to target the low-level one (in this case fetch) you can get consistent behaviour much more easily than having to make your API work around differences between different APIs in node and the browser.

Loading

@devsnek
Copy link
Member

@devsnek devsnek commented Mar 19, 2018

@thecodingdude any reason this was closed? discussion still seems to be moving

Loading

@thecodingdude
Copy link
Author

@thecodingdude thecodingdude commented Mar 19, 2018

@devsnek yeah, I am not satisfied with the way this discussion and issue went, and I realise that github issues are simply not the appropriate forum for discussing whether features should land into core. I much prefer the node-eps since that was a proper document outlining the proposal. My intention was to gather the thoughts of the core team, and whomever else is involved in merging features into core (planning, coding, testing etc). Instead the discussion here so far is basically meaningless. It should be about feasibility, what would be required, and when such a feature could be merged.

Go take a look at this issue and try and follow the avalanche of conversation about websockets in core. There's too much noise and not a straightforward approach to proposing new features. I think PHP's rfc system is a good example of getting new features added. Clear, concise, accountable.

I'm down for discussing an rfc and its merits in a separate issue but it's clear this isn't going anywhere, with little to no interaction from TSC or anyone else that matters so there's no point leaving it up just to spam people's inboxes.

I'll refrain from making such issues in the future, my bad.
cc @jasnell

Loading

@mikemaccana
Copy link
Contributor

@mikemaccana mikemaccana commented Mar 19, 2018

@Jamesernator Why does fetch need to have high level features like "HTTP verbs as method names" for it to be included in core? You can trivially write your own methods that do this, in addition to the other high level features you've mentioned

Sure. I, and everyone else that needs to use query strings, can write a method to encode query strings. Better yet: let's include a good quality, batteries included HTTP client in node where everyone doesn't need to fix it.

@thecodingdude looks like you deleted your comment, but to answer anyway: HTTP2 is necessary because it is required to speak the current version of HTTP. fetch is not necessary, because it is but one of many HTTP clients, and not a very good one, for the reasons discussed.

Loading

@Jamesernator
Copy link

@Jamesernator Jamesernator commented Mar 19, 2018

@mikemaccana But the reason for including fetch in Node core isn't about batteries included, it's so that you can write code that targets both the browser and Node at once. That's why Node also supports the URL constructor (and it's even now global like in the browser in node 10) and the docs even refer to the old url.Url objects as the legacy url API.

This is a good thing, as it means some module that uses dynamic import for instance they don't need to include any library to be able to load resources relative to it e.g.:

...

export default async function isWordForLanguage(language="english") {
    const wordDataUrl = new URL(`./resources/${ language }.csv`, import.meta.url)
    const words = await loadSomehow(wordDataUrl)
    return makeIsWord(words)
}

Now that code has no platform specific behavior (assuming a loadSomehow method which could potentially be fetch). The benefits to this should be fairly obvious:

  • Using APIs that are standardised and included in both the browser and Node means you'll get support in both contexts if there's any issues with implementations (and hence no chance of dead or deprecated libraries)
  • Smaller libraries for high-level features as they don't need to smooth over platform differences, they just use the universal API (URL or fetch or whatever) and build directly on top of that
  • Ditto for your own libraries that need high-level behaviour

I think it's a pointless endeavour to create another http library in Node if it isn't for interoperability. If Node creates another one that isn't in the browser then you simply get the same situation you have now, if you want browser inter-operable code you another library built on top of it just to get interoperability with browsers.

If you think it's important for browsers to support high-level APIs out of the box then it's probably worth supporting the efforts to make high level APIs part of the browser and make clear your goals what you want to see out of high-level APIs. I'm sure many developers would appreciate getting high-level API specifications.

Even better would be if there was a way to get Node and Browsers to collaborate on high-level features, I'm not sure how this might work but if it could be done then it'd be a lot easier to propose high-level APIs that target all environments instead of being silo-ed into Node or the browsers.

Loading

@mikemaccana
Copy link
Contributor

@mikemaccana mikemaccana commented Mar 20, 2018

Thanks @Jamesernator - it's nice to know there's an effort to resolve this on a larger scale. Thanks also for being polite and actually reading my comments before responding.

Loading

@guybedford
Copy link
Contributor

@guybedford guybedford commented Mar 25, 2018

I tend to think that were NodeJS to have been developed today, there's no doubt that fetch would have been implemented in the name of aligning with browser APIs. Yes there are some really interesting edge cases, but @TimothyGu has handled these really well in the node-fetch integration.

@thecodingdude do you mind if I reopen? We potentially could still have things line up here - I don't think the detailed discussion has run its course fully quite yet and I think there's still fruitful discussion to be had personally.

@bnoordhuis node-fetch handles the differences here by supporting the NodeJS-specific agent option that delegates features like proxy https://github.com/bitinn/node-fetch#options. It breaks the universal API, but could also be done conditionally quite easily, so I tend to think it seems a fairly good compromise with this stuff. It could also be decided that such a divergence is a bad idea as well though certainly.

The key thing though is that this fetch API is purposely restricted - HTTP and HTTP/2 APIs still remain fundamental, this could just provide the easy universal experience. There's nothing wrong with having two APIs to do the same thing, when one builds on top of the other to provide a compelling user experience.

Loading

@thecodingdude
Copy link
Author

@thecodingdude thecodingdude commented Mar 25, 2018

@guybedford feel free to make a new issue discussing this further if that's the direction that is most suitable. I'd much prefer technical discussions about the implementation itself, and not what fetch does/doesn't do vs alternatives which is why I closed the issue in the first place.

Quite honestly, I don't see why this discussion needs to happen on github - at the least is should be locked to collaborators who are going to be working on the code and discussions focused entirely on implementing fetch, identical to the browser where possible, and nothing more and nothing less.

Loading

@guybedford
Copy link
Contributor

@guybedford guybedford commented Mar 26, 2018

@thecodingdude I'm still not quite sure I follow why you need a new issue here. Discussing the problem boundaries should surely be the first step for any new feature, and allowance should be made for a wider discussion at the start as well. 16 comments hardly seems off the rails as well.

Certainly the technicalities of whether it should be JS or C++, or where to draw the line on pooling and proxies are important, but things have to run their own course.

Loading

@bnoordhuis
Copy link
Member

@bnoordhuis bnoordhuis commented Mar 26, 2018

There's nothing wrong with having two APIs to do the same thing, when one builds on top of the other to provide a compelling user experience.

But it's not 'on top of' - they do the same thing, just with different interfaces.

Arguments in favor of fetch() boil down to 'convenience' - not a good reason for inclusion in core.

Loading

@guybedford
Copy link
Contributor

@guybedford guybedford commented Mar 26, 2018

Thanks @bnoordhuis these are important points as to where the line is in Node, and I must admit I'm not familiar with these sorts of discussions historically, but so long as Node strives to provide core utilities, this seems like an important one to include to me.

But it's not 'on top of' - they do the same thing, just with different interfaces.

Currently node-fetch is built on top of the http module, I'd personally prefer such a JS-based implementation here, working from exactly what @TimothyGu has already built as one of the most popular libraries. The pattern of having high and low level APIs for the same thing is an established one.

Arguments in favor of fetch() boil down to 'convenience' - not a good reason for inclusion in core.

Convenience is a hugely important argument, the question is by how much to justify the maintenance costs. Node doesn't just support modules and native bindings - it provides server utilities and universal APIs where appropriate. As I've said, fetch seems to me a critical part of the universal toolbox these days.

I'd even push for fetch to transparently integrate HTTP/2 in future, allowing such a feature to not only build on top of the existing builtins, but also to provide a unification that we don't currently have. I know there are complexities to be tackled here, and it's not a given, but it feels like it would be huge win for Node and its users.

Loading

@devsnek
Copy link
Member

@devsnek devsnek commented Jun 8, 2021

I did try to integrate WPT but it's currently quite a lot more complex than what our wpt runner can currently handle (and also conflicts with the test runner's fetch stub 😱). The node-fetch tests are heavily choi'd and we would probably have to write some sort of fake choi to run them. I don't know when I'll find the personal motivation to pursue one of those two options, so that's where this is rn.

Loading

@Repugraf
Copy link

@Repugraf Repugraf commented Jun 10, 2021

@devsnek is there any PR where you working on it? Maybe you could share the link and somebody would help?

Loading

@devsnek
Copy link
Member

@devsnek devsnek commented Jul 8, 2021

I pushed my code here: https://github.com/devsnek/node/tree/fetch

Loading

@MJCD

This comment was marked as spam.

@benjamingr
Copy link
Member

@benjamingr benjamingr commented Jul 9, 2021

@devsnek is there a constructive way to contribute to your effort or is the best course of action right now to just wait?

Loading

@devsnek
Copy link
Member

@devsnek devsnek commented Jul 9, 2021

@benjamingr if someone could figure out how to get tests working that would probably be the most helpful thing. node-fetch tests require mocha and the wpt tests require a bunch of changes to our test runner

Loading

@MJCD

This comment was marked as disruptive content.

@jimmywarting
Copy link

@jimmywarting jimmywarting commented Jul 9, 2021

@devsnek do you wan't to chat with me? I have helped out a lot with node-fetch
I'm available on messenger, hangout and discord

Loading

@anlexN

This comment was marked as spam.

@max-hk
Copy link

@max-hk max-hk commented Jul 10, 2021

@devsnek Isn't there undici-fetch? It seems to be maintained by Node.js TSC members (@mcollina, @Ethan-Arrowood, @ronag) and would probably merged into undici, a http library under the nodejs organization. Wouldn't it a better choice than node-fetch?

Loading

@devsnek
Copy link
Member

@devsnek devsnek commented Jul 10, 2021

@maxloh I'm not entirely sure what you're asking. I'm not suggesting that we merge node-fetch into node core, if that's what you mean.

Loading

@jimmywarting
Copy link

@jimmywarting jimmywarting commented Jul 10, 2021

☝️ exactly. Taking something from userland and put into the core is not as simple as copy & paste.
needs to be rewritten with NodeJS syntax flavor, lazyloaded, requiring builtin AbortController instead of polyfilled userland, confirm 100% to the spec and not include any any external dependencies and possible do some webidl magic conversion on arguments

node-fetch isn't a silver bullet when it comes to supporting http2/3 quic or whatever protocol we have nowdays - I'm a bit outdated in that area.
we have some node specific that should not be in the core. response.body should for instances be a whatwg stream

Loading

@Ethan-Arrowood
Copy link
Contributor

@Ethan-Arrowood Ethan-Arrowood commented Jul 10, 2021

There's an branch on undici-fetch where I was starting to do a whole "run node-fetch tests using a different fetch implementation", but I stopped working on that to focus on v1 and benchmarking.

There's also a fetch headers class hopefully landing in node core soonish; I've been very very busy at work and haven't had time to work on it recently but it is top of my priority list when I get time for open source again (shouldn't be more than a week or two at most)

Loading

@jimmywarting
Copy link

@jimmywarting jimmywarting commented Jul 10, 2021

@Ethan-Arrowood That's grate news, more spec'ed things to make fetch happen in core

Loading

@devsnek
Copy link
Member

@devsnek devsnek commented Jul 11, 2021

i realized core fetch means core instantiateStreaming :dab:

Loading

@jimmywarting
Copy link

@jimmywarting jimmywarting commented Jul 23, 2021

One thing many have asked for in node-fetch is the ability to measure speed. We have long thought of plugin system, but really the right way to go according to some spec is to implement PerformanceResourceTiming I know Deno is looking into it also. Would be cool to have a spec to follow that works the same across everywhere

Loading

@ronag
Copy link
Member

@ronag ronag commented Aug 11, 2021

We're pretty close to landing an initial fetch implementation in https://github.com/nodejs/undici (which we are discussing whether to include in core). Jump over to the PR if you want to help with review.

Loading

@tom-sherman
Copy link

@tom-sherman tom-sherman commented Sep 15, 2021

@ronag Where can I find discussions about including undici/fetch in core?

Loading

@jfsiii
Copy link

@jfsiii jfsiii commented Sep 15, 2021

Loading

@tom-sherman
Copy link

@tom-sherman tom-sherman commented Sep 15, 2021

Now that the undici-fetch implementation has landed, can work begin on including the fetch (and friends) as globals? I'd love to lend a hand with this

Loading

@mcollina
Copy link
Member

@mcollina mcollina commented Sep 15, 2021

I'm not sure we are exactly ready yet, I would prefer to have a few things stabilize in core and undici before adding to core.

If you would like to help, check out https://github.com/nodejs/undici/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22stable+fetch%22.

Loading

@voxpelli
Copy link

@voxpelli voxpelli commented Sep 15, 2021

If the conclusion here is that when/if window.fetch is going to be implemented in core, then it's going to be through undici, then I guess this issue can be closed in favor of #38533 as that one is discussing how to move forward with the future of the Node HTTP Client and how undici fits into that and how it can be the way forward.

It in turn depends on nodejs/TSC#1041 / #39779 which discusses the principles around what belongs in core or not.

Anyone of a different conclusion?

Loading

@mcollina
Copy link
Member

@mcollina mcollina commented Sep 15, 2021

I concur with @voxpelli

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet