Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request’s Past, Present and Future #3142

Open
mikeal opened this Issue Mar 30, 2019 · 31 comments

Comments

Projects
None yet
@mikeal
Copy link
Member

mikeal commented Mar 30, 2019

Before I go into the details and reasoning I’ll get straight to the point. The most valuable thing request can do for the JavaScript ecosystem is to go into maintenance mode and stop considering new features or major releases.

Apologies in advance to the other committers on request that have been doing their best to improve it, but it’s for the best.

2009

The first version of request was one of the first modules ever created for the Node.js ecosystem. The earliest versions were written to APIs that pre-date the standard callback interface, streams, node_modules and npm. For the first few years, request and Node.js evolved together, each learning from the other. As Node.js improved and migrated core interfaces so did request. As request adopted changes to the core http library and streams it also informed improvements like the pipe event (which enabled request’s one line proxy) and one of Core http’s many re-writes (the one I had to write).

npm

request was one of the first modules added to the npm registry. As npm grew so did dependence on request. Even now, when npm is used far more for front-end than back-end work, request remains one of the most depended on modules in the registry. As I write this, 41K modules depend on request and it is downloaded 14 million times a week.

The place request has in the Node.js ecosystem is no longer one of an innovator but of an incumbent. If you Google for how to do something with HTTP in Node.js the examples are likely to show request as the client and express as the server. This has two notably bad effects.

It’s much harder for new libraries accomplishing similar tasks to gain adoption because of the incumbent position request holds over the ecosystem. It’s also very hard to change request in any meaningful way as the change not only may not be adopted by the majority of its dependents but it would put it out of alignment with the thousands of blog posts and stack overflow responses that use request.

Modern JavaScript

The last few years have been dramatic ones in JavaScript. Features people had talked about for years went from ideas, to standards, to features you can reliably depend on in most environments. The speed at which these have been adopted is staggering, mostly thanks to auto-updating browsers and an aggressive Node.js release schedule.

The patterns at the core of request are out of date. A few people might argue with that assessment, and I know who they are so I won’t be surprised, but it’s true. I have often been skeptical of the impact some of these features would have only to find myself adopting them wholesale not long after they are available in only the latest release of Node.js.

There’s a transition happening now in the ecosystem to these patterns. How messy that will be is still up in the air and I’m not going to try and read the tea leafs and figure out what the future looks like in that regard. The question for request is “Do we try to survive through that transition?” A year ago, I thought the answer was obvious and that we would, but now I’m convinced of the opposite.

A version of request written to truly embrace these new language patterns is, effectively, a new module. I’ve explored this space a bit already and have a project I’m quite happy with but it is incompatible with request in every conceivable way. What’s the value in a version of request that is incompatible with the old patterns yet not fully embracing the new ones? What’s the point in being partially compatible when there’s a whole world of new modules, written by new developers, that are re-thinking these problems with these patterns in mind?

The best thing for these new modules is for request to slowly fade away, eventually becoming just another memory of that legacy stack. Taking the position request has now and leveraging it for a bigger share of the next generation of developers would be a disservice to those developers as it would drive them away from better modules that don’t have the burden of request’s history.

Maintenance Mode

Here’s the plan.

  • request will stop accepting new features.
  • request will stop considering breaking changes.
  • The committers that are still active will try to merge fixes in a timely fashion, no promises though.
  • Releases will be fully automated, any merge into master will be published. I’ve already built this for some other projects using GitHub Actions.
    • We’re going to have to remove inactive collaborators and enforce 2fa, because commit rights will effectively become npm publish rights.
@reconbot

This comment has been minimized.

Copy link
Contributor

reconbot commented Mar 30, 2019

I fully support this, I think a warning message and/or deprecating new releases is in order.

As for the change in process and guidelines, it makes my job a lot easier 👌

@simov

This comment has been minimized.

Copy link
Member

simov commented Mar 31, 2019

Very well said @mikeal. I'm pinning this issue to gain more visibility.

@simov simov pinned this issue Mar 31, 2019

@reconbot

This comment has been minimized.

Copy link
Contributor

reconbot commented Mar 31, 2019

Things we might do - please discuss and volunteer!

  • update readme with current state of project
  • update ci publishing pipeline @mikeal
  • provide a doc with some guidance on request alternatives #3143
  • add a warning message on install of the package to use another package and reference the doc
  • pick a date to stop support (I vote 6 months, but 12 is probably friendlier)
  • close all feature requests and feature prs
  • review and merge relevant bug fixes
  • add github issue and pr templates explaining that features wont be merged
  • deprecate the next major version (3.x) so project’s in active maintenance get a warning but older projects continue as usual
@analog-nico

This comment has been minimized.

Copy link
Member

analog-nico commented Apr 1, 2019

It makes a lot of sense! I will slowly adopt this policy for the request-promise family as well. Cheers to your important contributions to the node ecosystem!

@CliffS

This comment has been minimized.

Copy link

CliffS commented Apr 4, 2019

This decision must have been very hard to take but is commendable in the extreme. Well done.

@rimiti

This comment has been minimized.

Copy link

rimiti commented Apr 4, 2019

I'm proud having used this amazing tool. It forced the community to improve. 🙏
If you need help to maintain it do not hesitate to contact me.

@pdxmholmes

This comment has been minimized.

Copy link

pdxmholmes commented Apr 4, 2019

While I respect your decision, I would ask you to consider how much real world, production, code relies on request currently. It's far more than even NPM stats can tell you. I fully understand wanting to move on to a new thing and doing something in a new, more interesting way...this is the JavaScript ecosystem after all, have to chase the new thing. But please consider the amount of time and money you'd be costing professional engineering organizations by wholesale deprecating request. If you want to leave it in maintenance mode, that's fine, but understand that plenty of people have absolutely no practical reason to change libraries. Forcing people to change because of ideology is going to lead to frustration.

Regardless, thanks for the hard work everyone has put in to this library.

@kibertoad

This comment has been minimized.

Copy link

kibertoad commented Apr 5, 2019

I wonder what library could be considered modern and recommended now. Superagent is mostly in maintenance mode right now, axios not too active altogether.

@svozza

This comment has been minimized.

Copy link

svozza commented Apr 5, 2019

Just a quick note to say thank you (and all the other contributors) for all the hard work over the years on this module; it was one of the first I ever used back when I started with Node so will always have a special place in my heart.

@Vivalio

This comment has been minimized.

Copy link

Vivalio commented Apr 5, 2019

Let’s just bump the major version when we deprecate. That way most people depending on the project won’t see this error until they try to upgrade to a new major, which means they are actively developing it and really should look for an alternative.

i think this is still a viable solution for the mention above.

@millette

This comment has been minimized.

Copy link

millette commented Apr 5, 2019

@kibertoad Looks like @mikeal is working on https://github.com/mikeal/bent. I've been using https://github.com/sindresorhus/got for many years and it's well supported and evolving.

@riclf

This comment has been minimized.

Copy link

riclf commented Apr 5, 2019

With all this talk and the possibility of it being deprecated, I think there has to be equal mention of a current maturity replacement module, of parallel utility. We can't just announce its end and then suggest nothing, or a replacement of much less maturity and confidence. Request is used in serious applications. Why does this matter? Because for all its "outdated patterns at its core", it works on a daily basis, for thousands. This is not about the perfect world but the real world. What is the real world replacement, of confidence, on the day request is put in maintenance mode or is deprecated? That is an imperative.

@reconbot

This comment has been minimized.

Copy link
Contributor

reconbot commented Apr 5, 2019

You can find that discussion over here #3143

You can find a current working plan (which direct feedback is welcome) can be found here #3142 (comment)

@aalimovs

This comment has been minimized.

Copy link

aalimovs commented Apr 8, 2019

Thanks for your work on request!

The patterns at the core of request are out of date.

Patterns change every few months and years, especially in JavaScript community. Aren't the reasons why request was originally created are still valid today?

request has 10 years of commits, stability and tests. Why start from scratch? Isn't this just adding more "JavaScript fatigue", resulting in more libraries doing the same thing - HTTP requests?

It's sad to see such an important and historic library in Node's history go away because streams and callbacks are not fancy in 2019 anymore.

@stcktrce

This comment has been minimized.

Copy link

stcktrce commented Apr 8, 2019

I don't believe that deprecating the library is really needed, it's been around for about 10 years now, used in lots of places and is actually pretty stable, and in the end. all it does is make HTTP requests, what else would the library need? Support for the JS fad of the month? 👎

@cboden

This comment has been minimized.

Copy link

cboden commented Apr 8, 2019

The committers that are still active will try to merge fixes in a timely fashion, no promises though.

ba-dum-chhh! 🥁

@cphoover

This comment has been minimized.

Copy link

cphoover commented Apr 8, 2019

This is responsible deprecation. Well communicated, with a plan to follow through on. I think other OSS maintainers can look to this as a standard to aim for.

This is much better than forgetting about a package and letting random people (who can inject back-doors into the code) in as maintainers to take over when you no longer care.

Request was a great package, and we thank you tremendously for your contributions to the early node ecosystem. You are right in your assessment that callback style is no longer idiomatic JavaScript, and there are other packages like fetch which mirror WHATWG standards.

@DiegoRBaquero

This comment has been minimized.

Copy link

DiegoRBaquero commented Apr 8, 2019

@stcktrce Exactly, the library doesn't need anything else, it works just as it is. But there has been major improvements in the whole ecosystem. Deprecating the library is just marking the opportunity for others to check new and more modern libraries instead of simply relying on the most popular out there.

@mikeal thank you for all your efforts in the library (r2 too) and the ecosystem. Also, for setting this precedence of a well thought through and planned deprecation in the ecosystem.

@jasonswearingen

This comment has been minimized.

Copy link

jasonswearingen commented Apr 8, 2019

Let’s just bump the major version when we deprecate. That way most people depending on the project won’t see this error until they try to upgrade to a new major, which means they are actively developing it and really should look for an alternative.

@mikeal I don't think that's a good idea.

The problem is that most of replacements are of lower quality than request. I just moved to request from axios about a week ago.

Axios has multi-year persistent bugs around proxy support, modifying https agents, and unhandled promise exceptions. You only find these out after investing into axios heavily.

To new users axios looks superficially as good as request (similar number of users, promises by design, etc)

@jbunton-atlassian

This comment has been minimized.

Copy link

jbunton-atlassian commented Apr 9, 2019

Thanks for request :)

If anybody is looking for a minimal promise-based HTTP library with pluggable filters and good support for streams you could check out httplease. We've been using it for a few years in production.

@robberfree

This comment has been minimized.

Copy link

robberfree commented Apr 11, 2019

I love the request module.Thanks a lot.
You mean request get too much focus to prevent other same new module come out?

@reconbot

This comment has been minimized.

Copy link
Contributor

reconbot commented Apr 11, 2019

If there are specific bugs in comparable features in other libraries I’d like to specifically identify them. Proxy support is a complex feature and having a test case that request passes but other libraries fail is very valuable.

@jasonswearingen

This comment has been minimized.

Copy link

jasonswearingen commented Apr 11, 2019

@reconbot in the latest axios (^0.18.0) you can't connect to a https site through a proxy server. doing so results in EPROTO errors. this is an open bug regarding this, but the issue goes back years: axios/axios#1981

edit: specifically, you can't use axios to do https requests via a http proxy. maybe a dedicated https proxy works, didn't try that.

@mivanovaxway

This comment has been minimized.

Copy link

mivanovaxway commented Apr 11, 2019

I sure hope fixes are not considered new features, such as my pull request for Maximum Response Size, which I see as a standard required feature of any mature library.

Also I did review other request libs before I choose this one and most of them are very problematic, incomplete and buggy. Their docs do not measure either. I do not really see what can another library bring but untested code and bugs, it's not like there's a new approach to making HTTP requests. It's all about wrapping http/https module and providing sane defaults such as buffering response, decoding responses, and of course the ability to promisify the whole thing. The biggest problem of this library here is the aim of total compatibility, trying to be compatible with legacy stuff only brings pain and legacy coding practices. But this can be fixed in many ways. There's a good base that can be refactored into something elegant, modern and minimalist. And most of all reliable. There are many ways to do this - split into more files, use ECMA6 with Babel or Typescript.

No sane developer wants 10 libraries that do the same thing but lack different features, are buggy, undocumented. This library really works and I am thankful for it and hope that it's not deprecated but instead revived.

@mikeal

This comment has been minimized.

Copy link
Member Author

mikeal commented Apr 11, 2019

@rogerkmp2

This comment has been minimized.

Copy link

rogerkmp2 commented Apr 22, 2019

TIL 41k packages just became vulnerable.

Look, I agree that request should go away, but I’m always fearful of mainstream packages like this changing their release pipeline. One bad actor or one compromised dev box publishing malicious code would effectively spread to every project out there.

Please consider tightening the npm push requirements. Set up a branch for ci, require multiple approvals, something more than simply pushing to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.