[HTTP package] Pass unrecognized options through to request module

[dandv]

The HTTP package calls request, which in turn calls Node's http or https modules. The latter especially can take rather esoteric options for TSL configuration.

It would be great if HTTP.call() would pass through any options it doesn't recognize. This solves a use case I ran into, where I had to set {rejectUnauthorized: false}.

[glasser]

The goal of the HTTP package is to provide a consistent interface between client and server, so adding lots of server-only options doesn't seem look a good idea. Maybe a specific option that's called "extraNodeRequestOptions" and documented as "We might change the backend to not be request and this option will break at that point."

That said, if you want to do complex HTTP things that the Meteor http package doesn't implement, there's the workaround of just using Npm.require('http') or Npm.require('request') (the latter with Npm.depends in a package, though it may coincidentally work for now without that since request is in the dev bundle) yourself.

[dandv]

I and others who've requested proxy support on SO would be happy to have an extraNodeRequestOptions option.

As for a consistent interface between client and server, the problem is similar to Minimongo vs. MongoDB. The latter has more options than the former, which is fantastic. Right now the client can't perform CORS requests for instance, while that's not a problem for the server.

The client and server are just different.

Why not empower the server to use proxies? Presumably the feature would be kept in the event that meteor would switch away from the request backend.

[raix]

@dandv agree #1814
If the http should be consistent between client and server well shouldn't followredirects go then?

[dandv]

For those who don't want to reinvent the wheel and use the low-level Npm https module, I've created a fork of Meteor's HTTP package called http-more.

[guaka]

I'd like to have the possibility to use a proxy, especially if it's somehow possible to let the client side revert to proxy chaining, (http://proxy.example.com/http://server.example.com, see also http://stackoverflow.com/questions/6988441/can-client-javascript-use-its-own-http-proxy)

[LeCoupa]

The package http-more is now available here. (Dan's url seems to redirect to a previous version of the package)

[ghost]

Surely there must be a way to specify an outbound proxy server for meteor to use ? Meteor is useless in a corporate environment without that. Client side is handled automatically by the browser config so is transparent, but server has to explicitly configured somewhere. Industry standard http_proxy environment variable would be nice.

[mitar]

I cannot believe that after more than a year after my #1664, this is still not fixed. Really crazy. It is really so hard to add some underscore prefixed options while we are waiting for "perfect" solution?

[dandv]

@LeCoupa: http-more is available at https://github.com/dandv/meteor-http-more, can you please update the link in your comment or delete it (I've fixed mine above)? Thanks.

I agree with @mitar and I'll be happy to retire my package in favor of Meteor's HTTP package simply passing along options it doesn't specifically recognize.

[mitar]

In fact I need https://github.com/aldeed/meteor-http-extras. :-) I need to specify binary options.

[aldeed]

@mitar, I am biased but I have to agree. I put effort into making sure my PR (which is now the aldeed:http add-on package) fixed the glaring problems with the current package while keeping the client and server API the same, other than a few tiny differences that are unavoidable.

I think the issue is probably that MDG just doesn't see any need for an official http package anymore, so it won't be high priority. Our best bet is probably to combine all the features of http plus aldeed:http plus http-more into a new community-supported package that is a complete replacement for http. It's really not even a huge undertaking, but I don't have time in the foreseeable future.

[niallobrien]

The real world shows us that client apps are appearing on devices that were never considered when Meteor was first created, eg. watches, microwaves, cars, refrigerators etc. We cannot be expected to use Meteor across all of these. More server-side options to get at our data is greatly needed.
Furthermore, Meteor was clearly never designed to scale in a cost-efficient manner. Having each DDP client eat RAM on the server is silly. There are plenty of use cases for alternative methods of getting data in and out of our Meteor apps, without always having to buy into the entire ecosystem.

[Anonyfox]

As someone who has to deal with mostly non-english content, it outright sucks that i just can't set {encoding: null} to get a buffer instead of an utf-8 string. Really. I have to deal with crappy (and wrong encoded) websites using iconv-lite and just to set {encoding: null} I have to jump through some hoops with third party packages.

[glasser]

Feedback has been heard. We should make it much easier to use request directly via the HTTP module, with appropriate warnings that the API could change with updates. I hope to do this for 1.0.4 if time permits.

[dandv]

@glasser, thanks for reconsidering this feature request and implementing a fix.

For others who check this issue - you can now pass npmRequestOptions to HTTP.call.

[digz6666]

Any example using npmRequestOptions?

[digz6666]

Nevermind, I found it:

HTTP.post(url, {
    npmRequestOptions: {
        rejectUnauthorized: false // TODO remove when deploy
    },
    timeout: 30000, // 30s
    data: xml
}, function(error, result) {
    console.log('error: ' + error);
    console.log('resultXml: ' + result);
});