Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Support for HTTP CONNECT proxy #108

Closed
OferE opened this Issue May 1, 2013 · 19 comments

Comments

Projects
None yet
7 participants

OferE commented May 1, 2013

Hi,
When I use a proxy server to access EC2 I get "Signature" problem error.
When I use the same code without the proxy it works fine.
I also checked other functionality of EC2 and the behaviour is the same.
Thanks.

Here is my code (written as a mocha test) - with the proxy settings:

var AWS = require('aws-sdk');
AWS.config.update({
   region: 'us-east-1',
   'httpOptions': {
      proxy: 'http://myProxy.com:8080' // I changed the proxy details for the github issue
   },
   sslEnabled: true
});

describe('checking amazon API', function() {

   it('should launch instances on amazon', function(done) {
      this.timeout(120000);
       var params = {
          ImageId: 'ami-1a249873', // changed the ami for the github issue
          MinCount: 1,
          MaxCount: 1,
          InstanceType: 't1.micro'
       },
       ec2 = new AWS.EC2({maxRetries: 15});

      ec2.runInstances(params, function(error, result) {
         console.log(this.request.httpRequest);
         console.log('result: ', result);
         console.log('error: ', error);
         done();
      });
   });
});

Here is the output I get from the console.log:

{ method: 'POST',
  path: '/',
  headers: 
   { 'User-Agent': 'aws-sdk-nodejs/v0.9.8-pre.9 linux/v0.8.18',
     'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8',
     'Content-Length': 283 },
  body: 'AWSAccessKeyId=***&Action=RunInstances&ImageId=ami-00fe0569&InstanceType=t1.micro&MaxCount=1&MinCount=1&Signature=***&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2013-05-01T07%3A33%3A00.120Z&Version=2013-02-01',
  endpoint: 
   { protocol: 'https:',
     slashes: true,
     host: 'ec2.us-east-1.amazonaws.com',
     hostname: 'ec2.us-east-1.amazonaws.com',
     href: 'https://ec2.us-east-1.amazonaws.com/',
     pathname: '/',
     path: '/',
     port: 443,
     constructor: { [Function: Endpoint] __super__: [Function: Object] } },
  region: 'us-east-1',
  params: 
   { Version: '2013-02-01',
     Action: 'RunInstances',
     ImageId: 'ami-00fe0569',
     MinCount: '1',
     MaxCount: '1',
     InstanceType: 't1.micro',
     Timestamp: '2013-05-01T07:33:00.120Z',
     SignatureVersion: '2',
     SignatureMethod: 'HmacSHA256',
     AWSAccessKeyId: '***',
     Signature: '***' } }
result:  null
error:  { [SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.]
  message: 'The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.',
  code: 'SignatureDoesNotMatch',
  name: 'SignatureDoesNotMatch',
  statusCode: 403,
  retryable: false }
Contributor

lsegal commented May 1, 2013

If your proxy being served over HTTP and not HTTPS, this might cause issues with signing. A quick way to verify would be to use sslEnabled: false or use your proxy over HTTPS.

Can you confirm whether this is the issue?

OferE commented May 1, 2013

Hi,
Thanks for your quick reply!
I tried to use sslEnabled: false - the issue remains.
What do you mean by "use your proxy over HTTPS"?
(The proxy I use supports https tunnelling - if that is what you mean).

OferE commented May 1, 2013

Hi,
The proxy I use supports both http and https -
the proxy address's schema is http for both ssl and regular connection.

As you can see in my original code - I stated that I want to use ssl using the flag:
sslEnabled: true
and you can see in the print of: this.request.httpRequest
that it tries to access aws using port 443 - which is ssl.
this is why I think it is a bug.

OferE commented May 6, 2013

I found some workaround for this bug:
The AWS.config.httpOptions.proxy should not be used - it is causing bad signature issues when working with remote proxy.
Instead, one can set the environment variable https_proxy and work with sslEnabled=true.

Contributor

lsegal commented May 6, 2013

@OferE what proxy software are you using?

OferE commented May 6, 2013

I work for an enterprise company (HP SW), the proxy is being configured by others.
I only know how to configure clients to work with it...
I tried few proxies - they all give the same results.

Contributor

lsegal commented May 6, 2013

I did some more digging into this and I am unable to reproduce this using tinyproxy with sslEnabled: false. Note that our implementation does not (yet) support CONNECT tunneling, so if the proxy server requires the use of CONNECT, the SDK will not be able to handle that (again: yet). There are other workarounds for CONNECT proxies, in the meantime, like using a node-tunnel agent, which can be done by constructing a tunnelingAgent object and specifying:

AWS.config.httpOptions = { agent: tunnelingAgent };

On another note, you mentioned you tried this in other proxies-- can you specify which so that I can try these too?

OferE commented May 6, 2013

all of our proxies use CONNECT tunneling AFAIK, so probably this explains all.
I will give the node-tunnel a try.
thanks!

This was referenced Jul 31, 2013

wabmca commented Jan 6, 2014

I think proxy setting is still not working.
I tried export https_proxy='http://USERID:Password@proxy1.wipro.com"
and also export proxy='http://USERID:Password@proxy1.wipro.com" //Using username password as my proxy authenticates.
So there is no solution as of now in AWS node.js to do this?

OferE commented Jan 6, 2014

I wrote my own AWS-EC2 REST implementation because of this...
(not too complicated)

bindugh commented Jan 6, 2014

Hi.. I too have the same proxy issue.. I have tried node_tunnel and httpOptions poxy set - both giving

{ [NetworkingError: read ECONNRESET]
message: 'read ECONNRESET',
code: 'NetworkingError',
errno: 'ECONNRESET',

How to resolve this ??

my code :

var AWS = require('aws-sdk');
var tunnel = require('tunnel');
var tunnelingAgent = tunnel.httpOverHttp({
proxy: { // Proxy settings
host: 'proxyhost',
port: 8080,
proxyAuth: "user:pass",

}
});

AWS.config.httpOptions = { agent: tunnelingAgent };
AWS.config.update({accessKeyId: 'key', secretAccessKey: 'secret', region: 'region'});
var ec2 = new AWS.EC2();
ec2.describeInstances(function(error, data){
if(error){
console.log("\n===================ERROR================= \n")
console.log(error);
} else {
console.log("\n===================DATA================= \n")
console.log(data);
}
});

Contributor

lsegal commented Jan 6, 2014

@bindugh the SDK uses SSL connections by default, which means you would have to use an HTTPS over HTTP tunnel. Have you tried tunnel.httpsOverHttp? You can also use new AWS.EC2({sslEnabled: false}) to disable SSL, but that is not recommended, in general.

Contributor

lsegal commented Jan 6, 2014

@wabmca I'm not sure what the https_proxy or proxy environment variables that you are exporting are. Can you explain what environment / libraries you are using? Have you tried using the node-tunnel library mentioned above?

@lsegal lsegal changed the title from Bug in ec2 access using proxy to Support for HTTP CONNECT proxy Aug 15, 2014

@nickdk nickdk referenced this issue Sep 10, 2014

Closed

Proxy #355

Contributor

lsegal commented Sep 30, 2014

I'm going to mark this as a feature request, since the SDK is behaving as is but does not (currently) support CONNECT proxies.

Unfortunately there are no current plans to add CONNECT proxy support to the SDK. I'm going to leave this feature request open for a bit in case anyone wants to pick this up with a pull request, but I think proxy support should be implemented by Node itself, not the SDK. It seems out of scope to implement the CONNECT proxy protocol and deal with all of the potential pitfalls of implementations that may not be to-spec. There's a great opportunity to build a generic HTTP proxy Node.js library here, and if there are extra things the SDK could do to tie into these HTTP proxies that would be something we could do, but I think this code would belong outside of the SDK itself.

FWIW, I agree the fundamental problem, and solution, lies within node itself. But that is of little consolation to those of us behind CONNECT proxies.

I was able to get the aws sdk to work through my company's CONNECT proxy by piecing together the code and suggestions above. With all due credit to the others in this thread, here is what I did in hopes it will get others unstuck:

$ npm install aws-sdk
$ npm install https-proxy-agent

Then in your code:

var aws = require('aws-sdk');
var HttpProxyAgent = require('https-proxy-agent');
var proxyAgent = new HttpProxyAgent(process.env.https_proxy || process.env.HTTPS_PROXY);

aws.config.httpOptions = { agent: proxyAgent };
// Now you can use aws.s3, or aws.ec2, or etc

This is valid if your company's proxy uses the CONNECT verb and is specified with an http:// protocol. That is, your proxy string looks like this "http://...." even for https urls, then, fingers-crossed, the above code will hopefully help you.

OferE commented Oct 15, 2014

I work for a huge enterprise company which allows internet connections through proxies only.

I couldn't wait for this to get fixed so I wrote the code myself.
I used request.js for http requests - and configured the proxies with no problem.
It took me few hours to make all the API that I need to work...
I just don't understand the attitude of "let's wait for node to fix things" - it is really embarrassing...

Member

AdityaManohar commented Nov 24, 2014

@OferE
I did research this problem some more. It turns out that Node.js isn't going to be implementing proxy support in the immediate future as per joyent/node#2474 (comment). This does seem a bit dated, but there has been no change in direction regarding proxy support since then.

I did also spend some time figuring out an approach to implement proxy support with the SDK. However this involves a re-write of the http.Agent class, which is a private API. Doing this to support proxies poses a maintainability problem, owing to the numerous different proxy types and authentication schemes.

There are a number of http.Agent implementations and request itself uses tunnel-agent to support HTTP tunneling.

You can use tunnel-agent with the SDK as well to support proxy behavior. Here is a code sample showing how that can be done:

var AWS = require('aws-sdk');

/* Pick an agent from:
 * httpOverHttp
 * httpOverHttps
 * httpsOverHttp
 * httpsOverHttps
 */ 
var Agent = require('tunnel-agent').httpsOverHttps;

var agent = new Agent({proxy: 'http://127.0.0.1'});

AWS.config.update({httpOptions: { agent: agent}});

var s3 = new AWS.S3();
s3.listBuckets();

I hope this helps, and do let me know if there is anything else I can help with!

Member

AdityaManohar commented May 4, 2015

The "Configuring a Proxy" section of the developer guide, as well as the "Using the AWS SDK for JavaScript from Behind a Proxy" blog post document using the SDK from behind a network proxy.

I'm marking this as closed. Feel free to open another issue if you have any questions.

monken commented Feb 19, 2017

I would also like to mention https://www.npmjs.com/package/aws-sdk-proxy which makes this less painful and honors the HTTPS_PROXY environmental variable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment