Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Fetching contributors…

Cannot retrieve contributors at this time

350 lines (280 sloc) 12.37 kb

urllib

NPM version build status appveyor build status Test coverage Gittip David deps io.js version node.js version npm download

logo

Request HTTP URLs in a complex world — basic and digest authentication, redirections, cookies, timeout and more.

Install

$ npm install urllib --save

Usage

callback

var urllib = require('urllib');

urllib.request('http://cnodejs.org/', function (err, data, res) {
  if (err) {
    throw err; // you need to handle error
  }
  console.log(res.statusCode);
  console.log(res.headers);
  // data is Buffer instance
  console.log(data.toString());
});

Promise

If you've installed bluebird, bluebird will be used. urllib does not install bluebird for you.

Otherwise, if you're using a node that has native v8 Promises (v0.11.13+), then that will be used.

Otherwise, this library will crash the process and exit, so you might as well install bluebird as a dependency!

var urllib = require('urllib');

urllib.request('http://nodejs.org').then(function (result) {
  // result: {data: buffer, res: response object}
  console.log('status: %s, body size: %d, headers: %j', result.res.statusCode, result.data.length, result.res.headers);
}).catch(function (err) {
  console.error(err);
});

co & generator

If you are using co or koa:

var co = require('co');
var urllib = require('urllib');

co(function* () {
  var result = yield urllib.requestThunk('http://nodejs.org');
  console.log('status: %s, body size: %d, headers: %j',
    result.status, result.data.length, result.headers);
})();

Global response event

You should create a urllib instance first.

var httpclient = require('urllib').create();

httpclient.on('response', function (info) {
  error: err,
  ctx: args.ctx,
  req: {
    url: url,
    options: options,
    size: requestSize,
  },
  res: res
});

httpclient.request('http://nodejs.org', function (err, body) {
  console.log('body size: %d', body.length);
});

API Doc

Method: http.request(url[, options][, callback])

Arguments

  • url String | Object - The URL to request, either a String or a Object that return by url.parse.
  • options Object - Optional
    • method String - Request method, defaults to GET. Could be GET, POST, DELETE or PUT. Alias 'type'.
    • data Object - Data to be sent. Will be stringify automatically.
    • content String | Buffer - Manually set the content of payload. If set, data will be ignored.
    • stream stream.Readable - Stream to be pipe to the remote. If set, data and content will be ignored.
    • writeStream stream.Writable - A writable stream to be piped by the response stream. Responding data will be write to this stream and callback will be called with data set null after finished writing.
    • contentType String - Type of request data. Could be json. If it's json, will auto set Content-Type: application/json header.
    • dataType String - Type of response data. Could be text or json. If it's text, the callbacked data would be a String. If it's json, the data of callback would be a parsed JSON Object. Default callbacked data would be a Buffer.
    • headers Object - Request headers.
    • timeout Number - Request timeout in milliseconds. Defaults to exports.TIMEOUT. Include remote server connecting timeout and response timeout. When timeout happen, will return ConnectionTimeout or ResponseTimeout.
    • auth String - username:password used in HTTP Basic Authorization.
    • digestAuth String - username:password used in HTTP Digest Authorization.
    • agent http.Agent - HTTP Agent object. Set false if you does not use agent.
    • httpsAgent https.Agent - HTTPS Agent object. Set false if you does not use agent.
    • ca String | Buffer | Array - An array of strings or Buffers of trusted certificates. If this is omitted several well known "root" CAs will be used, like VeriSign. These are used to authorize connections. Notes: This is necessary only if the server uses the self-signed certificate
    • rejectUnauthorized Boolean - If true, the server certificate is verified against the list of supplied CAs. An 'error' event is emitted if verification fails. Default: true.
    • pfx String | Buffer - A string or Buffer containing the private key, certificate and CA certs of the server in PFX or PKCS12 format.
    • key String | Buffer - A string or Buffer containing the private key of the client in PEM format. Notes: This is necessary only if using the client certificate authentication
    • cert String | Buffer - A string or Buffer containing the certificate key of the client in PEM format. Notes: This is necessary only if using the client certificate authentication
    • passphrase String - A string of passphrase for the private key or pfx.
    • followRedirect Boolean - follow HTTP 3xx responses as redirects. defaults to false.
    • maxRedirects Number - The maximum number of redirects to follow, defaults to 10.
    • beforeRequest Function - Before request hook, you can change every thing here.
    • customResponse Boolean - let you get the res object when request connected, default false.
    • gzip Boolean - Accept gzip response content and auto decode it, default is false.
  • callback(err, data, res) Function - Optional callback.
    • err Error - Would be null if no error accured.
    • data Buffer | Object - The data responsed. Would be a Buffer if dataType is set to text or an JSON parsed into Object if it's set to json.
    • res http.IncomingMessage - The response.

Returns

http.ClientRequest - The request.

Calling .abort() method of the request stream can cancel the request.

Options: options.data

When making a request:

urllib.request('http://example.com', {
  method: 'GET',
  data: {
    'a': 'hello',
    'b': 'world'
  }
});

For GET request, data will be stringify to query string, e.g. http://example.com/?a=hello&b=world.

For others like POST, PATCH or PUT request, in defaults, the data will be stringify into application/x-www-form-urlencoded format if Content-Type header is not set.

If Content-type is application/json, the data will be JSON.stringify to JSON data format.

Options: options.content

options.content is useful when you wish to construct the request body by yourself, for example making a Content-Type: application/json request.

Notes that if you want to send a JSON body, you should stringify it yourself:

urllib.request('http://example.com', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json'
  },
  content: JSON.stringify({
    a: 'hello',
    b: 'world'
  })
});

It would make a HTTP request like:

POST / HTTP/1.1
Host: example.com
Content-Type: application/json

{
  "a": "hello",
  "b": "world"
}

This exmaple can use options.data with application/json content type:

urllib.request('http://example.com', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json'
  },
  data: {
    a: 'hello',
    b: 'world'
  }
});

Options: options.stream

Uploads a file with formstream:

var urllib = require('urllib');
var formstream = require('formstream');

var form = formstream();
form.file('file', __filename);
form.field('hello', '你好urllib');

var req = urllib.request('http://my.server.com/upload', {
  method: 'POST',
  headers: form.headers(),
  stream: form
}, function (err, data, res) {
  // upload finished
});

Response Object

Response is normal object, it contains:

  • status or statusCode: response status code, -1 meaning some network error like ENOTFOUND
  • headers: response http headers, default is {}
  • size: response size
  • aborted: response was aborted or not
  • rt: total request and response time in ms.

Response: res.aborted

If the underlaying connection was terminated before response.end() was called, res.aborted should be true, and return RemoteSocketClosedError error.

var server = require('http').createServer(function (req, res) {
  req.resume();
  req.on('end', function () {
    res.write('foo haha\n');
    setTimeout(function () {
      res.write('foo haha 2');
      setTimeout(function () {
        res.socket.end();
      }, 300);
    }, 200);
    return;
  });
});

urllib.request('http://127.0.0.1:1984/socket.end', function (err, data, res) {
  err.name.should.equal('RemoteSocketClosedError');
  err.message.should.equal('Remote socket was terminated before `response.end()` was called');
  data.toString().should.equal('foo haha\nfoo haha 2');
  should.ok(res.aborted);
  done();
});

TODO

  • Support component
  • Browser env use Ajax
  • [√] Upload file like form upload
  • [√] Auto redirect handle
  • [√] https & self-signed certificate
  • [√] Connection timeout & Response timeout
  • [√] Support Accept-Encoding=gzip by options.gzip = true
  • [√] Support Digest access authentication

Authors

Below is the output from git-summary.

$ git summary

 project  : urllib
 repo age : 3 years, 9 months
 active   : 78 days
 commits  : 230
 files    : 24
 authors  :
   188  fengmk2                 81.7%
    10  XiNGRZ                  4.3%
     9  dead_horse              3.9%
     6  Jackson Tian            2.6%
     6  ibigbug                 2.6%
     3  haoxin                  1.3%
     2  alsotang                0.9%
     1  popomore                0.4%
     1  Jonathan Dahan          0.4%
     1  Yiyu He                 0.4%
     1  aleafs                  0.4%
     1  coderhaoxin             0.4%
     1  fishbar                 0.4%

License

MIT

Jump to Line
Something went wrong with that request. Please try again.