Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
6 changed files
with
1,410 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
language: node_js | ||
node_js: | ||
- "4" | ||
|
||
node_js: | ||
- stable | ||
|
||
install: | ||
- npm install | ||
|
||
script: | ||
- npm run cover | ||
|
||
# Send coverage data to Coveralls | ||
after_script: "cat ./coverage/lcov.info | ./node_modules/coveralls/bin/coveralls.js" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,35 @@ | ||
# googlebot-verify | ||
Verify that a request is from Google using Google's DNS verification steps | ||
[![Build Status](https://travis-ci.org/jcowley/googlebot-verify.svg?branch=master)](https://travis-ci.org/jcowley/googlebot-verify) | ||
[![Coverage Status](https://coveralls.io/repos/github/jcowley/googlebot-verify/badge.svg?branch=master)](https://coveralls.io/github/jcowley/googlebot-verify?branch=master) | ||
## Synopsis | ||
|
||
Verify that an IP address is from Google using Google's recommended DNS verification steps. | ||
|
||
## Motivation | ||
|
||
You may wish to verify that a web crawler accessing your server is Googlebot (or another Google user-agent) and not spammers or other bots scraping your site while claiming to be Googlebot. Since you cannot rely on the `User-Agent` header which is easily spoofed, you need to use DNS look up to verify that the IP address belongs to Google. | ||
|
||
This library implements Google's own verification steps outlined here: https://support.google.com/webmasters/answer/80553?hl=en | ||
|
||
## Code Example | ||
|
||
Usage in an express application: | ||
|
||
```javascript | ||
const isGoogle = require('googlebot-verify'); | ||
const ipAddress = req.headers['x-forwarded-for'] || req.connection.remoteAddress; | ||
if (isGoogle(ipAddress)) { | ||
... | ||
``` | ||
## Installation | ||
`npm install googlebot-verify` | ||
## Tests | ||
`npm test` | ||
## License | ||
[MIT](https://github.com/pillarjs/parseurl/blob/master/LICENSE) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
'use strict'; | ||
const dns = require('dns'); | ||
const url = require('url'); | ||
|
||
module.exports = (ip, callback) => { | ||
callback = callback || function () {} | ||
|
||
return new Promise(function (resolve, reject) { | ||
|
||
dns.reverse(ip, (error, hosts) => { | ||
if (error) { | ||
reject(error); | ||
return callback(error); | ||
} | ||
|
||
const tld = hosts[0] && hosts[0].split('.').slice(-2, -1)[0]; | ||
if (tld !== "google" && tld !== "googlebot") { | ||
resolve(false); | ||
return callback(null, false); | ||
} | ||
|
||
dns.lookup(hosts[0], (error, address) => { | ||
const match = address === ip; | ||
resolve(match); | ||
return callback(error, match); | ||
}); | ||
}); | ||
}); | ||
} |
Oops, something went wrong.