Verify that a request is from Twitter crawlers using Twitter's DNS verification steps
You may wish to verify that a web crawler accessing your server is Twitter and not spammers or other bots scraping your site while claiming to be Twitterbot. Since you cannot rely on the User-Agent
header which is easily spoofed, you need to use DNS look up to verify that the IP address belongs to Twitter. Twitter is usually scanning URLs to detect metadata.
npm install --save is-twitter
const isTwitter = require('is-twitter')
let ip = '199.59.150.182'
isTwitter(ip).then((outcome) => {
if (outcome) {
// it's twitter.
}
}).catch(console.error)
app.enable('trust proxy')
app.use((req, res, next) => {
let ip = req.ip || req.connection.remoteAddress
isTwitter(ip).then(outcome => {
if (outcome) {
res.status(404).text('Nothing to scan') // block twitter crawler
} else {
next() // it's a user
}
})
})
npm test
MIT
Rocco Musolino @roccomuso