Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Hello Traveler

The tour begins with,

Public Restful API

Search for businesses with free wifi.

Host: ryanberg.io Path: /wifi/search

Query String:

Required Arguments: latitude=inDegrees&longitude=inDegrees

Options Parameter: radius=inMiles [default 1 mile, max=10(any amount above returns results with 10)]

Example Query: http://ryanberg.io/wifi/search?latitude=34.261555&longitude=-118.301857&radius=2

Optionally set your "Content-Encoding" header to "gzip" for compressed responses. (For most modern browsers, this is already set.)

(•_•)

( •_•)>⌐■-■

(⌐■_■)

Example JSON Response:

{

id: String(),
name: String(),
phoneNumber: String(),//'(012) 456-7890'
category: Array(),
address:
{
    fullAddress: String(),
    street: String(),
    city: String(),
    state: String(),
    zip: String()
},
latitude: Number(),
longitude: Number(),
ratingCount: Number(),
ratingAverage: Number()

}

The Database

The mongodb database is housed on a Debian Linux VM.

Scraping

mainDirector.js gets a random city from the database at random intervals. It then sends the request info to a proxy in 1 of 3 geographical locations.

proxy.js is a node listener that creates a child process of phantomjs. The proxy then passes the data to the phantom process through standard output.

phantomRequest.js sends a phantomjs request with a random user agent set in the http header. It then executes the javascript on the responding html and sends the resulting html to the html interpreter.

htmlInterpreter.js breaks apart the html with regular expressions to build a javascript object. It then inserts the resulting json business in the businesses collection in the database.

Client

There is an example web app seen on my website sunnyscripts.com which parses the Rest API. It also adds some functionality with getting directions, checking wifi availability and looking up by location. Conveniently, the id of the business object matches yelp.com. Meaning, yelp.com/biz/id is the yelp page for that business.

There is also an iOS app now available that I worked on heavily. Check it out in the App Store.

About

Where's Wifi Scraper

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages