Skip to content
A simple browser/client-side web scraper.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src
.gitignore
CNAME Create CNAME Apr 22, 2017
LICENSE.md Added MIT LICENSE.md. Apr 20, 2017
README.md Bumped up download version. Apr 22, 2017
index.css
index.html
index.js Added documentation for new methods. Apr 22, 2017
package.json
rollup.config.js First commit. Apr 20, 2017
tsconfig.json Updated build scripts. Apr 24, 2017
tslint.json Add resolveURLs, options, and tests. Apr 21, 2017
yarn.lock

README.md

Getsy

A simple browser/client-side web scraper. Try it out in a REPL: http://www.getgetsy.com

TODOS:

  • Support for websites with infinite scroll.
  • Support for websites with click pagination.

Installation options:

  • Run npm install --save getsy or yarn add getsy
  • Download the umd build and link it using a script tag

How to use:

This library exposes a single function: getsy(url: string, optionsObject?: options): Promise<Getsy>

parameters:

  • url: The url of the website you wish to scrape.

  • optionsObject(optional):

    • corsProxy(optional string): The endpoint of the corsProxy you wish to use. (Read corsProxy for more info)

    • resolveURLs(optional boolean): Wether you want getsy to resolve all relative urls in the resource to absolute urls so they don't fail when they load in another page. (defaults to true)

    • iframe: A boolean or object with width and height properties indicating if getsy should start in iframeMode or not. iframe mode will wait for the resource to be mounted in a hidden iframe so you can extract more data through pagination or infinite scrolling. (defaults to false)

The function returns a promise that resolves to a Getsy object on success and rejects if it was unable to load the requested page.

Getsy objects have a method getMe for scraping the resource's contents. This method is just a wrapper over the jQuery function so you can chain other jQuery methods on it. If you need to use the raw data you can access it's content property. (More on Getsy below)

Example (Promises):

import getsy from 'getsy'

getsy('https://en.wikipedia.org/wiki/"Hello,_World!"_program').then(myGetsy => {
  console.log(myGetsy.getMe('#firstHeading').text())
})

Example (Async/Await):

import getsy from 'getsy'

async function testing() {
  const myGetsy = await getsy('https://en.wikipedia.org/wiki/"Hello,_World!"_program')

  console.log(myGetsy.getMe('#firstHeading').text())
}

testing()

Here's how you might use it with a website that has infinite scrolling:

async function infiniteScrape() {
  myGetsy = await getsy('http://scrollmagic.io/examples/advanced/infinite_scrolling.html', { iframe: true })
  
  console.log(`${myGetsy.getMe('.box1').length} boxes.`)
  
  const { succesfulTimes, totalRetries } = await myGetsy.scroll(10)
  
  console.log(`New content loaded ${succesfulTimes} times with ${totalRetries} total retries.`)
  console.log(`${myGetsy.getMe('.box1').length} boxes.`) // More content!
}

infiniteScrape()

The Getsy Object:

The Getsy object has the following properties and methods:

  • corsProxy: The same one passed from the options object or the default value.

  • content: The original string data received from the request.

  • iframe: A reference to its iframe element if in iframe mode.

  • iframeDoc: A reference to its iframe's document if in iframe mode.

  • content: The original string data received from the request.

  • getMe(selector: string): JQuery: Query the resource's DOM or the iframe if in iframe mode with a jQuery selector. Returns a JQuery object.

  • scroll(numberOfTimes: number, element?: HTMLElement, interval?: number, retries?: number): Promise<scrollResolve>: Scroll to the bottom of an element (defaults to body) to load new data a specified numberOfTimes. The interval (defaults to 2000) is the time in milliseconds that Getsy waits before checking if new content has loaded. If no new content has loaded it will retry as many times as specified by retries (defaults to 5). If no new content has loaded and scroll is out of retries then it will resolve the Promise early to avoid waiting for the remaining numberOfTimes. Note: retries reset to 0 on every succesful content load. Returns a Promise that resolves to an object with the number of .succesfulTimes that new content was loaded and the .totalRetries.

  • hideFrame(): void: Hides the iframe if applicable.

  • showFrame(): void: Shows the iframe if applicable.


CorsProxy:

This library uses a corsProxy to get by the CORS Origin issue. If you don't provide one it will default to: https://crossorigin.me/.

Some node CorsProxy servers:

You can’t perform that action at this time.