Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 

README.md

py-parsehub

Python3 ParseHub module


What is ParseHub?

ParseHub is a service providing automated webscraping. You design a template using their Mozilla plugin and then access it through a REST API. Results can be retrieved in both JSON and CSV (coming soon) format. The templates are pretty flexible and use machine-learning to grasp complex hierarchies.

You can extract data from anywhere. ParseHub works with single-page apps, multi-page apps and just about any other modern web technology. ParseHub can handle Javascript, AJAX, cookies, sessions and redirects. You can easily fill in forms, loop through dropdowns, login to websites, click on interactive maps and even deal with infinite scrolling.

see ParseHub homepage


###Prerequisities

  • python3 (3.2.3 tested)
  • urllib3 ($ pip3 install urllib3)

Usage

Initialize

>>> from ph2 import ParseHub  
>>> ph = ParseHub('<redacted API-key>')  

Get all projects

>>> print(ph.projects)
[<PhProject 'Project1' token 'tCWOS3cB-ZM8qtShXw6j8tyOHZ84hLik'>, <PhProject 'Project2' token 'tfs9Gv10cixnCtrk0iz0-u62r7lSdNt8'>]

Run a given project

>>> p1 = ph.projects[0]
>>> r1 = p1.run()

Is data available for download?

>>> r1.check_available()
1

Get data if available

>>> r1.get_data()
[{'link': 'http://www.123greetings.com/', 'title': '123Greetings'}, {'link': 'https://webmail.123-reg.co.uk/', 'title': 'Welcome to 123-reg Webmail | Webmail log in | 123-reg'}.....]

A blocking request for data

>>> r2 = p1.run()
>>> r2.get_data_sync()
[{'link': 'http://www.123greetings.com/', 'title': '123Greetings'}, {'link': 'https://webmail.123-reg.co.uk/', 'title': 'Welcome to 123-reg Webmail | Webmail log in | 123-reg'}.....]

Cancel a running job

>>> r3 = p1.run()
>>> r3.cancel()

Or delete it alltogether

>>> r3.delete()

Get array of runs of a project

>>> p1.get_runs()
[<PhRun object token:tbcBSs9i7WHWtx3nqXW7vwp9>, ....]

You can specify offset to leave out the last x runs

>>> p1.get_runs(5)
[<PhRun object token:tbcBSs9i7WHWtx3nqXW7vwp9>, ....]

Projects hold reference to their last completed run...

>>> p1.last_ready_run
<PhRun object token:tCNPbuLm7wd-Aqmb9WHHZMV0>

...and the last run no matter what its status is

>>> p1.last_run.status
'running'

Runs can be compared based on their md5sum to detect changes between runs

>>> p1.get_runs()[0] == p1.get_runs()[1]
True

Both runs and projects can have their attributes easily printed printed for debugging

>>> p1.last_run.pprint()
data : [...]
data_ready : 1
end_time : 2015-04-13T15:30:10
md5sum : 51b246040a0ee389dd5eb6bb46e1b06b
pages : 1
ph : <ParseHub object API key:'<redacted>'>
project_token : tCWOS3cB-ZM8qtShXw6j8tyOHZ84hLik
run_token : tCNPbuLm7wd-Aqmb9WHHZMV0
start_time : 2015-04-13T15:30:03
start_url : https://www.google.com/search?q=...
start_value : {}
status : complete

###Todo

  • package
  • refactor
  • SSL
  • CSV

About

Python API for parsehub.com web scraping service

Resources

License

Releases

No releases published

Packages

No packages published

Languages

You can’t perform that action at this time.