Skip to content

CompileInc/hodor

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
September 7, 2016 16:02
September 6, 2016 16:54
September 7, 2016 16:02
November 6, 2023 19:26

Hodor PyPI

A simple html scraper with xpath or css.

Install

pip install hodorlive

Usage

As python package

WARNING: This package by default doesn't verify ssl connections. Please check the arguments to enable them.

Sample code

from hodor import Hodor
from dateutil.parser import parse


def date_convert(data):
    return parse(data)

url = 'http://www.nasdaq.com/markets/stocks/symbol-change-history.aspx'

CONFIG = {
    'old_symbol': {
        'css': '#SymbolChangeList_table tr td:nth-child(1)',
        'many': True
    },
    'new_symbol': {
        'css': '#SymbolChangeList_table tr td:nth-child(2)',
        'many': True
    },
    'effective_date': {
        'css': '#SymbolChangeList_table tr td:nth-child(3)',
        'many': True,
        'transform': date_convert
    },
    '_groups': {
        'data': '__all__',
        'ticker_changes': ['old_symbol', 'new_symbol']
    },
    '_paginate_by': {
        'xpath': '//*[@id="two_column_main_content_lb_NextPage"]/@href',
        'many': False
    }
}

h = Hodor(url=url, config=CONFIG, pagination_max_limit=5)

h.data

Sample output

{'data': [{'effective_date': datetime.datetime(2016, 11, 1, 0, 0),
           'new_symbol': 'ARNC',
           'old_symbol': 'AA'},
          {'effective_date': datetime.datetime(2016, 11, 1, 0, 0),
           'new_symbol': 'ARNC$',
           'old_symbol': 'AA$'},
          {'effective_date': datetime.datetime(2016, 8, 16, 0, 0),
           'new_symbol': 'MALN8',
           'old_symbol': 'AHUSDN2018'},
          {'effective_date': datetime.datetime(2016, 8, 16, 0, 0),
           'new_symbol': 'MALN9',
           'old_symbol': 'AHUSDN2019'},
          {'effective_date': datetime.datetime(2016, 8, 16, 0, 0),
           'new_symbol': 'MALQ6',
           'old_symbol': 'AHUSDQ2016'},
          {'effective_date': datetime.datetime(2016, 8, 16, 0, 0),
           'new_symbol': 'MALQ7',
           'old_symbol': 'AHUSDQ2017'},
          {'effective_date': datetime.datetime(2016, 8, 16, 0, 0),
           'new_symbol': 'MALQ8',
           'old_symbol': 'AHUSDQ2018'}]}

Arguments

  • ua (User-Agent)
  • proxies (check requesocks)
  • auth
  • crawl_delay (crawl delay in seconds across pagination - default: 3 seconds)
  • pagination_max_limit (max number of pages to crawl - default: 100)
  • ssl_verify (default: False)
  • robots (if set respects robots.txt - default: True)
  • reppy_capacity (robots cache LRU capacity - default: 100)
  • trim_values (if set trims output for leading and trailing whitespace - default: True)

Config parameters:

  • By default any key in the config is a rule to parse.
    • Each rule can be either a xpath or a css
    • Each rule can extract many values by default unless explicity set to False
    • Each rule can allow to transform the result with a function if provided
  • Extra parameters include grouping (_groups) and pagination (_paginate_by) which is also of the rule format.