Permalink
Commits on May 31, 2018
  1. Merge pull request #88 from seomoz/cbf/drop-unused-imports

    b4hand committed May 31, 2018
    Drop unused imports.
  2. Drop unused imports.

    b4hand committed May 31, 2018
Commits on May 12, 2018
  1. Merge pull request #85 from seomoz/update-py2

    b4hand committed May 12, 2018
    Update local pyenv version to 2.7.15.
  2. Update local Python version to 2.7.15.

    b4hand committed May 12, 2018
Commits on Nov 20, 2017
  1. Merge pull request #82 from seomoz/crawl-delay-str

    b4hand committed Nov 20, 2017
    Add Crawl-Delay to string representation of Agent.
  2. Try upgrading Cython.

    b4hand committed Nov 20, 2017
  3. Fix __str__ on Python 3.

    b4hand committed Nov 20, 2017
  4. Add Crawl-Delay to string representation of Agent.

    b4hand committed Nov 17, 2017
    Pulls in changes from seomoz/rep-cpp#31
    
    Also, add some tests demonstrating functionality.
  5. Merge pull request #81 from seomoz/lower-log-level

    b4hand committed Nov 20, 2017
    Lower log level for TTL extraction to warn.
Commits on Nov 17, 2017
  1. Lower log level for TTL extraction to warn.

    b4hand committed Nov 17, 2017
    These aren't fatal errors and servers can give back all sorts of wacky
    strings. The default values should handle these cases fine, so some
    people may not want the noisy error in their logs. Lowering the
    severity makes that easier and the exception level is equivalent to
    ERROR which is not something that should be silenced.
    
    Fixes #76.
Commits on Nov 15, 2017
  1. Merge pull request #79 from seomoz/call-hook-on-error

    b4hand committed Nov 15, 2017
    Ensure after_response_hook is called on errors.
  2. Ensure after_response_hook is called on errors.

    b4hand committed Nov 15, 2017
Commits on Oct 23, 2017
  1. Merge pull request #78 from seomoz/cbf/unused-make-target

    b4hand committed Oct 23, 2017
    Drop unused make target.
  2. Drop unused make target.

    b4hand committed Oct 23, 2017
    This is no longer required since 3e0e86d.
  3. Drop tox.ini since it doesn't fully work.

    b4hand committed Oct 23, 2017
    The Travis build currently builds all versions of Python that we care
    about, so we don't need local support for tox.
    
    Fixes #42.
  4. Fix tox.ini to partially work.

    b4hand committed Oct 23, 2017
    This was my last attempt at getting tox to work correctly. It
    partially works, but fails in py33,py34,py35 despite those working in
    Travis.
Commits on Oct 9, 2017
  1. Add new py3 version to CI (#77)

    stevelle authored and b4hand committed Oct 9, 2017
    * Add new py3 version to CI
    
    I noticed when submitting a PR for another project to add support
    for respecting robots.txt files that reppy was not building under
    python 3.7-dev and nightly builds.
    
    To ensure proper visibility for coming versions, I am proposing
    additional versions for the testing matrix.
    
    * Allow failure in Python 3.7-dev
    
    Addresses instability in an emerging Python version while still
    providing insight.
    
    * Correcting allowed failures syntax for travis ci
Commits on Sep 11, 2017
  1. Merge pull request #75 from seomoz/release-v0.4.8

    b4hand committed Sep 11, 2017
    Release version 0.4.8.
Commits on Sep 8, 2017
  1. Release version 0.4.8.

    b4hand committed Sep 8, 2017
  2. Merge pull request #74 from seomoz/add-fetch-hooks

    b4hand committed Sep 8, 2017
    Add fetch hook support.
  3. Add fetch hook support.

    b4hand committed Sep 8, 2017
    This will allow Aardwolf to peek at the response and the full Robots
    object to collect additional information during crawls.
  4. Merge pull request #73 from seomoz/makefile-fixes

    b4hand committed Sep 8, 2017
    Fix circular dependency on reppy/robots.so
  5. Remove `reppy/*.so` during `make clean`.

    b4hand committed Sep 8, 2017
  6. Fix circular dependency on reppy/robots.so

    b4hand committed Sep 8, 2017
    Previously, when rebuilding with make, it would report the following
    warning:
    
    make: Circular reppy/robots.so <- reppy/robots.so dependency dropped.
    
    This is because reppy/robots.so itself matches the pattern reppy/%.*,
    so by restricting the dependency to reppy/%.py*, it won't have a
    self-dependency.
  7. Shorten test case names.

    b4hand committed Sep 8, 2017
  8. Merge pull request #72 from seomoz/ignore-external-directives

    b4hand committed Sep 8, 2017
    Ignore Allow/Disallow directives for external URLs.
  9. Ignore Allow/Disallow directives for external URLs.

    b4hand committed Sep 8, 2017
    Integrate changes from seomoz/rep-cpp#28 and bring in other most
    recent changes from rep-cpp.
Commits on Sep 6, 2017
  1. Fix typo in docstring.

    b4hand committed Sep 6, 2017
Commits on Aug 31, 2017
  1. Merge pull request #71 from seomoz/lindsey/add-str

    lindseyreno committed Aug 31, 2017
    Add str() method to Agent.
  2. Add str() method to Agent.

    lindseyreno committed Aug 31, 2017
Commits on Jun 20, 2017
  1. Merge pull request #69 from seomoz/switch-to-setuptools

    b4hand committed Jun 20, 2017
    Prefer setuptools to distutils.
  2. Prefer setuptools to distutils.

    b4hand committed Jun 20, 2017
  3. Merge pull request #68 from seomoz/cbf/release-0.4.7

    b4hand committed Jun 20, 2017
    Release 0.4.7.
Commits on Jun 19, 2017
  1. Release 0.4.7.

    b4hand committed Jun 19, 2017
    Also, update package metadata while I'm at it.
  2. Merge pull request #67 from seomoz/bump-rep-cpp

    b4hand committed Jun 19, 2017
    Add str() methods to C++ objects