Permalink
Commits on Mar 10, 2018
  1. fix src.ca redirection test

    mna committed Mar 10, 2018
  2. add new Go versions to travis

    mna committed Mar 10, 2018
Commits on Dec 22, 2017
  1. Merge pull request #63 from smlrepo/master

    mna committed Dec 22, 2017
    Use robotstxt rather than robotstxt.go
Commits on Dec 21, 2017
  1. Fixing the failing test case.

    h8liu committed Dec 21, 2017
Commits on Mar 24, 2017
  1. add go 1.8, use latest patches

    mna committed Mar 24, 2017
Commits on Nov 20, 2016
  1. doc

    mna committed Nov 20, 2016
  2. added comma

    oherych committed Nov 20, 2016
Commits on May 24, 2016
  1. doc: update changelog

    mna committed May 24, 2016
Commits on May 23, 2016
  1. make cloneForRedirect private

    mna committed May 23, 2016
Commits on May 19, 2016
Commits on May 4, 2016
Commits on Feb 24, 2016
  1. doc formatting

    mna committed Feb 24, 2016
  2. add travis badge

    mna committed Feb 24, 2016
  3. add travis

    mna committed Feb 24, 2016
  4. lint, documentation, use RobotUserAgent only for robots.txt policy ma…

    mna committed Feb 24, 2016
    …tching, use UserAgent for all requests
Commits on Nov 7, 2015
  1. fix redirect tests

    mna committed Nov 7, 2015
  2. Merge branch 'master' of https://github.com/statusmachine/gocrawl int…

    mna committed Nov 7, 2015
    …o statusmachine-master
Commits on Nov 6, 2015
Commits on Oct 22, 2015
  1. More robust handling of base[href] cases

    juliend2 committed Oct 22, 2015
    1. Allow for non-/-ending base[href], like the spec supports
    2. Handle "Relative Path References" and "Relative Reference With Absolute
    Paths" href values for anchors
Commits on Oct 21, 2015
  1. Merge pull request #48 from nanolab/master

    mna committed Oct 21, 2015
    Update worker.go
  2. Update worker.go

    nanolab committed Oct 21, 2015
    fetchDuration fix
Commits on Oct 17, 2015
  1. Remove test log

    juliend2 committed Oct 17, 2015
  2. One liner

    juliend2 committed Oct 17, 2015