Skip to content

@bopoda bopoda released this Jan 2, 2019 · 3 commits to master since this release

  • added composer.json file (needs for packagist.org)
  • move all code to src/RobotsTxtParser folder
  • move tests to tests/RobotsTxtParser folder
  • fix code style to PSR compatible format
  • improve RobotsTxtValidator (getRelativeUrl function)
Assets 2

@bopoda bopoda released this Dec 29, 2018 · 4 commits to master since this release

Fixed parsing error when the latest allow/disallow directive in the block is empty and the next directive is User-agent.

User-agent: *
Disallow:

User-agent: Linguee
Disallow: /api/showcase

URL "/api/showcase" should be assigned (as Disallow) only to User-agent Linguee, not any.

See PR #54 for more information.

Assets 2

@bopoda bopoda released this Apr 10, 2017 · 17 commits to master since this release

Release notes:

  1. use only English language for README.md (#45)
  2. added more phpunit tests for RobotsTxtParser, RobotsTxtValidator (#44, #49)
  3. fixed bug in RobotsTxtParser with processing of list of user-agents (#47)
    For example:
User-Agent: ahrefs
User-Agent: SurdotlyBot
Disallow: /

Now will be disallowed path "/" for both user-agents (earlier Disallow: / has been applied only for the latest agent SurdotlyBot)
4. fixed bug with handle character '+' in allow,disallow directives in RobotsTxtValidator (#50)

Assets 2

@bopoda bopoda released this Apr 1, 2017 · 32 commits to master since this release

Main note: refactoring of RobotxTxtParser class to significantly improve performance of parsing a large robots.txt files (100-1000 times) due to improvement of an algorithm of parsing.

  • refactored RobotxTxtParser
  • save full backward compatibility with previous version
  • all phpunit tests passed
Assets 2

@bopoda bopoda released this Mar 16, 2017 · 42 commits to master since this release

class RobotsTxtParser
class RobotsTxtValidator.

all tests passed.

Allowed Directives:

  • noindex
  • allow
  • disallow
  • host
  • sitemap
  • user-agent
  • crawl-delay
  • clean-param
Assets 2
You can’t perform that action at this time.