Skip to content
Choose a tag to compare
  • fixed deprecation warning "PHP Deprecated: usort(): Returning bool from comparison function is deprecated", see #61
  • added php 7.4 to travis CI (all tests are passed), see #62
  • return rule for directive "Crawl-Delay" with "0" value, test case is added, see #62
  • test file CrawlDelay.php renamed to proper CrawlDelayTest.php, see #62
Choose a tag to compare
  • never fail on regexp with compilation failed error when run isUrlAllow from RobotsTxtValidator
  • fix travisCI
Choose a tag to compare
  • added composer.json file (needs for
  • move all code to src/RobotsTxtParser folder
  • move tests to tests/RobotsTxtParser folder
  • fix code style to PSR compatible format
  • improve RobotsTxtValidator (getRelativeUrl function)
Choose a tag to compare

Fixed parsing error when the latest allow/disallow directive in the block is empty and the next directive is User-agent.

User-agent: *

User-agent: Linguee
Disallow: /api/showcase

URL "/api/showcase" should be assigned (as Disallow) only to User-agent Linguee, not any.

See PR #54 for more information.

Choose a tag to compare

Release notes:

  1. use only English language for (#45)
  2. added more phpunit tests for RobotsTxtParser, RobotsTxtValidator (#44, #49)
  3. fixed bug in RobotsTxtParser with processing of list of user-agents (#47)
    For example:
User-Agent: ahrefs
User-Agent: SurdotlyBot
Disallow: /

Now will be disallowed path "/" for both user-agents (earlier Disallow: / has been applied only for the latest agent SurdotlyBot)
4. fixed bug with handle character '+' in allow,disallow directives in RobotsTxtValidator (#50)

Choose a tag to compare

Main note: refactoring of RobotxTxtParser class to significantly improve performance of parsing a large robots.txt files (100-1000 times) due to improvement of an algorithm of parsing.

  • refactored RobotxTxtParser
  • save full backward compatibility with previous version
  • all phpunit tests passed
Choose a tag to compare

class RobotsTxtParser
class RobotsTxtValidator.

all tests passed.

Allowed Directives:

  • noindex
  • allow
  • disallow
  • host
  • sitemap
  • user-agent
  • crawl-delay
  • clean-param