Skip to content
A small PHP based website for testing the essentials of a web crawler
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
assets
css
features
files
html
includes
javascript
methods
rss
special
statuscodes
subdomains
.gitignore
CHANGELOG.md
LICENSE.md
README.md
about.php
changelog.php
composer.json
composer.lock
favicon.ico
index.php
license.php
prepend.php
web.config

README.md

HTTP Crawler Test Site

This website was created to replace the original static website used for testing WebCopy features, expanded to cover additional scenarios that the original didn't handle and to ensure a lean site that doesn't slow down integration testing unnecessarily. Of course, writing better tests is just as important! But every little helps, and it's already found some bugs in WebCopy's crawl engine so is serving its purpose quite nicely.

Cyotek is a .NET shop and usually all our products are written in C# using Visual Studio. This time however we decided to keep it simple and this site has been written with PHP, originally using WebMatrix and currently Visual Studio Code. As we aren't PHP coders and don't have much of a clue on best practices for writing PHP or even most of the language the code quality is probably less than stellar. Although it was fairly fun to write - it doesn't need an ever increasing amount of binary dependencies, no assembly bindings or version woes... it just works.

You can view a live demo running at https://demo.cyotek.com. To run the source you should only need PHP 5.3 or above and some form of web server. This site has been tested using IIS Express, IIS 8.5, IIS 10 and Apache running on some variant of Linux.

Acknowledgements

CSS has been taken from Primer by GitHub.

Various bits of PHP were lifted from the PHP user manual.

The fav icon is derived from Material Core Check.

Markdown processing is handled by PHP Markdown Lib.

Some text has been borrowed from Wikipedia and the Mozilla Developer Network.

The srcset demonstration was derived from a WebKit.org demo.

The skyline images were generated by the imaginatively titled Skyline Generator.

The photograph of St Paul's Cathedral in London taken by Vadim Sherbakov.

Generic Avatar graphic by WingTillDie

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.