Flask App to scrape hrefs from websites and validate links
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
lib
static
templates
tests
.dockerignore
.gitattributes
.gitignore
.travis.yml
Dockerfile
README.md
main.py
package.json
requirements.txt

README.md

web-link-tester

Build Status Docker Repository on Quay Docker Automated buil

Asyncronous Flask App to scrape and validate links via GUI or REST API

Usage

Why use the web-link-tester?

  1. Test your production site via GUI or programmatically to ensure no bad page routes or forgotten references.
  2. It's free.

To deploy the Web App/API


Docker

docker run -d --name link-tester -p 80:80 circa10a/web-link-tester

Access via http://localhost

Python

python main.py

Note This method may require to run as root unless you change the port number in main.py

Access via http://localhost

API Usage

curl -X POST --data "https://www.github.com" http://localhost/api

Example JSON Output

$ curl -X POST --data "http://caleblemoine.me" http://localhost/api
{
  "links": [
    {
      "code": 200,
      "url": "https://github.com/circa10a/"
    },
    {
      "code": 200,
      "url": "https://hub.docker.com/r/circa10a/"
    },
    {
      "code": 200,
      "url": "https://circa10a.github.io/monitor/"
    },
    {
      "code": 200,
      "url": "https://circa10a.github.io/smart-mirror/"
    },
    {
      "code": 200,
      "url": "http://scrapeyour.site"
    }
  ]
}

Stack

  • Utilizes gunicorn for multiple workers/threading.
  • Python 3
  • BeautifulSoup4
  • Jquery

Screenshots

alt text alt text