Skip to content

Commit

Permalink
add execution entrance: python main.py
Browse files Browse the repository at this point in the history
  • Loading branch information
debugtalk committed Mar 27, 2017
1 parent e06f968 commit 69a91ff
Show file tree
Hide file tree
Showing 3 changed files with 23 additions and 1 deletion.
16 changes: 15 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,31 @@ A simple web crawler, mainly targets for link validation test.
- configure hyper links regex, including match type and ignore type
- group visited urls by HTTP status code
- flexible configuration in YAML
- send test result by mail
- send test result by mail, through SMTP protocol or mailgun service

## Install

`WebCrawler` can be installed as a CLI tool, or just be used as a script. You can make your preference choice.

If you want to install `WebCrawler`, execute the following command, and all dependencies will be installed as well. Then you can use `webcrawler` CLI tool.

```bash
$ python setup.py install
$ webcrawler -h
```

If you prefer to use `WebCrawler` as a script, you should install dependencies first, then you can start `WebCrawler` through `python main.py` entrance.

```bash
$ pip install -r requirements.txt
$ python main.py -h
```

## Usage

```text
$ python main.py -h
# same as:
$ webcrawler -h
usage: webcrawler [-h] [--log-level LOG_LEVEL] [--seeds SEEDS]
[--crawl-mode CRAWL_MODE] [--max-depth MAX_DEPTH]
Expand Down
2 changes: 2 additions & 0 deletions main.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
from webcrawler import main
main()
6 changes: 6 additions & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
termcolor
PyYAML
future
lxml
cssselect
requests

0 comments on commit 69a91ff

Please sign in to comment.