Skip to content

Commit

Permalink
added docs for DomainFilter usage
Browse files Browse the repository at this point in the history
  • Loading branch information
Ziinc committed Dec 20, 2019
1 parent 370b17b commit 6ce3192
Showing 1 changed file with 13 additions and 2 deletions.
15 changes: 13 additions & 2 deletions lib/crawly/middlewares/domain_filter.ex
Original file line number Diff line number Diff line change
@@ -1,12 +1,23 @@
defmodule Crawly.Middlewares.DomainFilter do
@moduledoc """
Filters out requests which are going outside of the crawled domain
Filters out requests which are going outside of the crawled domain.
The domain that is used to compare against the request url is obtained from the spider's `c:Crawly.Spider.base_url` callback.
Does not accept any options. Tuple-based configuration optionswill be ignored.
### Example Declaration
```
middlewares: [
Crawly.Middlewares.DomainFilter
]
```
"""

@behaviour Crawly.Pipeline
require Logger

def run(request, state) do
def run(request, state, _opts \\ []) do
base_url = state.spider_name.base_url()

case String.contains?(request.url, base_url) do
Expand Down

0 comments on commit 6ce3192

Please sign in to comment.