Skip to content
JSON configurable concurrent scraper
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
input
output
scraper
.gitignore
LICENSE
README.md
build.sh
clean.sh
main.go

README.md

jsonscraper

JSON configurable concurrent scraper. Written in Go.

For given JSON config file(s), produces JSON file(s) with results.

Instructions

$ jsonscraper configPath

Output will be saved in file path that is provided in configuration.

You can also run it with multiple configs at once: $ jsonscraper configPath1 configPath2 configPath3 ...

Config documentation

Configuration is an object which consists of urls, targets and output.

urls (Array)

Array of URLs which will be scraped.

targets (Array)

Each object in this array should have atleast:

  • selector is selection query to be perfomed
  • type can be inner HTML (html), plain text (text) or attribute (for example attr:href)
  • tag is field key which appears in output file

Optional is submatch which can contain regular expression. selector can be omitted if submatch is present, then whole document will be used for lookups.

output (Object)

  • path is an output path where data will be saved. $FILENAME will be replaced with input file name.

Example

Input file: input/example.json

{
    "urls": [
        "https://news.ycombinator.com/"
    ],
    "targets": [
        {
            "selector": ".storylink",
            "type": "text",
            "tag": "storyTitleText"
        },
        {
            "selector": ".title",
            "type": "html",
            "tag": "storyTitleHtml"
        },
        {
            "selector": ".storylink",
            "type": "text",
            "tag": "storyTitleWords",
            "submatch": "([a-zA-Z]+)+"
        },
                {
            "selector": ".storylink",
            "type": "attr:href",
            "tag": "storyTitleLinks"
        }
    ],
    "output": {
        "path": "output/$FILENAME"
    }
}

Above configuration will produce following data:

Output file: output/example.json

{
   "storyTitleHtml":[
      "<span class=\"rank\">1.</span>",
      "<a href=\"https://redditblog.com/2017/04/13/how-we-built-rplace/\" class=\"storylink\">How We Built r/Place</a><span class=\"sitebit comhead\"> (<a href=\"from?site=redditblog.com\"><span class=\"sitestr\">redditblog.com</span></a>)</span>",
      "<span class=\"rank\">2.</span>",
      ...
   ],
   "storyTitleLinks":[
      "https://redditblog.com/2017/04/13/how-we-built-rplace/",
      "http://www.bbc.com/news/science-environment-39592059",
      "https://stripe.com/blog/increment",
      ...
   ],
   "storyTitleText":[
      "How We Built r/Place",
      "Saturn moon 'able to support life'",
      "Introducing Increment",
      ...
   ],
   "storyTitleWords":[
      "How",
      "We",
      "Built",
      ...
   ]
}
You can’t perform that action at this time.