Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

jsonscraper

JSON configurable concurrent scraper. Written in Go.

For given JSON config file(s), produces JSON file(s) with results.

Instructions

$ jsonscraper configPath

Output will be saved in file path that is provided in configuration.

You can also run it with multiple configs at once: $ jsonscraper configPath1 configPath2 configPath3 ...

Config documentation

Configuration is an object which consists of urls, targets and output.

urls (Array)

Array of URLs which will be scraped.

targets (Array)

Each object in this array should have atleast:

  • selector is selection query to be perfomed
  • type can be inner HTML (html), plain text (text) or attribute (for example attr:href)
  • tag is field key which appears in output file

Optional is submatch which can contain regular expression. selector can be omitted if submatch is present, then whole document will be used for lookups.

output (Object)

  • path is an output path where data will be saved. $FILENAME will be replaced with input file name.

Example

Input file: input/example.json

{
    "urls": [
        "https://news.ycombinator.com/"
    ],
    "targets": [
        {
            "selector": ".storylink",
            "type": "text",
            "tag": "storyTitleText"
        },
        {
            "selector": ".title",
            "type": "html",
            "tag": "storyTitleHtml"
        },
        {
            "selector": ".storylink",
            "type": "text",
            "tag": "storyTitleWords",
            "submatch": "([a-zA-Z]+)+"
        },
                {
            "selector": ".storylink",
            "type": "attr:href",
            "tag": "storyTitleLinks"
        }
    ],
    "output": {
        "path": "output/$FILENAME"
    }
}

Above configuration will produce following data:

Output file: output/example.json

{
   "storyTitleHtml":[
      "<span class=\"rank\">1.</span>",
      "<a href=\"https://redditblog.com/2017/04/13/how-we-built-rplace/\" class=\"storylink\">How We Built r/Place</a><span class=\"sitebit comhead\"> (<a href=\"from?site=redditblog.com\"><span class=\"sitestr\">redditblog.com</span></a>)</span>",
      "<span class=\"rank\">2.</span>",
      ...
   ],
   "storyTitleLinks":[
      "https://redditblog.com/2017/04/13/how-we-built-rplace/",
      "http://www.bbc.com/news/science-environment-39592059",
      "https://stripe.com/blog/increment",
      ...
   ],
   "storyTitleText":[
      "How We Built r/Place",
      "Saturn moon 'able to support life'",
      "Introducing Increment",
      ...
   ],
   "storyTitleWords":[
      "How",
      "We",
      "Built",
      ...
   ]
}

About

JSON configurable concurrent scraper

Topics

Resources

License

Packages

No packages published