Skip to content

Releases: wintermi/get-linked-data

v0.3.1

09 Dec 06:44
Compare
Choose a tag to compare
  • Now shuffles the URL list, changing the order which Colly will scrape the data
  • Change the User Agent
  • Change the defaults for Parallelism and Wait Time

v0.3.0

08 Dec 16:08
Compare
Choose a tag to compare
  • Switch to use asynchronous processing within Colly
  • Add additional arguments allowing you to specify the parallelism and maximum delay wait time
  • Replace the automatic retries with outputting the Failed Request URLs to a file

v0.2.0

06 Dec 05:43
Compare
Choose a tag to compare
  • Added a new selector using jq to allow you to reduce a JSON object to just the elements you want, works in conjunction to the element selector.

v0.1.0

05 Dec 22:55
Compare
Choose a tag to compare

A command line application designed to crawl a given set of URLs and scrape the JSON Linked Data (JSON-LD) contained within the webpage before writing the data entries out to a CSV file.