Squidwarc uses json
file format for its crawl configuration.
The full schema of the configuration file is displayed below, however not all fields are required.
{
use: string
mode: string
depth: int
headless: bool
seeds: [string] | string
script: path
warc: {
naming: string
output: path
append: bool
isPartOf: string
infoDescription: string
}
connect: {
host: string
port: string
launch: bool
userDataDir: string
executable: string
}
crawlControl: {
globalWait: int,
inflightIdle: int
navWait: int
numInflight: int
}
}
The only field required bu Squidwarc is the seeds
field for obvious reasons.
If any field is left out besides the seeds
field Squidwarc will either use the default values, listed below,
or set the field to a correct value based on the values of other fields.
The value of this field can be either an array of string (URLs) or a path to an JSON file containing the array of seed URLs
Should Squidwarc use Chrome/Chromium directly or via puppeteer.
When this field is chrome
(default value) then either Chrome/Chromium must be installed and usable.
If the value of this field is puppeteer
then Chrome/Chromium is not required to be installed,
Squidwarc will ensure a usable version of Chromium is automatically downloaded and used.
If the value of the script field is set and the script it points to contains a valid function this value
is set to puppeteer
.
- values:
chrome
orpuppeteer
- default:
chrome
The mode this crawl is to be operating in.
- default: page-only
Preserve the only the page, no links are followed
- values:
page-only
orpo
page-only
mode plus all links found on the page that are on the same domain as the page
- values:
page-same-domain
orpsd
page-only
mode plus all links found on the page
- values:
page-all-links
orpal
The depth of this crawl (how many pages out from a starting seed should be crawled)
- default: 1
Should the browser launched and used by Squidwarc be in headless mode
- default: true
A path to script (js file) to be executed per page, causes the value of use
to be set to puppeteer
.
The js file the script
field points to is expected to export a single async function
that accepts one argument:
module.exports = async function (page) { .... }
Because Squidwarc requires the file to be a Node.js module that exports a single async function
,
the script has full access to the Node.js API.
When Squidwarc executes this function, Squidwarc will pass it the puppeteer Page object for the page being crawled.
Squidwarc will then wait for the Promise returned by the function to resolve and the network to idle before WARC generation begins.
An example script is provided called userFns.js found in the root of this project.
Squidwarc is not responsible for ill-behaved scripts
Options for how the crawls WARCs should be created
The naming scheme to be used for WARC generation
Scheme url
- warcs will be created named
[url crawled]-datetime
.warc - and
append = true
warcs will be created named[first crawled url]-datetime
.warc
The only other value accepted is a file name to be used, this will cause append to be set to true.
Squidwarc will ensure the URLs used apart of filenames will be a valid file name.
- default: url
Path to the directory the WARCs are to be created in
- default: current working directory
Should Squidwarc create a single WARC file for the crawl or a WARC file per page crawled
- default: false
Information about how to connect to or launch Chrome/Chromium
The host name the browsers CDP endpoint is listing on
- default: localhost
The port number the browsers CDP endpoint is listing on
- default: 9222
Should Squidwarc launch and manage the browser or connect to an already running instance.
If this value is set to true and executable is not set, Squidwarc will attempt to find a usable executable
when use = chrome
.
- default: true
Path to the browser executable or command to be used to launch the browser
If this value is set and launch is true
, Squidwarc will use this value rather than attempting to find
an usable executable.
Path to a user data directory to be used rather than a temporary one.
This field is useful when you want to preserve a page that is behind authentication and you are already signed in and view that page using your local version of Chrome/Chromium.
It is recommended to use the full path to the user data directory to ensure 100% usability.
On Linux the default location is in ~/.config:
- [Chrome Stable] ~/.config/google-chrome
- [Chrome Beta] ~/.config/google-chrome-beta
- [Chrome Dev] ~/.config/google-chrome-unstable
- [Chromium] ~/.config/chromium
On macOS the default location is in the Application Support folder:
- [Chrome] ~/Library/Application Support/Google/Chrome
- [Chrome Canary] ~/Library/Application Support/Google/Chrome Canary
- [Chromium] ~/Library/Application Support/Chromium
On Windows the default location is in the local app data folder:
- [Chrome] %LOCALAPPDATA%\Google\Chrome\User Data
- [Chrome Canary] %LOCALAPPDATA%\Google\Chrome SxS\User Data
- [Chromium] %LOCALAPPDATA%\Chromium\User Data
Options for fine tuning the crawl
Maximum amount of time, in milliseconds, that Squidwarc should wait before generating a WARC and moving to the next URL.
- default: 60000
The number of inflight requests (requests with no response) that should exist before starting the inflightIdle timer.
- default: 2
Amount of time, in milliseconds, that should elapse when there are only numInflight requests for network idle to be determined.
- default: 1000
Maximum amount of time, in milliseconds, that Squidwarc should wait for indication that the browser has navigated to the page being crawled.
- default: 8000