Both server and agent use the scrapydd.conf
file for system configuration.
The file will be looked up in the following locations:
- /etc/scrapydd/scrapydd.conf
- /etc/scrapyd/conf.d/*
- ./scrapydd.conf
- ~/.scrapydd.conf
Config can be also overriden by environment variables, environment variables should have a "SCRAPYDD_" prefix and then the config name with upper case. For example to override a server address on agent, an environment variable should be "SCRAPYDD_SERVER=xxx".
Server configurations should appears under the [server]
section.
The ipaddress which web server bind on. Default: 0.0.0.0
The port web server running on. Default: 6800
Whether validate client's certificate on SSL, Default: false
Database connection url. This will be passed to the inside sqlalchemy create_engine method.
Default: sqlite:///database.db
Whether run server on debug mode. Debug mode will set logging level to DEBUG.
Default: false
.
Whether enable authentication, once this option is on, user need to login to make operation.
Default: true
HTTPS port to listen on, specify this key will enable SSL mode.
Default: None
Project package runner, Default: venv.
Available options: venv(run sub-command on VirtuanEnv), docker (run sub-command on Docker container)
Runner container image name, Default: kevenli/scrpaydd
This effects when runner_type is docker.
Server's hostname.
When SSL enabled, the public certificate will be loaded as filename server_name.crt and
private certificate will be loaded as filename server_name.key in the keys
folder.
Default: localhost
Agent configurations should appears under the [agent]
section.
Whether run agent on debug mode. Debug mode will set logging level to DEBUG. Default: false
The IP address or hostname of the server which this agent connect to. Default: localhost
The port of server. Default: 6800
How many concurrent jobs the agent would run. Default: 1
Request timeout in seconds when communicating to server. Default: 60
Server configuration:
[server] bind_address = 0.0.0.0 bind_port = 6800 debug = false
Agent configuration:
[agent] server = localhost server_port = 6800 debug = false slots = 1