Beta release, still a work in progress
This script has two functions.
- Generate
gen - Curl wrapper
curl
This function takes a curltab (a crontab like format) generates a normal crontab.
If you don't want to use the curl wrapper function you can add --wrapper "curl" to the gen command.
./curltab gen < curltab.txt
This function wraps curl. It has two main uses.
Logs all output to syslog and if curl exits non-zero it will output to stdout causing cron to send an email to the admin.
Environment variables may be specified at the top of the curltab. Environment variables must begin with an uppercase letter and can only contain upper case letters, numbers and underscores.
TZ= is a special environment variable. (Example: TZ=Australia/Sydney)
The curltab format is very similar to the crontab format, in fact the only difference is the 6th field where a crontab would normally have the command to be run. In the curltab format the 6th field and the rest of the line should only contain curl arguments. (Note: the command curl should not be specified in the curltab).
<environment variable>
<minute> <hour> <day of month> <month> <day of week> <curl arguments>...
HOST=http://google.com
0 * * * * -sS --connect-timeout 5 --fail --max-time 20 -XGET ${HOST}
The above is the equivalent for running the below curl command hourly
curl -sS --connect-timeout 5 --fail --max-time 20 -XGET http://google.com
--connect-timeout <seconds>
Maximum time in seconds that you allow curl's connection to take. This only limits the connection phase, so if curl connects within the given period it will continue - if not it will exit. Since version 7.32.0, this option accepts decimal values.
See also the -m, --max-time option.
If this option is used several times, the last one will be used.
-f, --fail
(HTTP) Fail silently (no output at all) on server errors. This is mostly done to better enable scripts etc to better deal with failed attempts. In normal cases when an HTTP server fails to deliver a document, it returns an HTML document stating so (which often also describes why and more). This flag will prevent curl from outputting that and return error 22.
This method is not fail-safe and there are occasions where non-successful response codes will slip through, especially when authentication is involved (response codes 401 and 407).
-m, --max-time <seconds>
Maximum time in seconds that you allow the whole operation to take. This is useful for preventing your batch jobs from hanging for hours due to slow networks or links going down. Since 7.32.0, this option accepts decimal values, but the actual timeout will decrease in accuracy as the specified timeout increases in decimal precision. See also the --connect-timeout option.
If this option is used several times, the last one will be used.
-s, --silent
Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute. It will still output the data you ask for, potentially even to the terminal/stdout unless you redirect it.
-S, --show-error
When used with -s it makes curl show an error message if it fails.
The connect-timeout is recommended so curl will fail if the server is not available or is very slow to respond. Similarly max-time is recommended so the whole operation doesn't take too long or hang forever. These two options are especially important if your cron job are every couple of minutes.
The --fail option is important so the curl exit code reflects the http status and failure reports are generated by the curltab curl wrapper.
Finally -sS (short for --silent --show-error) is recommended so the progress meter is hidden but errors are still shown for debugging.