Add missing option and settings documentation

doc/man/de/linkchecker.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH "LINKCHECKER" "1" "August 06, 2020" "" "LinkChecker"
+.TH "LINKCHECKER" "1" "August 08, 2020" "" "LinkChecker"
 .SH NAME
 linkchecker \- command line client to check HTML documents and websites for broken links
 .
@@ -250,6 +250,11 @@ Prüfe URLs die auf den regulären Ausdruck zutreffen, aber führe keine Rekursi
 .UNINDENT
 .INDENT 0.0
 .TP
+.B \-\-no\-robots
+Check URLs regardless of any robots.txt files.
+.UNINDENT
+.INDENT 0.0
+.TP
 .B \-p, \-\-password
 Liest ein Passwort von der Kommandozeile und verwende es für HTTP und FTP Autorisierung. Für FTP ist das Standardpasswort anonymous@. Für HTTP gibt es kein Standardpasswort. Siehe auch \fI\%\-u\fP\&.
 .UNINDENT

doc/man/de/linkcheckerrc.5

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH "LINKCHECKERRC" "5" "August 06, 2020" "" "LinkChecker"
+.TH "LINKCHECKERRC" "5" "August 08, 2020" "" "LinkChecker"
 .SH NAME
 linkcheckerrc \- configuration file for LinkChecker
 .
@@ -40,6 +40,11 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 \fBcookiefile=\fP\fIDateiname\fP
 Lese eine Datei mit Cookie\-Daten. Das Cookie Datenformat wird in \fBlinkchecker(1)\fP erklärt. Kommandozeilenoption: \fB\-\-cookiefile\fP
 .TP
+\fBdebugmemory=\fP[\fB0\fP|\fB1\fP]
+Write memory allocation statistics to a file on exit, requires \fI\%meliae\fP\&.
+The default is not to write the file.
+Command line option: none
+.TP
 \fBlocalwebroot=\fP\fISTRING\fP
 Beachten Sie dass das angegebene Verzeichnis in URL\-Syntax sein muss, d.h. es muss einen normalen statt einen umgekehrten Schrägstrich zum Aneinanderfügen von Verzeichnissen benutzen. Und das angegebene Verzeichnis muss mit einem Schrägstrich enden. Kommandozeilenoption: none
 .TP
@@ -70,14 +75,35 @@ Falls der Wert Null ist werden SSL Zertifikate nicht überprüft. Falls er auf E
 \fBmaxrunseconds=\fP\fINUMMER\fP
 Hört nach der angegebenen Anzahl von Sekunden auf, neue URLs zu prüfen. Dies ist dasselbe als wenn der Benutzer nach der gegebenen Anzahl von Sekunden stoppt (durch Drücken von Strg\-C). Kommandozeilenoption: none
 .TP
+\fBmaxfilesizedownload=\fP\fINUMBER\fP
+Files larger than NUMBER bytes will be ignored, without downloading anything
+if accessed over http and an accurate Content\-Length header was returned.
+No more than this amount of a document will be downloaded.
+The default is 5242880 (5 MB).
+Command line option: none
+.TP
+\fBmaxfilesizeparse=\fP\fINUMBER\fP
+Files larger than NUMBER bytes will not be parsed for links.
+The default is 1048576 (1 MB).
+Command line option: none
+.TP
 \fBmaxnumurls=\fP\fINUMMER\fP
 Maximale Anzahl von URLs die geprüft werden. Neue URLs werden nicht angenommen nachdem die angegebene Anzahl von URLs geprüft wurde. Kommandozeilenoption: none
 .TP
 \fBmaxrequestspersecond=\fP\fINUMMER\fP
 Limit the maximum number of requests per second to one host.
+The default is 10.
+Command line option: none
+.TP
+\fBrobotstxt=\fP[\fB0\fP|\fB1\fP]
+When using http, fetch robots.txt, and confirm whether each URL should
+be accessed before checking.
+The default is to use robots.txt files.
+Command line option: \fB\-\-no\-robots\fP
 .TP
 \fBallowedschemes=\fP\fINAME\fP[\fB,\fP\fINAME\fP\&...]
 Allowed URL schemes as comma\-separated list.
+Command line option: none
 .UNINDENT
 .SS filtering
 .INDENT 0.0

doc/man/en/linkchecker.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH "LINKCHECKER" "1" "August 06, 2020" "" "LinkChecker"
+.TH "LINKCHECKER" "1" "August 08, 2020" "" "LinkChecker"
 .SH NAME
 linkchecker \- command line client to check HTML documents and websites for broken links
 .
@@ -32,7 +32,7 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
 ..
 .SH SYNOPSIS
 .sp
-\fBlinkchecker\fP [\fIoptions\fP] [\fIfile\-or\-url\fP]...
+\fBlinkchecker\fP [\fIoptions\fP] [\fIfile\-or\-url\fP]…
 .SH DESCRIPTION
 .sp
 LinkChecker features
@@ -81,7 +81,7 @@ $ linkchecker http://www.example.com/
 Beware that this checks the whole site which can have thousands of
 URLs. Use the \fI\%\-r\fP option to restrict the recursion depth.
 .sp
-Don\(aqt check URLs with \fB/secret\fP in its name. All other links are
+Don’t check URLs with \fB/secret\fP in its name. All other links are
 checked as usual:
 .INDENT 0.0
 .INDENT 3.5
@@ -224,7 +224,7 @@ Do not print check status messages.
 .INDENT 0.0
 .TP
 .B \-\-no\-warnings
-Don\(aqt log warnings. Default is to log warnings.
+Don’t log warnings. Default is to log warnings.
 .UNINDENT
 .INDENT 0.0
 .TP
@@ -255,9 +255,9 @@ Define a regular expression which prints a warning if it matches any
 content of the checked link. This applies only to valid pages, so we
 can get their content.
 Use this to check for pages that contain some form of error, for
-example "This page has moved" or "Oracle Application error".
+example “This page has moved” or “Oracle Application error”.
 Note that multiple values can be combined in the regular expression,
-for example "(This page has moved|Oracle Application error)".
+for example “(This page has moved|Oracle Application error)”.
 See section \fI\%REGULAR EXPRESSIONS\fP for more info.
 .UNINDENT
 .SS Checking options
@@ -297,6 +297,11 @@ See section \fI\%REGULAR EXPRESSIONS\fP for more info.
 .UNINDENT
 .INDENT 0.0
 .TP
+.B \-\-no\-robots
+Check URLs regardless of any robots.txt files.
+.UNINDENT
+.INDENT 0.0
+.TP
 .B \-p, \-\-password
 Read a password from console and use it for HTTP and FTP
 authorization. For FTP the default password is anonymous@. For
@@ -325,7 +330,7 @@ username. See also \fI\%\-p\fP\&.
 .TP
 .B \-\-user\-agent=STRING
 Specify the User\-Agent string to send to the HTTP server, for
-example "Mozilla/4.0". The default is "LinkChecker/X.Y" where X.Y is
+example “Mozilla/4.0”. The default is “LinkChecker/X.Y” where X.Y is
 the current version of LinkChecker.
 .UNINDENT
 .SH CONFIGURATION FILES
@@ -584,7 +589,7 @@ It must not match the ignored URL list. This is controlled with the
 \fI\%\-\-ignore\-url\fP option.
 .IP 6. 3
 The Robots Exclusion Protocol must allow links in the URL to be
-followed recursively. This is checked by searching for a "nofollow"
+followed recursively. This is checked by searching for a “nofollow”
 directive in the HTML header data.
 .UNINDENT
 .sp
@@ -607,7 +612,7 @@ automatically.
 .sp
 You can supply multiple user/password pairs in a configuration file.
 .sp
-When checking \fBnews:\fP links the given NNTP host doesn\(aqt need to be the
+When checking \fBnews:\fP links the given NNTP host doesn’t need to be the
 same as the host of the user browsing your pages.
 .SH ENVIRONMENT
 .INDENT 0.0

doc/man/en/linkcheckerrc.5

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH "LINKCHECKERRC" "5" "August 06, 2020" "" "LinkChecker"
+.TH "LINKCHECKERRC" "5" "August 08, 2020" "" "LinkChecker"
 .SH NAME
 linkcheckerrc \- configuration file for LinkChecker
 .
@@ -45,6 +45,11 @@ Read a file with initial cookie data. The cookie data format is
 explained in \fBlinkchecker(1)\fP\&.
 Command line option: \fB\-\-cookiefile\fP
 .TP
+\fBdebugmemory=\fP[\fB0\fP|\fB1\fP]
+Write memory allocation statistics to a file on exit, requires \fI\%meliae\fP\&.
+The default is not to write the file.
+Command line option: none
+.TP
 \fBlocalwebroot=\fP\fISTRING\fP
 When checking absolute URLs inside local files, the given root
 directory is used as base URL.
@@ -82,7 +87,7 @@ Command line option: \fB\-\-timeout\fP
 .TP
 \fBuseragent=\fP\fISTRING\fP
 Specify the User\-Agent string to send to the HTTP server, for
-example "Mozilla/4.0". The default is "LinkChecker/X.Y" where X.Y is
+example “Mozilla/4.0”. The default is “LinkChecker/X.Y” where X.Y is
 the current version of LinkChecker.
 Command line option: \fB\-\-user\-agent\fP
 .TP
@@ -100,6 +105,18 @@ seconds.
 The default is not to stop until all URLs are checked.
 Command line option: none
 .TP
+\fBmaxfilesizedownload=\fP\fINUMBER\fP
+Files larger than NUMBER bytes will be ignored, without downloading anything
+if accessed over http and an accurate Content\-Length header was returned.
+No more than this amount of a document will be downloaded.
+The default is 5242880 (5 MB).
+Command line option: none
+.TP
+\fBmaxfilesizeparse=\fP\fINUMBER\fP
+Files larger than NUMBER bytes will not be parsed for links.
+The default is 1048576 (1 MB).
+Command line option: none
+.TP
 \fBmaxnumurls=\fP\fINUMBER\fP
 Maximum number of URLs to check. New URLs will not be queued after
 the given number of URLs is checked.
@@ -108,9 +125,18 @@ Command line option: none
 .TP
 \fBmaxrequestspersecond=\fP\fINUMBER\fP
 Limit the maximum number of requests per second to one host.
+The default is 10.
+Command line option: none
 .TP
-\fBallowedschemes=\fP\fINAME\fP[\fB,\fP\fINAME\fP\&...]
+\fBrobotstxt=\fP[\fB0\fP|\fB1\fP]
+When using http, fetch robots.txt, and confirm whether each URL should
+be accessed before checking.
+The default is to use robots.txt files.
+Command line option: \fB\-\-no\-robots\fP
+.TP
+\fBallowedschemes=\fP\fINAME\fP[\fB,\fP\fINAME\fP…]
 Allowed URL schemes as comma\-separated list.
+Command line option: none
 .UNINDENT
 .SS filtering
 .INDENT 0.0
@@ -119,7 +145,7 @@ Allowed URL schemes as comma\-separated list.
 Only check syntax of URLs matching the given regular expressions.
 Command line option: \fB\-\-ignore\-url\fP
 .TP
-\fBignorewarnings=\fP\fINAME\fP[\fB,\fP\fINAME\fP\&...]
+\fBignorewarnings=\fP\fINAME\fP[\fB,\fP\fINAME\fP…]
 Ignore the comma\-separated list of warnings. See \fI\%WARNINGS\fP for
 the list of supported warnings.
 Command line option: none
@@ -178,14 +204,14 @@ form.
 .SS output
 .INDENT 0.0
 .TP
-\fBdebug=\fP\fISTRING\fP[\fB,\fP\fISTRING\fP\&...]
+\fBdebug=\fP\fISTRING\fP[\fB,\fP\fISTRING\fP…]
 Print debugging output for the given modules. Available debug
 modules are \fBcmdline\fP, \fBchecking\fP, \fBcache\fP, \fBdns\fP,
 \fBthread\fP, \fBplugins\fP and \fBall\fP\&. Specifying \fBall\fP is an alias
 for specifying all available loggers.
 Command line option: \fB\-\-debug\fP
 .TP
-\fBfileoutput=\fP\fITYPE\fP[\fB,\fP\fITYPE\fP\&...]
+\fBfileoutput=\fP\fITYPE\fP[\fB,\fP\fITYPE\fP…]
 Output to a file \fBlinkchecker\-out.\fP\fITYPE\fP, or
 \fB$HOME/.linkchecker/blacklist\fP for \fBblacklist\fP output.
 Valid file output types are \fBtext\fP, \fBhtml\fP, \fBsql\fP, \fBcsv\fP,
@@ -321,7 +347,7 @@ See \fI\%[text]\fP section above.
 Set CSV separator. Default is a comma (\fB,\fP).
 .TP
 \fBquotechar=\fP\fICHAR\fP
-Set CSV quote character. Default is a double quote (\fB"\fP).
+Set CSV quote character. Default is a double quote (\fB“\fP).
 .UNINDENT
 .SS sql
 .INDENT 0.0
@@ -445,10 +471,10 @@ valid or invalid, with messages
 1 or 0, only in some logger types reported
 .TP
 \fBbase\fP
-base href=...
+base href=…
 .TP
 \fBname\fP
-<a href=...>name</a> and <img alt="name">
+<a href=…>name</a> and <img alt=”name”>
 .TP
 \fBparenturl\fP
 if any
@@ -469,10 +495,10 @@ check time
 the original url name, can be relative
 .TP
 \fBintro\fP
-the blurb at the beginning, "starting at ..."
+the blurb at the beginning, “starting at …”
 .TP
 \fBoutro\fP
-the blurb at the end, "found x errors ..."
+the blurb at the end, “found x errors …”
 .UNINDENT
 .SH MULTILINE
 .sp
@@ -532,11 +558,11 @@ get their content.
 .TP
 \fBwarningregex=\fP\fIREGEX\fP
 Use this to check for pages that contain some form of error message,
-for example "This page has moved" or "Oracle Application error".
+for example “This page has moved” or “Oracle Application error”.
 \fIREGEX\fP should be unquoted.
 .sp
 Note that multiple values can be combined in the regular expression,
-for example "(This page has moved|Oracle Application error)".
+for example “(This page has moved|Oracle Application error)”.
 .UNINDENT
 .SS SslCertificateCheck
 .sp
@@ -557,9 +583,9 @@ Check the syntax of HTML pages with the online W3C HTML validator. See
 Print HTTP headers in URL info.
 .INDENT 0.0
 .TP
-\fBprefixes=\fP\fIprefix1\fP[,*prefix2*]...
+\fBprefixes=\fP\fIprefix1\fP[,*prefix2*]…
 List of comma separated header prefixes. For example to display all
-HTTP headers that start with "X\-".
+HTTP headers that start with “X\-“.
 .UNINDENT
 .SS CssSyntaxCheck
 .sp
@@ -584,7 +610,7 @@ Parse Word files for URLs to check. Needs the \fI\%pywin32\fP Python
 extension installed.
 .SH WARNINGS
 .sp
-The following warnings are recognized in the \(aqignorewarnings\(aq config
+The following warnings are recognized in the ‘ignorewarnings’ config
 file entry:
 .INDENT 0.0
 .TP

doc/src/man/linkchecker.rst

@@ -208,6 +208,10 @@ Checking options
     This option can be given multiple times.
     See section `REGULAR EXPRESSIONS`_ for more info.
 
+.. option:: --no-robots
+
+    Check URLs regardless of any robots.txt files.
+
 .. option:: -p, --password
 
     Read a password from console and use it for HTTP and FTP

doc/src/man/linkcheckerrc.rst

@@ -21,6 +21,10 @@ checking
     Read a file with initial cookie data. The cookie data format is
     explained in :manpage:`linkchecker(1)`.
     Command line option: :option:`--cookiefile`
+**debugmemory=**\ [**0**\ \|\ **1**]
+    Write memory allocation statistics to a file on exit, requires :pypi:`meliae`.
+    The default is not to write the file.
+    Command line option: none
 **localwebroot=**\ *STRING*
     When checking absolute URLs inside local files, the given root
     directory is used as base URL.
@@ -67,15 +71,33 @@ checking
     seconds.
     The default is not to stop until all URLs are checked.
     Command line option: none
+**maxfilesizedownload=**\ *NUMBER*
+    Files larger than NUMBER bytes will be ignored, without downloading anything
+    if accessed over http and an accurate Content-Length header was returned.
+    No more than this amount of a document will be downloaded.
+    The default is 5242880 (5 MB).
+    Command line option: none
+**maxfilesizeparse=**\ *NUMBER*
+    Files larger than NUMBER bytes will not be parsed for links.
+    The default is 1048576 (1 MB).
+    Command line option: none
 **maxnumurls=**\ *NUMBER*
     Maximum number of URLs to check. New URLs will not be queued after
     the given number of URLs is checked.
     The default is to queue and check all URLs.
     Command line option: none
 **maxrequestspersecond=**\ *NUMBER*
     Limit the maximum number of requests per second to one host.
+    The default is 10.
+    Command line option: none
+**robotstxt=**\ [**0**\ \|\ **1**]
+    When using http, fetch robots.txt, and confirm whether each URL should
+    be accessed before checking.
+    The default is to use robots.txt files.
+    Command line option: :option:`--no-robots`
 **allowedschemes=**\ *NAME*\ [**,**\ *NAME*...]
     Allowed URL schemes as comma-separated list.
+    Command line option: none
 
 filtering
 ^^^^^^^^^