Emit new digest file with all web URLs encountered in the content

[amartini51]

Feature Name

URL destination checking

Description

I recently found and fixed a number of broken links from The Swift Programming Language (TSPL) to the Swift standard library documentation. As far as I can tell, DocC doesn't currently have a way to detect dead links, so I built a script by hand using cmark and curl to check for this in the TSPL build script.

This seems like functionality that other projects besides TSPL could use, so it would be beneficial to incorporate it as a diagnostic for DocC. Because checking the validity of links takes time and requires a network connection, it's probably something that should be opt-in.

Motivation

No response

Importance

No response

Alternatives Considered

No response

[amartini51]

For reference, here's what I added to the TSPL preflight script:

cmark --to xml TSPL.docc/*/*.md |
xpath -q -n -e '//link/@destination' |
sort -u | cut -f2 -d= | grep -v '^"doc:' | sed 's/^"//; s/"$//' |
while read url
do
    if curl --silent --output /dev/null --fail --head $url
    then
        echo "     $url"
    else
        echo "BAD: $url"
        exit_status=1
    fi
done

[d-ronnqvist]

I would argue that making web requests to verify that web links aren't broken is not something that DocC itself should do.

The specific issue with links to the Swift standard library would probably be better solved by #208 or #419.

What we could possibly do is create another opt-in digest file with all web links in the content so that other scripts don't need to parse the markdown, in-source comments, and tutorial files to find all the links themselves.