ci: add periodic link check lychee (#733)

# Summary

This updates the pre-commit hooks and adds a periodic link check against
a set of urls.

It deprecates the use of markdownlinkchecker in favour of lychee.

- [x] First, we add the a job be run daily which uses lychee to check
the links present in a series of pages under a given root.
- [x] Next, we remove the existing link checker periodic actions.

Subsequent PRs can enable opening issues with results of broken links.

Other changes:

- remove check-spelling action: conflcits with cspell, not available in
megalinter via pre-commit, is a source of unnecessary toil.
- Remove links to community.egi.eu to show that lychee is actually doing
its job
- Remove unnused packages in `package.json` which seemed to be included
for potential testing activities which were never done. This addresses
several critical vulnerabilities identified by github (but which we are
not actually exposed by).

---

**Related issue :**

Closed #157

---------

Signed-off-by: Bruce Becker <bruce.becker@egi.eu>

Author: brucellino

.cspell.json

@@ -0,0 +1,11 @@
+---
+language: "en-gb"
+dictionaries:
+  - custom-words
+ignorePaths:
+  - ".github/actions/**"
+  - ".cspell.json"
+dictionaryDefinitions:
+  - name: custom-words
+    path: ./.github/actions/spelling/expect.txt
+    addWords: true

.cspell.yml

@@ -32,7 +32,7 @@ and orientation.
 
 ## Our Standards
 
-Examples of behavior that contributes to creating a positive environment
+Examples of behaviour that contributes to creating a positive environment
 include:
 
 - Using welcoming and inclusive language
@@ -41,7 +41,7 @@ include:
 - Focusing on what is best for the community
 - Showing empathy towards other community members
 
-Examples of unacceptable behavior by participants include:
+Examples of unacceptable behaviour by participants include:
 
 - The use of sexualized language or imagery and unwelcome sexual attention or
   advances
@@ -55,13 +55,13 @@ Examples of unacceptable behavior by participants include:
 ## Our Responsibilities
 
 Project maintainers are responsible for clarifying the standards of acceptable
-behavior and are expected to take appropriate and fair corrective action in
-response to any instances of unacceptable behavior.
+behaviour and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behaviour.
 
 Project maintainers have the right and responsibility to remove, edit, or reject
 comments, commits, code, wiki edits, issues, and other contributions that are
 not aligned to this Code of Conduct, or to ban temporarily or permanently any
-contributor for other behaviors that they deem inappropriate, threatening,
+contributor for other behaviours that they deem inappropriate, threatening,
 offensive, or harmful.
 
 ## Scope
@@ -75,8 +75,8 @@ further defined and clarified by project maintainers.
 
 ## Enforcement
 
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the EGI Foundation team at contact@egi.eu. The team will
+Instances of abusive, harassing, or otherwise unacceptable behaviour may be
+reported by contacting the EGI Foundation team at <contact@egi.eu>. The team will
 review and investigate all complaints, and will respond in a way that it deems
 appropriate to the circumstances. The team is obligated to maintain
 confidentiality with regard to the reporter of an incident. Further details of

.github/CODE_OF_CONDUCT.md

@@ -1,13 +1,13 @@
 # See https://github.com/check-spelling/check-spelling/wiki/Configuration-Examples:-excludes
-# Ignore some docsy-realted folders
+# Ignore some docsy-related folders
 ^archetypes/
 ^assets/
 ^themes/
 ^layouts/
 ^public/
 ^static/
 ^resources/
-^\.github/
+^\.github/workflows
 ^\.gitmodules
 CODEOWNERS
 AUTHORS

.github/actions/spelling/excludes.txt

@@ -1,15 +1,20 @@
 ADFS
 allowfullscreen
 autoplay
+autoprefixer
 checkin
 creds
 csr
 CVMFS
+CYFRONET
+DataHub
 dearmor
 delims
 dependant
 diracsgm
+docsy
 findstr
+flox
 galdyn
 gecos
 geodata
@@ -22,12 +27,22 @@ Initialising
 jscpd
 jwt
 ligo
+linter
+linters
 lsb
 markdownlint
 mega
+megalinter
 mytoken
 mytokens
 newkey
+Oneclient
+Oneclients
+Onedata
+Oneprovider
+Oneproviders
+Onezone
+Onezones
 pem
 pheno
 pravda

.github/actions/spelling/expect.txt

@@ -1,6 +1,7 @@
 ---
 name: Deploy pull request preview
 # Setup and build files with hugo
+# cspell:disable-next-line
 # https://github.com/peaceiris/actions-hugo
 # Deploy to GitHub pages
 # https://github.com/peaceiris/actions-gh-pages
@@ -56,7 +57,7 @@ jobs:
           echo "pr_number=$PR_NUMBER" >> "$GITHUB_OUTPUT"
 
       - name: Deploy Pull Request preview
-        uses: peaceiris/actions-gh-pages@v4
+        uses: peaceiris/actions-gh-pages@v4 # # cspell:disable-line
         with:
           # Use GITHUB_TOKEN allowing to write to local repository
           github_token: ${{ secrets.GITHUB_TOKEN }}
@@ -66,7 +67,7 @@ jobs:
           publish_branch: pr_previews
           # Source directory
           publish_dir: ./${{ steps.extract.outputs.pr_number }}/
-          # Destitnation directory
+          # Destination directory
           destination_dir: ${{ steps.extract.outputs.pr_number }}
           # Reuse PR commit message
           commit_message: ${{ github.event.head_commit.message }}
@@ -86,3 +87,172 @@ jobs:
               repo: context.repo.repo,
               body: output
             })
+  check_preview_links_users:
+    env:
+      BASE_URL: https://docs.egi.eu/documentation/users
+    runs-on: ubuntu-latest
+    needs:
+      - deploy_pr_preview
+    if: >
+      github.event.workflow_run.event == 'pull_request' &&
+      github.event.workflow_run.conclusion == 'success'
+    strategy:
+      matrix:
+        # This matrix is built from the directories in content/en/users
+        url:
+          - "aai"
+          - "compute"
+          - "data"
+          - "dev-env"
+          - "getting-started"
+          # - "machine-learning" # currently in draft
+          - "security"
+          - "training"
+          - "tutorials"
+    # cspell:disable
+    name: Check links of preview
+    steps:
+      - name: Checkout configuration
+        uses: actions/checkout@v4
+      - name: Get tools
+        # We will run hakrawler to crawl a page and sub-pages
+        # Then use lychee to check whether the urls in that page are ok.
+        run: |
+          curl -fSL https://github.com/lycheeverse/lychee/releases/download/lychee-v0.19.1/lychee-x86_64-unknown-linux-musl.tar.gz | tar xvfz -> lychee
+          go install github.com/hakluke/hakrawler@latest
+      - name: Extract links and check them.
+        # This passes a given endpoint to hakrawler, which gets all of the links
+        # in it up to a depth of 10 (-d) and without crawling back up the tree (-i)
+        # and returning only unique (-u) urls.
+        # The urls are then parsed via jq to get only hrefs, no scripts
+        # and the resulting list is sent to lyuchee via stdin (-)
+        # to see if there are any bad links in those pages
+        run: >-
+          echo "https://docs.egi.eu/documentation/${{
+          job.deploy_pr_preview.outputs.pr_number }}/users/${{ matrix.url }}" |
+            hakrawler -u -i -d 10 -json |
+            jq -r '. |
+            select(.Source == "href") | .URL' |
+            lychee -
+  check_preview_links_internal:
+    env:
+      BASE_URL: https://docs.egi.eu/documentation/internal
+    runs-on: ubuntu-latest
+    needs:
+      - deploy_pr_preview
+    if: >
+      github.event.workflow_run.event == 'pull_request' &&
+      github.event.workflow_run.conclusion == 'success'
+    strategy:
+      matrix:
+        # This matrix is built from the directories in content/en/users
+        url:
+          - "accounting"
+          - "collaboration-tools"
+          - "configuration-database"
+          - "getting-started"
+          - "guidelines-software-development"
+          - "helpdesk" # cspell:disable-line
+          - "messaging"
+          - "monitoring"
+          - "operations-portal"
+          - "security-coordination"
+    # cspell:disable
+    name: Check links of preview
+    steps:
+      - name: Checkout configuration
+        uses: actions/checkout@v4
+      - name: Get tools
+        # We will run hakrawler to crawl a page and sub-pages
+        # Then use lychee to check whether the urls in that page are ok.
+        run: |
+          curl -fSL https://github.com/lycheeverse/lychee/releases/download/lychee-v0.19.1/lychee-x86_64-unknown-linux-musl.tar.gz | tar xvfz -> lychee
+          go install github.com/hakluke/hakrawler@latest
+      - name: Extract links and check them.
+        # This passes a given endpoint to hakrawler, which gets all of the links
+        # in it up to a depth of 10 (-d) and without crawling back up the tree (-i)
+        # and returning only unique (-u) urls.
+        # The urls are then parsed via jq to get only hrefs, no scripts
+        # and the resulting list is sent to lyuchee via stdin (-)
+        # to see if there are any bad links in those pages
+        run: >-
+          echo "https://docs.egi.eu/documentation/${{
+          job.deploy_pr_preview.outputs.pr_number }}/internal/${{ matrix.url }}"
+          |
+            hakrawler -u -i -d 10 -json |
+            jq -r '. |
+            select(.Source == "href") | .URL' |
+            lychee -
+  check_preview_links_providers:
+    env:
+      BASE_URL: https://docs.egi.eu/documentation/providers
+    runs-on: ubuntu-latest
+    needs:
+      - deploy_pr_preview
+    if: >
+      github.event.workflow_run.event == 'pull_request' &&
+      github.event.workflow_run.conclusion == 'success'
+    strategy:
+      matrix:
+        # This matrix is built from the directories in content/en/users
+        url:
+          - "check-in"
+          - "cloud-compute"
+          - "datahub" # cspell:disable-line
+          - "high-throughput-compute"
+          - "joining"
+          - "notebooks"
+          - "online-storage"
+          - "operations-manuals"
+          - "rod"
+    # cspell:disable
+    name: Check links of preview
+    steps:
+      - name: Checkout configuration
+        uses: actions/checkout@v4
+      - name: Get tools
+        # We will run hakrawler to crawl a page and sub-pages
+        # Then use lychee to check whether the urls in that page are ok.
+        run: |
+          curl -fSL https://github.com/lycheeverse/lychee/releases/download/lychee-v0.19.1/lychee-x86_64-unknown-linux-musl.tar.gz | tar xvfz -> lychee
+          go install github.com/hakluke/hakrawler@latest
+      - name: Extract links and check them.
+        # This passes a given endpoint to hakrawler, which gets all of the links
+        # in it up to a depth of 10 (-d) and without crawling back up the tree (-i)
+        # and returning only unique (-u) urls.
+        # The urls are then parsed via jq to get only hrefs, no scripts
+        # and the resulting list is sent to lyuchee via stdin (-)
+        # to see if there are any bad links in those pages
+        run: >-
+          echo "https://docs.egi.eu/documentation/${{
+          job.deploy_pr_preview.outputs.pr_number }}/providers/${{ matrix.url
+          }}" |
+            hakrawler -u -i -d 10 -json |
+            jq -r '. |
+            select(.Source == "href") | .URL' |
+            lychee -
+  lychee-otherdocs: # cspell:disable-line
+    env:
+      BASE_URL: https://docs.egi.eu/
+    name: Check links at base of the about pages
+    strategy:
+      matrix:
+        url:
+          - ""
+          - about
+          - support
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Get tools
+        # On the "other" page, we do not want to crawl down, so we only use
+        # lyuchee to check the page itself.
+        run: |
+          curl -fSL https://github.com/lycheeverse/lychee/releases/download/lychee-v0.19.1/lychee-x86_64-unknown-linux-musl.tar.gz | tar xvfz -> lychee
+      - name: Check Links.
+        run:
+          lychee https://docs.egi.eu/documentation/${{
+          job.deploy_pr_preview.outputs.pr_number }}/${{ matrix.url }}
+  # cspell:enable