Crawlabout v2.2.0

Added

• Exports show their progress: a bar at the bottom right of the window,
  with a button to cancel. For a large check a PDF or HTML report with
  every URL takes a while.

Changed

• Every report (JSON, CSV, HTML, PDF and the list exports) is sent while
  it is written: the download starts at once instead of after the whole
  file was created, and Crawlabout no longer needs memory for the whole
  file.
• Large websites (100,000 URLs and more) no longer bring the browser to a
  halt. The page used to load every result several times a second while a
  check ran; now the server filters, sorts and counts, and the page only
  loads what it shows. During a check it gets updates pushed instead of
  asking for them.
• The results list is shown in pages of 250 entries, with buttons to
  page through it. Filters, sorting and the content search work on all
  results, and "Export list" still exports every page of the list.
• Graph tab: with more than 2,500 URLs the graph groups them by directory
  on its own, because the browser cannot draw that many pages one by one
  smoothly. The checkbox "Group by directory" is then on and greyed out;
  its tooltip says why. If all pages are in one directory, the graph shows
  the 2,500 pages closest to the start page, and the legend says how many
  are left out.
• The content search uses all processor cores, and while a check runs it
  only searches the pages added since. The link graph of a large check is
  built about ten times faster, and Crawlabout needs less memory for the
  links between the pages.
• Graph tab: a click on a page shows the pages it links to (one or two
  links away) instead of every page linking to it or from it, plus the
  shortest click path from the start page as before.
• Graph tab: names appear only for the dot under the mouse pointer. Its
  neighbours are highlighted without names; the names of all larger dots,
  groups and neighbours used to cover each other.

Fixed

• The password fields under "Advanced settings" (check and offline copy)
  look like the other input fields again.

Crawlabout v2.1.1

Changed

• Graph tab: the checkboxes have a row of their own below the search and
  colour fields. Options that change nothing for the current results (no
  files, no external links, no navigation links, a single directory) are
  greyed out instead of having no effect; their tooltip says why.
• Graph tab: the legend notes when URLs have more than 50 incoming links –
  their lines come from the first 50 linking pages only, the dot size counts
  all of them.

Fixed

• Crawlabout started from the app menu on Linux could not be stopped. The
  menu entry now starts it in a terminal window: close the window (or press
  Ctrl+C) to quit. Running offline copies stop cleanly and can be continued
  later, and a browser that Crawlabout opened stays open.

Crawlabout v2.1.0

Added

• Link graph: a new Graph tab between Sitemap and Results shows
  the pages of the site as a network – every dot a URL, every line a link.
  Drag to move, scroll or use + / − to zoom, drag a dot to pin it; hovering
  a dot highlights its links. Colours by status or by clicks from the start
  page, files and external links can be shown too, and a search field
  highlights matching pages. The graph grows while the check runs and keeps
  itself in view. Further options:
  • Hide navigation links leaves out links to URLs that at least half of
    the pages link to (menu, footer) – what remains is the linking in the
    content.
  • Group by directory shows one dot per directory (/blog/, /shop/ …),
    the lines add up the links between them; colours by directory too.
  • A click on a dot focuses it: only the page, its neighbours (1 or 2 links
    away) and the shortest click path from the start page stay visible;
    a second click opens its details.
  • Full screen shows the graph over the whole window (Escape closes it).
• Real number of internal links: the details, reports and the CSV
  (new column inlinks) show how many pages link to a URL, also beyond 50
  (before: "50+"). The list of linking pages still shows the first 50.

Crawlabout v2.0.0

Added

• Save offline – edit, export and import the download list: every
  entry that is not running can be edited (its settings are loaded into the
  form; changing the URL or the target directory starts the entry over).
  "Export list" saves all entries with their settings as a JSON file
  (passwords only if you choose "With passwords" – they are stored as
  plain text), "Import list" adds the entries of such a file again
  – for example after a restart of Crawlabout.
• Windows: every release has a single crawlabout-windows-amd64.exe
  (no installer). It is built and tested on Windows, opens the interface in
  the default browser and shows it in the language of Windows.

Changed

• LinkScout is now called Crawlabout – the name LinkScout was already
  used by a similar project. This is why the version jumps to 2.0: scripts
  that call linkscout, links to the old release files and clients of the
  local API (header X-Crawlabout instead of X-LinkScout) need the new
  name; command-line options and the JSON report format are unchanged.
  The command is crawlabout, the packages are crawlabout-*.deb /
  crawlabout-*.pkg.tar.zst (they replace an installed linkscout
  package) and the website moved to
  https://dlehmann.github.io/crawlabout/. Existing offline copies are
  updated as before: their .linkscout folder is renamed, and the default
  target directory stays ~/Downloads/linkscout as long as there is no
  ~/Downloads/crawlabout. LINKSCOUT_PASSWORD and the settings of the
  web interface still work. Reports and download lists from LinkScout can
  be imported.
• New logo: an open "C" – a round trip of linked pages with the next
  one about to be discovered. It is the icon of the desktop launcher, the
  favicon of the app and of the website.

LinkScout v1.5.0

Added

• External services in the overview: which hosts the pages load fonts,
  scripts, images, videos or maps from (Google Fonts, Google Tag Manager,
  YouTube …), grouped by category, with the number of pages – for example
  for the privacy policy. A click on a host lists its pages; the results
  list has a filter "External service", the details of a page list the
  resources. Also in the HTML, PDF, CSV and JSON exports. Only what is in
  the HTML is visible; what scripts load later is not.
• Content search in the results list: search the visible text or the
  HTML source of all checked pages for a text or a regular expression, as
  you type. The list shows only the pages with matches, each with the
  matches in context; the exports include them. Headless: -search, -search-regex,
  -search-source. JSON reports keep the text and HTML of the pages, so an
  imported report can be searched too.
• Export the list: a menu above the results list exports exactly the
  listed URLs in their order – CSV and JSON with all data (the JSON can be
  imported again), a detailed PDF with the key figures and details of
  these URLs, or a summary PDF with one entry per URL. The files name the
  active filters, the sort order and the content search.
• Save offline – update and continue: downloading into a directory that
  already holds a copy only fetches what changed (the server is asked with
  ETag / Last-Modified) and removes the files that are no longer on the
  site; files that fail temporarily keep their previous version. A stopped
  download continues where it ended. The list buttons say "Update" and
  "Continue"; "Download everything again" (-mirror-fresh) starts over.
• Save offline – what belongs to the copy: files linked on other
  websites (PDFs, images, archives; -mirror-external-files), the pages of
  sitemap.xml that no link leads to (below the start directory), and files that JavaScript refers to
  ("/img/slide-2.jpg") are downloaded too; robots.txt can be respected
  (off by default, -mirror-robots), blocked URLs are listed; rules include or
  exclude URLs (-/private/* -*.zip +cdn.example.com/*, -mirror-rules).
• Limits: requests per second per server (-rate, also for checks),
  bandwidth (-bandwidth) and maximum file size (-max-file-size) of
  offline copies, also in the advanced settings of the form.

Changed

• JSON reports contain the text and HTML of the checked pages (for the
  content search after an import). An imported list export keeps the SEO
  issues found on the whole site until it is checked again.
• German texts address the user consistently with "du", including the
  messages of required and invalid form fields, which came from the
  browser before.

Fixed

• Save offline: stopping a download no longer lists the URLs that were
  being loaded at that moment as errors.

LinkScout v1.4.0

Added

• Packages for Debian/Ubuntu (.deb) and Arch Linux (.pkg.tar.zst),
  for x86_64 and arm64, attached to every release; they install the program,
  the app-menu entry and the icon.
• The commands on the project page have a Copy button.
• Save offline: pages that answer with an error status (404, 410, 500 …)
  are saved anyway when the server sent content, so custom error pages and
  sites that answer every page with an error work in the copy. Entries of the
  download list with such URLs are marked; a click expands the list of these
  URLs with status or error, whether they were saved and the page that links
  to them. -mirror prints the same list.

Changed

• Save offline: host names with umlauts or other non-ASCII characters
  (müller.de) are converted to Punycode (xn--mller-kva.de) right in the
  URL field, and the <host> / <domain> placeholders of the target
  directory use that form too. URLs in the download list are shown readably
  (/über-uns instead of /%C3%BCber-uns).

Fixed

• Save offline: on phones the entries of the download list ran past the
  right edge of the card.

LinkScout v1.3.0

Added

• Sitemap tab: the site structure – every internal HTML page with status
  200 as a tree by its URL path, with title, last modification date and the
  number of pages below each folder; a click on a page opens its details.
  Exports: sitemap.xml for Google Search Console (with lastmod;
  pages with noindex, another canonical URL or blocked by robots.txt are
  marked in the tree and left out; every URL uses the scheme and host the
  start page answers under, so www/non-www and http/https mixes do not
  break the Search Console) and the tree as HTML (nested list)
• Last modified: the Last-Modified header of every URL in the results
  list (column and sort order), in the details, in the HTML and PDF reports
  and in the JSON and CSV exports

Changed

• The HTML and PDF reports are much more detailed: they contain every
  figure and chart of the overview (status, SEO health, length of titles and
  descriptions, response times, clicks from the start page, status codes,
  MIME types, size by MIME type) and, for every URL, what its detail view
  shows – redirect chain, linking pages, SEO data and every SEO issue with
  its explanation and what exactly was found. The separate list of SEO
  issues is gone. Lists are never shortened. The PDF is A4 portrait now and
  its URLs are clickable. The figures follow the
  overview's "include external links" setting
• The CSV export contains all data of a URL: redirect chain, MIME type
  and content type, check time, linking pages, worst issue and all SEO
  facts (titles, descriptions, robots, canonicals, headings, hreflang,
  JSON-LD, images without alt text, mixed content, security headers …).
  Several values in one cell are separated by line breaks
• The JSON export additionally contains the figures of the overview
  (overview)
• Export menu: "JSON export" and "CSV export" instead of "JSON report" and
  "CSV table"

LinkScout v1.2.0

Added

• PDF export of the report (A4 landscape: key figures, SEO issues, all
  URLs), in the export menu and with -report report.pdf
• SEO charts in the overview: SEO health of the pages (most severe issue
  per page) and the length of titles and meta descriptions with the
  recommended range; every part opens the filtered results list. New filters
  for the worst issue, title length and description length
• The details of a URL explain every SEO issue: severity, what the check
  means and what exactly was found (lengths, the pages with the same title,
  the images without alt text, the missing headers, the redirect chain …)

Changed

• Results list: two-line entries – the URL gets the full width of the first
  line; the second line shows status and title on the left and MIME type,
  kind, clicks, size, time and SEO issues on the right, in columns that line
  up across all URLs (an empty cell where a value is missing, numbers
  right-aligned). Sorting moved to a "Sort by" selection
• Filters have labels, their first option is simply "All"; a filter that is
  set is highlighted
• The SEO tab is gone: its charts moved to the overview, the issues are
  shown per URL in the results list (column, filter, details)
• Sub-tab order: overview first, results list last

LinkScout v1.1.0

Added

• SEO checks in a new SEO tab: 48 checks grouped into errors, warnings
  and notices. A click on a check opens the results list with the affected
  URLs; the result details show the SEO data of a page.
  • Indexability: noindex (meta robots and X-Robots-Tag), missing,
    multiple, foreign or broken canonicals, URLs blocked by robots.txt
  • Titles and meta descriptions: missing, multiple, duplicate, too long or
    too short; H1 missing, multiple or duplicate, skipped heading levels
  • Redirect chains and loops, internal links to redirects, broken
    #fragment anchors, mixed content (http resources on https pages)
  • Images without alt text, images larger than 300 KiB
  • Pages more than 3 clicks deep, pages linked only once
  • hreflang: invalid codes, broken targets, missing return links, conflicts
    with canonicals
  • Invalid JSON-LD, missing viewport, missing lang, missing OpenGraph /
    Twitter preview, exact duplicate content
  • Missing security headers and certificates that expire within 30 days
  • Long URLs, uppercase letters, underscores and parameters in URLs
• sitemap.xml: after the link crawl the URLs of the sitemap(s) (from
  robots.txt or /sitemap.xml, including sitemap indexes and .gz) are
  checked too. Pages that are only listed there show up as not linked
  (orphan pages). Option in the advanced settings, -no-sitemap on the
  command line
• robots.txt is read and blocked URLs are marked (they are still checked –
  LinkScout is the site owner's tool)
• Compare the results with an earlier JSON report: new and removed URLs,
  changed status codes, new and fixed SEO issues. In the UI (Compare …) and
  with -compare old.json on the command line
• Export as CSV and HTML in addition to JSON. On the command line the
  format follows the file extension of -report
• Results list: column and filter for SEO issues; the click depth filter and
  chart know not linked URLs

Changed

• The first tab is now called Check website (Website prüfen), because it
  checks more than links
• Redirect loops are reported as such instead of "too many redirects"

LinkScout v1.0.0

Full Changelog: https://github.com/dlehmann/linkscout/commits/v1.0.0