Skip to content
David Foster edited this page Aug 26, 2024 · 133 revisions

For smaller items, please see the open issue list.

v2.0 Final - Welcome early adopters & developer-contributors: Good documentation. Stable UI.

Audience assumptions for v2.0:

  • Knows what a "URL" is. Can be taught to copy from browser's address bar.

Entice to install:

  • [▶︎] README: Add intro/sizzle video showing how to download various sites (#140)
    • Draft script:
      • Download a web page
      • Download a simple web site
      • Download a complex web site
    • Consider remove/minimize jargon "root" and "URL" from various places, like "New Root URL".
      • Change "New Root URL" → "New Root Page" or "New Page"
      • Change icon from (a bare ⚓) → (a 📄 with an ⚓ superimposed).
        • Pro: Aligns with icon for a regular "page" (📄)
        • Pro: Looks like an "anchored page"
        • Con: Cannot use Unicode icon to represent on macOS/Linux inside the [New Group > Source] dropdown.
      • ✖️ Change shortcut ⌘R (for "New Root URL") → «something else»
        • Maybe not. Not a lot of alternatives that are easy to type, especially with the left hand. ⌘D, ⌘E
    • Alter quickstart to use a mirrored copy of {xkcd, The Pragmatic Engineer} as an example, rather than the real site.
      • Does reduce bandwidth load on real site.
      • Does prevent instructions from breaking.
      • But does require ability to easily export sites in format amenable to hosting on GitHub Pages.
  • README: Provide more info about how Crystal's downloaded sites are in a "file format suitable for archival". Suggest expand on the existing bullet with that text in the current README. (NEEDS ISSUE)
    • Document .crystalproj package format, and anything else that a digital archivist would care about (NEEDS ISSUE)

Streamline the "download web page" and "download simple web site" experiences, in the intro/sizzle video:

  • When the “New Root URL” dialog appears, introduce the "Download URL Immediately" option, checked by default, to eliminate an extra click to start downloading the new URL. ⏳
  • When the “New Root URL” dialog appears, introduce the "Create Group to Download Entire Site" option, unchecked by default, to eliminate manual creation (and download) of a group covering the entire site. ⏳
  • Closing/reopening a project restores any (download) tasks that were in progress. ⏳
    • AKA: Support resume of (download) tasks when project reopened (#115)
  • A top-level (download) task can be cancelled manually, now that they cannot be automatically cancelled by closing a project. ⏳

Legal:

  • License: Add license that allows others to distribute widely but NOT to sell commercially. Source-Available Shareware. (#16)
    • Probably one of the Polyform licenses.
  • Donations: Provide way for early adopters to contribute to development financially (#147)

v2.1

Streamline the "download complex web site" experience, in the intro/sizzle video:

  • Error pages in the browser are clearly-identifiable as originating from Crystal, at least presenting the Crystal logo somewhere.
  • The "Not in Archive" / "Not Downloaded" error page should have (1) a button to download the individual page ⏳ and (2) either a button to go back to the previous page (if there is a previous page) or a button to close the tab (if there is no previous page)
  • ... (maybe more items, as the script for that part of the video is fleshed out further)

v2.2

Follow-ons to recently introduced features:

  • Do not Download (from v1.8): Persist this option after reopening a project (#191)

High-value features (Choose 1 or 2):

  • Sequential groups (#110)
    • Already have initial UI for this, behind the NewGroupDialog._SHOW_SEQUENTIAL_OPTION feature flag.
  • Ability to search for URLs that did not download at all or did not download successfully (with an error) (#84)
  • Support resume of (download) tasks when project reopened (#115)

Easy to install for early adopters, on Linux:

  • Fix install on Linux to not take 10+ minutes (#166)
  • Add Fedora to CI (#93)

v3.0 - Welcome end users: Great first-run experience. Great documentation.

Branding during README:

  • Branding: Consider changing name from "Crystal Web Archiver" to "Crystal" (#158)
  • Branding: Change PyPI package name from "crystal-web" to "crystal" (#159)

Easy to install for end users (Part 1):

  • Binary install steps work on all platforms:
    • Can open Crystal binary on macOS, without special security bypass (#20)
      • 💰 Requires an Apple Developer ID certificate ($99/year)
    • Can open Crystal binary installer on Windows, without a publisher warning (#144)
      • 💰 Probably requires some kind of paid certificate from Microsoft
  • Easy to install/run on Mac, from disk image:
    • Mac Installation: Streamline process to install Crystal to Applications folder (#156)
  • Easy to install/run on Windows, from installer:
    • Windows Installer: Offer to open Crystal after install. Offer to add desktop shortcut. (#155)

Easy to install for end users (Part 2):

  • Easy to run on Mac, from Mac App Store:
    • Build/submit .app for Mac App Store (with M1 support too) (#42)
      • 💰 Requires an Apple Developer ID certificate ($99/year)
  • Easy to run on Windows, from Windows Store:
    • [#] Investigate localization steps to make Crystal feel more like a native Windows app, if any...
    • Submit .exe to Windows App Store (#181)
      • 💰 Probably requires some kind of paid certificate from Microsoft
  • Easy to run on Linux, from local package managers:
    • Submit .deb to Debian (& Ubuntu) package repository (#90)
    • Submit .rpm to Fedora (& Red Hat) package repository (#91)

Branding during first run:

  • Better start dialog/window (#113)
    • Show Crystal logo and icon in first dialog
  • Main Window: Improve Crystal branding in lower-left corner (#99)
  • Icons:
    • Main Window: Add icon for Crystal that appears in top-left corner, for Windows (#151)
    • Main Window: Add dock icon/title and activity icon/title, for Linux (#152)
    • Main Window: Only show file extension if shown in Finder, for macOS (#160)
    • [#] Main Window: Use same dock icon when run from source as when run from .app or .exe, for macOS and Windows (#153)

Easy to create first project:

  • Support untitled projects (#102)
    • Eliminate requirement to name & save a project early
  • Launch: Long delay opening the Save/Load dialog (#148)

Easy to create first URL:

  • One/both of:
    • Main Window: Add section headers (#97)
    • Main Window: Add empty state for top pane ("URLs and Groups") (#137)
  • Better error messages for common error types (#180)

Easy to create first group:

  • Add Group: Warn when trying to create group using pattern with no wildcards (#77)

Easy to serve first URL:

  • View: Long delay opening web browser (#149)
  • Log Drawer: Animate open/close (#134)
    • Strengthen mental association between pressing View and the Log Drawer that appears

Easy/possible to export:

  • Support export of project to filesystem (NEEDS ISSUE)
    • A user asked about this in #119
  • Support export of project to static file server (like GitHub Pages) (#41)

Easy to quit:

  • Respond to Quit event from OS correctly (#150)

Quality:

  • Automated test coverage of core workflows:
    • Download Site Requiring Login:
      • test_can_download_and_serve_a_site_requiring_cookie_authentication (related to #68)
    • ... (probably more core workflows missing test coverage)

vFuture+0 Open Source - Other features

Any of the following items individually are major enough to warrant a release on their own (releaseworthy):

  • Support dynamic download of site entirely through browser (#69)
  • Support rewriting URLs on the frontend in JavaScript, which is necessary to archive more-modern sites that load many resources using JavaScript that cannot be fully statically analyzed to extract URLs in advance. 🧠 (#8)

vFuture+0 Commercial - Export/distribution of downloaded sites

...as an advanced contribution-required feature

  • Support export of downloaded site to plain files that can be deployed to a static file server, either to a custom domain (best) or to a subdirectory within an existing domain (worse, because some URLs may not be as straightforward to rewrite). (#41)
    • Does round out final piece of end-to-end preservation lifecycle: capture, retain, & share
    • Does enable removing real xkcd from quickstart instructions.

Separate from the release of Crystal itself:

  • Do use the export feature to export & host various notable sites I've archived that have fallen off the internet:
    • TMFFA -> tmffa.daarchive.net (#60)
    • PRFI -> rakhal.daarchive.net (#61)
    • (??) Apple Support Area -> ???.daarchive.net
      • ⚖️ Has special legal considerations: Would Apple disapprove of distributing the old files from this area?
    • –––
    • xkcd -> xkcd.daarchive.net
      • 🎗 Needs special robots.txt to prevent indexing, because original site still online
    • (??) OtakuWorld + The Big KiSS Page + Jenniverse -> (multiple!)
      • 🎗 Needs special robots.txt to prevent indexing, because original site still online
      • –––
      • -> *.otakuworld.daarchive.net
      • -> jenniverse.daarchive.net
      • -> *.unicornjelly.daarchive.net
      • -> pasteldefender.daarchive.net
      • -> transsexual.daarchive.net
      • -> kamishibai.daarchive.net

vFuture+1 Commerical - Support distributed crawling

...as an advanced contribution-required feature

  • Distributed web crawling: Separate out crawling logic such that a centralized "orchestrator" can direct a remote fleet of "crawlers" to quickly capture a large domain that is scheduled to fall off the public internet soon (ex: Geocities, DeviantArt Portfolios, etc).