-
Notifications
You must be signed in to change notification settings - Fork 5
Roadmap
David Foster edited this page Aug 26, 2024
·
133 revisions
For smaller items, please see the open issue list.
Audience assumptions for v2.0:
- Knows what a "URL" is. Can be taught to copy from browser's address bar.
Entice to install:
- [▶︎] README: Add intro/sizzle video showing how to download various sites (#140)
- Draft script:
- Download a web page
- Download a simple web site
- Download a complex web site
- Consider remove/minimize jargon "root" and "URL" from various places, like "New Root URL".
- Change "New Root URL" → "New Root Page" or "New Page"
- Change icon from (a bare ⚓) → (a 📄 with an ⚓ superimposed).
- Pro: Aligns with icon for a regular "page" (📄)
- Pro: Looks like an "anchored page"
- Con: Cannot use Unicode icon to represent on macOS/Linux inside the [New Group > Source] dropdown.
- ✖️ Change shortcut ⌘R (for "New Root URL") → «something else»
- Maybe not. Not a lot of alternatives that are easy to type, especially with the left hand. ⌘D, ⌘E
- Alter quickstart to use a mirrored copy of {xkcd, The Pragmatic Engineer} as an example, rather than the real site.
- Does reduce bandwidth load on real site.
- Does prevent instructions from breaking.
- But does require ability to easily export sites in format amenable to hosting on GitHub Pages.
- Draft script:
- README: Provide more info about how Crystal's downloaded sites are in a "file format suitable for archival". Suggest expand on the existing bullet with that text in the current README. (NEEDS ISSUE)
- Document .crystalproj package format, and anything else that a digital archivist would care about (NEEDS ISSUE)
Streamline the "download web page" and "download simple web site" experiences, in the intro/sizzle video:
- When the “New Root URL” dialog appears, introduce the "Download URL Immediately" option, checked by default, to eliminate an extra click to start downloading the new URL. ⏳
- When the “New Root URL” dialog appears, introduce the "Create Group to Download Entire Site" option, unchecked by default, to eliminate manual creation (and download) of a group covering the entire site. ⏳
- —
- Closing/reopening a project restores any (download) tasks that were in progress. ⏳
- AKA: Support resume of (download) tasks when project reopened (#115)
- A top-level (download) task can be cancelled manually, now that they cannot be automatically cancelled by closing a project. ⏳
Legal:
- License: Add license that allows others to distribute widely but NOT to sell commercially. Source-Available Shareware. (#16)
- Probably one of the Polyform licenses.
- Donations: Provide way for early adopters to contribute to development financially (#147)
Streamline the "download complex web site" experience, in the intro/sizzle video:
- Error pages in the browser are clearly-identifiable as originating from Crystal, at least presenting the Crystal logo somewhere.
- The "Not in Archive" / "Not Downloaded" error page should have (1) a button to download the individual page ⏳ and (2) either a button to go back to the previous page (if there is a previous page) or a button to close the tab (if there is no previous page)
- ... (maybe more items, as the script for that part of the video is fleshed out further)
Follow-ons to recently introduced features:
- Do not Download (from v1.8): Persist this option after reopening a project (#191)
High-value features (Choose 1 or 2):
- Sequential groups (#110)
- Already have initial UI for this, behind the
NewGroupDialog._SHOW_SEQUENTIAL_OPTIONfeature flag.
- Already have initial UI for this, behind the
- Ability to search for URLs that did not download at all or did not download successfully (with an error) (#84)
- Support resume of (download) tasks when project reopened (#115)
Easy to install for early adopters, on Linux:
Branding during README:
- Branding: Consider changing name from "Crystal Web Archiver" to "Crystal" (#158)
- Branding: Change PyPI package name from "crystal-web" to "crystal" (#159)
Easy to install for end users (Part 1):
- Binary install steps work on all platforms:
- —
- Easy to install/run on Mac, from disk image:
- Mac Installation: Streamline process to install Crystal to Applications folder (#156)
- Easy to install/run on Windows, from installer:
- Windows Installer: Offer to open Crystal after install. Offer to add desktop shortcut. (#155)
Easy to install for end users (Part 2):
- Easy to run on Mac, from Mac App Store:
- Build/submit .app for Mac App Store (with M1 support too) (#42)
- 💰 Requires an Apple Developer ID certificate ($99/year)
- Build/submit .app for Mac App Store (with M1 support too) (#42)
- Easy to run on Windows, from Windows Store:
- [#] Investigate localization steps to make Crystal feel more like a native Windows app, if any...
- Submit .exe to Windows App Store (#181)
- 💰 Probably requires some kind of paid certificate from Microsoft
- Easy to run on Linux, from local package managers:
Branding during first run:
- Better start dialog/window (#113)
- Show Crystal logo and icon in first dialog
- Main Window: Improve Crystal branding in lower-left corner (#99)
- Icons:
- Main Window: Add icon for Crystal that appears in top-left corner, for Windows (#151)
- Main Window: Add dock icon/title and activity icon/title, for Linux (#152)
- Main Window: Only show file extension if shown in Finder, for macOS (#160)
- [#] Main Window: Use same dock icon when run from source as when run from .app or .exe, for macOS and Windows (#153)
Easy to create first project:
- Support untitled projects (#102)
- Eliminate requirement to name & save a project early
- Launch: Long delay opening the Save/Load dialog (#148)
Easy to create first URL:
- One/both of:
- Better error messages for common error types (#180)
Easy to create first group:
- Add Group: Warn when trying to create group using pattern with no wildcards (#77)
Easy to serve first URL:
- View: Long delay opening web browser (#149)
- Log Drawer: Animate open/close (#134)
- Strengthen mental association between pressing View and the Log Drawer that appears
Easy/possible to export:
- Support export of project to filesystem (NEEDS ISSUE)
- A user asked about this in #119
- Support export of project to static file server (like GitHub Pages) (#41)
Easy to quit:
- Respond to Quit event from OS correctly (#150)
Quality:
- Automated test coverage of core workflows:
- Download Site Requiring Login:
- test_can_download_and_serve_a_site_requiring_cookie_authentication (related to #68)
- ... (probably more core workflows missing test coverage)
- Download Site Requiring Login:
Any of the following items individually are major enough to warrant a release on their own (releaseworthy):
- Support dynamic download of site entirely through browser (#69)
- Support rewriting URLs on the frontend in JavaScript, which is necessary to archive more-modern sites that load many resources using JavaScript that cannot be fully statically analyzed to extract URLs in advance. 🧠 (#8)
...as an advanced contribution-required feature
- Support export of downloaded site to plain files that can be deployed to a static file server, either to a custom domain (best) or to a subdirectory within an existing domain (worse, because some URLs may not be as straightforward to rewrite). (#41)
- Does round out final piece of end-to-end preservation lifecycle: capture, retain, & share
- Does enable removing real xkcd from quickstart instructions.
Separate from the release of Crystal itself:
- Do use the export feature to export & host various notable sites I've archived that have fallen off the internet:
- TMFFA -> tmffa.daarchive.net (#60)
- PRFI -> rakhal.daarchive.net (#61)
- (??) Apple Support Area -> ???.daarchive.net
- ⚖️ Has special legal considerations: Would Apple disapprove of distributing the old files from this area?
- –––
- xkcd -> xkcd.daarchive.net
- 🎗 Needs special robots.txt to prevent indexing, because original site still online
- (??) OtakuWorld + The Big KiSS Page + Jenniverse -> (multiple!)
- 🎗 Needs special robots.txt to prevent indexing, because original site still online
- –––
- -> *.otakuworld.daarchive.net
- -> jenniverse.daarchive.net
- -> *.unicornjelly.daarchive.net
- -> pasteldefender.daarchive.net
- -> transsexual.daarchive.net
- -> kamishibai.daarchive.net
...as an advanced contribution-required feature
- Distributed web crawling: Separate out crawling logic such that a centralized "orchestrator" can direct a remote fleet of "crawlers" to quickly capture a large domain that is scheduled to fall off the public internet soon (ex: Geocities, DeviantArt Portfolios, etc).