Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/Add support for loading webarchive files from manifest #105

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

ianrumac
Copy link

@ianrumac ianrumac commented May 6, 2024

What

  • Adds manifest : WebArchiveManifest field to Paywall to enable saving paywall as archive
  • Adds WebArchiveLibrary and ArchiveCompressor to manage and de/compress MHTML files given a manifest.
  • Adds ManifestDownloader to download manifest as File's
  • Adds support for non-serializable fetching via current Network layer
  • Adds support for non-serializable writing to current Storage layer
  • Adds a small DSL for prettier tests with nice crash output
  • Refactors WebViewClient into a DefaultWebViewClient and an ArchiveWebClient relying on androix.webkit.WebViewAssetLoader to load local files

How

Downloading

  • When a configuration is fetched, it checks for paywalls with manifests and queues them up for a download via WebArchiveLibrary. The library checks if a Manifest is downloaded already or is in Queue, then either skips it or downloads it if necessary.
  • The declared files and their necessary reference are downloaded in parallel and compressed into the archive itself.
  • Once a paywall and all it's related files (be it from manifest or referenced from the body itself) are downloaded,
  • ArchiveCompressor compresses them into an MHTML file and stores them to disk

Viewing

  • When a paywall is loaded, depending on the Manifest usage flag it either checks if manifest exists and loads URL if not, or awaits for the manifest to download before loading
  • The manifest is decompressed into a set of files and all webview requests are routed to those files through a mock host URL, done by WebArchiveClient overriding the URL requests

Extra

Adds a small Given/When/Then testing DSL to make reading/understanding tests easier

Adds WebArchiveLibrary and ArchiveCompressor to manage and de/compress MHTML files given an Archive Manifest.
The declared files and their necessary reference are also recursively downloaded and compressed into the archive itself.
When loading from archive, uses androidx.webkit.WebViewAssetLoader to route requests towards the appropriate archive file.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant