Gleaner · 拾穗

Collect full-resolution images, video with sound, and audio from any web page.
Zero telemetry. Apart from the optional cloud assistant and voice input, everything runs on your own machine.

English · 简体中文

Install · Features · How it works · Testing · Development · Contributing

The name comes from Millet's painting The Gleaners — picking up, one by one, the originals scattered across a web page.

Interface language: the extension UI is currently Simplified Chinese only.

Scan — every image is labelled with the resolution of the original file; video and audio are listed separately.	Download — pause / resume / cancel, per-file status, and a history with "reveal in folder".
Pick by description — "all the green images" is matched locally from pixel colours; nothing is uploaded.	About the screenshots: they are captured automatically by npm run docs:shots, which loads the real extension into a real Chrome. The demo page uses CC0 photos from Wikimedia Commons (authors listed in docs/demo/CREDITS.md); its short video was generated from one of those photos with ffmpeg.

• 🖼 Original-quality images. Built-in site rules rewrite thumbnail URLs to the original file (popular e-commerce, social and image-hosting CDNs, plus common OSS / Qiniu / COS processing parameters). The label on each card is the resolution of the original: after scanning, Gleaner fetches only the first bytes of each file (an HTTP Range request, usually under 128 KB) and reads the real dimensions from the header.
• 🔍 Thorough scanning. <img>, srcset (parsed per the HTML spec), <picture>, CSS backgrounds, links to images and ten common lazy-load attributes; the page is auto-scrolled first to trigger lazy loading. Multiple sizes of the same image produce a single card.
• 🎬 Video and audio. Media is captured three ways: network sniffing, a read-only page hook, and a DOM scan. When a site streams picture and sound as separate tracks (DASH), Gleaner pairs them and merges them locally into one MP4 with sound using a bundled ffmpeg.wasm. Where the player exposes its track list, Gleaner picks the most compatible codec (H.264) rather than the AV1 track the player chose — AV1 files won't open in QuickTime on M1/M2 Macs. If the primary CDN fails it falls back to backup CDNs, and it estimates file size from bitrate × duration to pick the highest quality that fits.
• ⏯ Works like Chrome's download manager. Real pause (the network request is aborted, bytes already received are kept), resume via HTTP Range (verified against a real CDN: the resumed file is byte-for-byte the same size as an uninterrupted download), and cancel. Downloads keep running when the popup is closed; the toolbar badge shows progress; starting another download adds to the list instead of replacing the running one; each batch gets its own history entry with one-click "reveal in Finder / Explorer".
• 🛡 Hotlink protection. A per-host Referer is added through declarativeNetRequest; the rules only match requests that don't belong to any tab (mainly the extension's own fetches), so the pages you browse are untouched. Blocked direct downloads are retried through the extension's own fetch channel, and if an upgraded original still can't be fetched, Gleaner falls back to the image that was actually shown on the page.
• 🤖 Pick by description. "All the green images", "videos in the comments", "the 3 largest images" are resolved locally. Requests that need to understand content ("photos with a cat") can optionally go to Claude — off by default, bring your own API key. Voice input is supported.
• 📦 Batch downloads. Adjustable concurrency, a folder per page title, or a single ZIP (entry extensions come from the file signature, and HTML error pages are never packed as images). File names are sanitised with rules verified against real Chrome, so emoji, zero-width characters, ~ and trailing dots never make a download fail.
• 🔒 Zero telemetry. No analytics, tracking or remote config. The extension only makes two kinds of requests: fetching the media itself (downloads, size probes, re-fetching thumbnails to read colours for local matching), and the cloud assistant if you turn it on.

1. Download the latest gleaner-*-chrome.zip from Releases and unzip it to a folder you won't delete.
2. Open chrome://extensions and switch on Developer mode (top right). Other Chromium browsers such as Edge should work the same way via edge://extensions, but haven't been tested individually.
3. Click Load unpacked and choose the unzipped folder.
4. Pin the Gleaner (wheat) icon to the toolbar. On any page, click it → 扫描当前页面 (Scan this page).

For streaming video, play it for a few seconds first, then open Gleaner and click 刷新 (Refresh).

To update, overwrite the same folder and click ↻ on Gleaner's card in chrome://extensions.

From source: npm ci && npm run build (Node ≥ 22.12), then choose .output/chrome-mv3 in step 3.

flowchart LR
  subgraph Page
    P[Web page] -->|fetch / XHR hook| H[injected.content.ts<br/>MAIN world]
    P -->|DOM| C[content.ts<br/>scanner]
    H -->|postMessage<br/>media URLs · track list| C
  end
  subgraph Extension
    W[background.ts<br/>jobs · sniffing · DNR]
    O[offscreen<br/>fetch · Range resume · ffmpeg merge]
    U[popup<br/>React UI]
  end
  P -.->|webRequest sniffing| W
  C -->|scan results / media| U
  U -->|download · pause · cancel| W
  W -->|direct| D[(chrome.downloads)]
  W -->|hotlink-protected / DASH| O
  O -->|blob URL| D

This project has been burned by "all tests pass, real use breaks", so verification is layered. Requirements: Node ≥ 22.12; the real-browser commands need Chrome installed (the macOS path is used by default; elsewhere set CHROME=<path to chrome>).

Every bug fix starts with a test that reproduces it, and the fix is then temporarily reverted to confirm the test fails — so tests can't quietly "pass" without checking anything.

npm install
npm run dev          # opens Chrome with the extension (persistent profile)
npm run build        # output in .output/chrome-mv3
npm run zip          # build a zip
npm test             # unit + integration
npm run e2e          # end-to-end in real Chrome
npm run docs:shots   # re-capture the README screenshots and banner

Requires Node ≥ 22.12. After npm install, scripts/prepare-public.mjs runs automatically: it copies ffmpeg.wasm from @ffmpeg/core and generates the license files bundled into the extension. Release packages are built by GitHub Actions in a clean environment (pushing a v* tag publishes a Release automatically). A local npm run zip:firefox puts every non-hidden file in the working tree into sources.zip — don't publish from a directory that contains anything else.

src/
├── core/
│   ├── rules.ts        ← site rules for original images (PRs welcome — see CONTRIBUTING.md)
│   ├── scanner.ts      ← DOM scanner (srcset / picture / lazy loading / backgrounds)
│   ├── media.ts        ← media detection, DASH track pairing, cross-CDN dedupe, HLS segment filtering
│   ├── bili.ts         ← DASH track-list selection: prefer H.264, pair audio, fit within the size limit
│   ├── imagesize.ts    ← read dimensions from file headers (JPEG / PNG / GIF / WebP / AVIF / HEIC / BMP)
│   ├── fmp4.ts         ← hev1 → hvc1 byte patch
│   ├── ffmpeg-mux.ts   ← merge video + audio tracks with ffmpeg.wasm
│   ├── filename.ts     ← file-name sanitiser (rules verified in real Chrome)
│   ├── zip.ts / queue.ts / page-hooks.ts / page-support.ts
│   └── agent/          ← the assistant (pure logic, fully unit-tested; claude.ts is the only file that talks to the network)
├── public/
│   ├── ffmpeg/         ← copied from @ffmpeg/core on install (not in git; MV3 forbids remote code)
│   ├── licenses/       ← generated on install: full license texts bundled into the extension
│   └── icon/
└── entrypoints/
    ├── background.ts   ← jobs, reconciliation, sniffing, DNR, original-size probing
    ├── offscreen/      ← fetching (Range resume, backup CDNs), merging, blob creation
    ├── content.ts / injected.content.ts
    ├── popup/          ← React UI
    └── mic/            ← microphone permission page
tests/                  ← node:test (real API response shapes with fake signatures, generated images, ffmpeg-generated media)
e2e/                    ← real-Chrome end-to-end, live video-site check, file-name probe
licenses/third-party/   ← license texts of every ffmpeg.wasm component

• Firefox runs on MV2 without declarativeNetRequest or offscreen documents, so the Referer rules, DASH video merging and hotlink retries are unavailable.
• m3u8 / HLS streams aren't supported yet: they're marked "m3u8·暂不支持" and can't be selected; live .ts segments aren't listed individually.
• 600 MB limit for whole-file fetches (peak memory for fetching + merging is roughly 3–4× the file size). Gleaner picks the highest quality that fits and says so in the list when nothing fits.
• Live testing so far covered logged-out viewing (480P); higher qualities go through the same code path but haven't each been tested live. Backup-CDN failover on mainland-China networks has only been verified in a simulated environment.
• Only the top-level document is scanned; images inside iframes aren't handled yet.

• v0.1 Batch image downloads + site rule system
• v0.2 Download manager semantics, direct media, DASH merging, the assistant, real pause & resume, original-size probing
• v0.3 HLS (parse m3u8, fetch segments, mux into MP4)
• v0.4 Floating button, context menu, drag-to-download

• No DRM decryption. There is no decryption capability and PRs adding one won't be accepted. Encrypted content won't play even if downloaded.
• No CAPTCHA solving, no bypassing login walls.
• Use it for personal backup and fair use only. Downloaded content belongs to its creators; obtain permission before any commercial use and respect each site's terms of service.

All site names and trademarks mentioned belong to their respective owners. This project is not affiliated with, endorsed or sponsored by any of them; they are mentioned only to describe compatibility.

The code is released under the MIT license. Release packages also include third-party components such as FFmpeg (ffmpeg.wasm, GPL-2.0-or-later); see THIRD_PARTY_NOTICES.md for licenses and where to obtain the source.