crawler plugin

Writing a folder crawler

A crawler answers the question a resolver cannot: what lies behind this address? A cloud folder, a directory share or a release page is one address and many files. crawl returns the list, and everything in it then goes through the same review, validation and routing a pasted link does — each file usually resolved afterwards by a resolver of the same provider.

Scaffold	rdownloader plugin new --type crawler
plugin_type	crawler
World	crawler-plugin
Exports	claims-url, crawl
Template	sdk/templates/crawler
Contract	Interface crawler

What the host promises

• You are asked before you are handed an address. claims-url runs first and reaches nothing at all, so a plugin that has no business with an address never fetches it. A crawler that claims addresses on hosts it does not declare fails conformance.
• You never see a credential. The account's stored secret reaches a request only as {{secret:<reference>}}, which the host expands on the way out — and only towards the hosts that reference may be sent to.
• You propose, the application decides. Nothing this plugin returns is queued by itself, and the host caps how many links it takes from one answer.

Permissions

The scaffold asks for net_http with the provider's API. A crawler may also ask for cookies, captcha, secrets for an account's slot, and key_derivation for a share whose key has to be computed from a credential it never sees.

[extension] claims names the provider slug whose account the crawler runs for, so the host knows which account's credential {{secret:<reference>}} may expand to. Leave it empty for a crawler that needs no account; a public folder share is one. A crawl makes several requests in one invocation, so its [limits] are larger than a resolver's — raise them together with the walk's own limits, never instead of them.

Ask for exactly what you use: the install preview shows every grant, and a component importing an interface its manifest does not grant cannot be instantiated. Every capability is listed under Capabilities.

Rules worth knowing first

Two rules matter most: the walk bounds itself rather than leaning on the fuel budget, which stops a runaway but does not bound a correct crawl; and an empty or unreachable folder is a failure with a stable code — never an empty list, which would create a package with nothing in it.

Step by step

1. Scaffold it

plugin new reads its templates from sdk/templates, so run it from a checkout of the repository:

git clone https://github.com/degoya/rDownloader.git
cd rDownloader
rdownloader plugin new --type crawler --out ../myfolders
cd ../myfolders

Without an installation of rDownloader, the packaging tool rd-pack — cargo build --profile release-test -p rd-pack in the checkout — takes the same arguments: target/release-test/rd-pack plugin new --type crawler --out ../myfolders.

The directory name becomes the crate name and the plugin's slug (--name sets another). The scaffold gets its own plugin id, its own copy of the contract in wit/rdownloader.wit and its own Ed25519 key pair: plugin-signing.key, whose public half is already in manifest.toml. Keep that file out of version control.

2. Make it yours

• src/target.rs decides which addresses are yours — claim narrowly.
• src/listing.rs reads one listing of your provider.
• src/walk.rs is the bounded breadth-first walk: depth, breadth and cycles.
• src/guest.rs holds claims-url and crawl.
• manifest.toml: [capabilities.net_http] domains, [extension] claims, [limits].
• locales/en.json: the English text of every failure code you report.

Change description and author under [metadata], and read the comment at the top of src/lib.rs: it states what the host promises this type.

3. Test it

cargo test

The unit tests need no WebAssembly toolchain: only src/guest.rs is wasm32-specific. Give every answer of your provider you read a test of its own.

4. Build the component

cargo build --release --target wasm32-unknown-unknown
wasm-tools component new target/wasm32-unknown-unknown/release/myfolders.wasm -o target/myfolders.wasm

This needs the wasm32-unknown-unknown target and wasm-tools 1.261.0, the version the bundled plugins are made with: cargo build makes a core module that carries the world src/guest.rs generates, and wasm-tools component new turns it into the component. The plugin is built without WASI: a component importing anything outside rdownloader:plugin is refused — see Building without WASI.

5. Package and sign it

rdownloader plugin package --manifest manifest.toml \
  --component target/myfolders.wasm \
  --locales locales --key plugin-signing.key --output myfolders.rdplug
rdownloader plugin conformance myfolders.rdplug --json

conformance answers whether this application would run the package; it has to be clean before you publish. For a key that outlives the scaffold, rdownloader plugin keygen --role plugin --output ./keys writes a pair: the Base64 in the .pub file goes into public_key, and key_id gets a name of your own. In CI, RDOWNLOADER_PLUGIN_SIGNING_KEY holding the private key's PEM text replaces --key. Details: Packaging, signing and distribution.

6. Install and trust it

For a first try without signing, package with --development instead of --key and start rdownloader serve --plugin-development-mode, the only mode that accepts an unsigned package.

Install a signed package under Settings → Plugins → Add as an upload, or with rdownloader plugin install myfolders.rdplug --root data/plugins. Review before installing shows every permission and the signing key; a key this installation has not seen asks to be trusted — compare its fingerprint. The message after installing says whether the plugin runs at once or after the next restart; see Installing and trust.

Paste a folder address into the LinkGrabber: the files behind it arrive as candidates.

7. Publish it

Publish the .rdplug together with your key's fingerprint, or — so that people get your new versions in the Catalogue — in a plugin repository of your own: a signed index built with rdownloader plugin index build and a repository-role key. A repository only delivers: every package in it still needs your plugin key, approved by the person installing it. See A repository index and Running a plugin repository.

The SDK's reusable CI workflow, sdk/ci/plugin.yml, builds, checks the imports, packages and runs conformance on every push and pull request; it signs only on a push to the default branch or a tag, so a pull request never sees the key.

Read next

• Folder crawlers in the plugin reference — what the host does with each call, in full
• Interface crawler — every function and type, generated from the contract
• Developing a plugin — the tools, translations, errors and what every type shares
• The crawler template on GitHub