object storage

Object storage

rDownloader downloads from three kinds of object storage and uploads finished packages to them, without an external client.

• Amazon S3 and S3-compatible services — MinIO, Ceph, Garage, Wasabi, Backblaze B2, Cloudflare R2 and others.
• Azure Blob Storage.
• Google Cloud Storage.

Profiles are managed under Settings → FTP, SFTP, WebDAV & S3 → S3, in the card Object storage.

Profiles

A link names a bucket (on Azure a container) and an object, never a host. Where it lives and how to sign in comes from a profile. The form asks for the Service first and then shows only the fields that service uses:

Field	Service	What it holds
Service	all	Amazon S3 and S3-compatible services, Azure Blob Storage or Google Cloud Storage
Name	all	Your name for the profile
Storage account	Azure	The account's name: 3 to 24 lowercase letters and digits
Endpoint	all	Empty for the service's own address — AWS S3 in the region below, <account>.blob.core.windows.net, storage.googleapis.com. An address of its own for an S3-compatible service, for Azurite or another Azure cloud, or for a Google emulator. http:// only for a local endpoint typed that way
Region	S3	Required for AWS; many S3-compatible services accept any value or us-east-1
Bucket / Container	all	Optional. Links to this bucket or container use this profile; left empty, the profile serves any of them (on Azure: any container of the account)
Addressing	S3	Automatic (virtual-host style for AWS, path style with a custom endpoint), Path style or Virtual-host style
Credentials	all	How requests are signed — see below
Send checksums with uploads	S3	Every uploaded part carries a SHA-256 checksum the service verifies. Switch it off only for a compatible service that refuses it

Test asks the service whether it answers and accepts the profile's credentials. A profile can be switched off without deleting it.

Bucket and container names follow each service's own rules and are checked when you save. TLS certificates are always validated, against the system's roots plus the custom CA under Settings → Network → Proxies & TLS; the global proxy applies as it does for HTTP.

Credentials

Every secret below is stored encrypted and never shown again; leave the field empty when editing to keep what is stored. A key that does not have the right shape is refused when you save. A profile that needs a secret and has none is marked No secret stored and cannot sign a request yet.

A stored secret stays with the address it was entered for. When you change a profile's Endpoint — or, on Azure, its storage account — the form says that the stored secret is not sent to the new address, and you enter it again before saving.

rDownloader follows no redirect from an object storage endpoint. A service that answers with one fails with The object storage answered with a redirect — enter the address it redirects to as the profile's Endpoint instead.

This machine's environment together with an Endpoint of your own needs one more switch, Send this machine's credentials to the endpoint: the machine's instance role, managed identity or Google token then goes to that address. Turn it on only for a service you run yourself or trust; without it the profile is not saved and is not used.

Amazon S3 and S3-compatible services

• Stored keys — an access key ID and a secret access key, optionally with a session token for temporary credentials.
• This machine's environment — the credentials of the machine rDownloader runs on: environment variables, web identity, the container (ECS/EKS) or EC2 instance metadata. Nothing is stored in rDownloader.
• Anonymous (public bucket) — unsigned requests, for publicly readable buckets only.

Azure Blob Storage

• Stored keys — the storage account's access key.
• Shared access signature (SAS) — a SAS token or the whole SAS URL. When it expires, enter a new one on the profile; downloads already started continue where they stopped.
• This machine's environment — the identity of the machine rDownloader runs on: a service principal (AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID), a workload identity or a managed identity. Nothing is stored in rDownloader.
• Anonymous (public bucket) — unsigned requests, for publicly readable containers only.

Google Cloud Storage

• Stored keys — the content of a service account's JSON key file. Requests always go to the profile's endpoint, whatever the key file names.
• This machine's environment — the machine's application default credentials (GOOGLE_APPLICATION_CREDENTIALS or gcloud auth application-default login) or its metadata server. Nothing is stored in rDownloader.
• Anonymous (public bucket) — unsigned requests, for publicly readable buckets only.

Downloading

Paste a link into the LinkGrabber. The three services take the same forms, each with its own scheme:

Link	What it does
s3://bucket/key, az://container/blob, gs://bucket/object	Downloads one object
s3://bucket/prefix/ (and the same with az://, gs://)	Lists everything under the prefix as a file tree to review, like an FTP directory; a key that turns out to be a prefix is treated the same way
s3://profile@bucket/key (and the same with az://, gs://)	Uses the named profile (by name or id) instead of choosing one

Without a name in the link, the profile of that service bound to the bucket or container is used, otherwise its only enabled profile bound to none. A named profile that is bound to another bucket or container is not used for this one. When no profile or several fit, the link is refused with a message that says which.

Names the service would refuse — an IP address or a port in particular — and keys with empty, . or .. segments are refused before anything is requested.

Resume is guarded by the object's version. A resume asks for the missing range only if the object is still the same. An object that was replaced in between refuses the resume — The object changed on the server; the partial download was kept — instead of appending new bytes to old ones. A change to the object's metadata alone does not count.

Throttling is waited out. When the service throttles the requests or a quota is used up, the download fails as The object storage throttles the requests or the quota is exhausted; it is tried again later.

Uploading finished packages

A profile of any of the three services is also an upload target for post-processing, globally or per category. Under Settings → Post-processing → Scripts & upload, Use an object storage profile fills the target in; written by hand it reads:

object-storage:<profile-id>/<bucket>/<prefix>

On Azure the container takes the bucket's place. A profile bound to a bucket or container takes only that one: name it, or write object-storage:<profile-id> alone for its top level. A target that names a profile that does not exist, or another bucket than the bound one, is refused when you save it. Each package goes into a folder of its own under the prefix.

• The whole package goes up, its folders included — a file in a folder per archive keeps its path, so Film/film.mkv becomes <prefix>/<package>/Film/film.mkv. Symbolic links are not followed.
• Files above 16 MiB go up in parts. Every confirmed part is recorded, so an upload that was stopped or killed continues at the first missing part after a restart.
• A local file that changed since its upload began gets the old upload aborted rather than continued. Uploads nobody continued within a week are aborted when the service starts.
• Nothing local is deleted before the service reports the whole size of the file. Under Move, a folder goes once that has emptied it.
• Uploads keep the upload limit and follow a profile switch while they run.
• A failed upload fails the package.

Switching it off

Object storage belongs to Settings → Services → Remote file transfer, together with FTP, SFTP and WebDAV. Switched off, s3://, az:// and gs:// links are refused at intake and anything queued moves to Blocked.

Related pages

• FTP, SFTP and WebDAV
• Post-processing
• The LinkGrabber
• Security and privacy