HuggingFace compatible API

HuggingFace-Hub-compatible API

🚧 Upcoming feature — not yet in production. Target: next release.

A read-only facade that makes DSpace items consumable by the HuggingFace client ecosystem: huggingface_hub's hf_hub_download/snapshot_download, transformers.from_pretrained, and the hf CLI. Items stored in CLARIN-DSpace (e.g. trained models with their weight files as bitstreams) can be pulled by any of these tools simply by pointing the client at the repository's own API endpoint via the HF_ENDPOINT environment variable — no code changes needed on the client side.

Backend implementation: ClarinHuggingFaceController / ClarinHuggingFaceService in org.dspace.app.rest.hf.

Status: phase 1 — anonymous/public access only. Bearer-token (Personal Access Token) authentication is planned for a later phase.

Usage

Point the HuggingFace client at your repository instead of huggingface.co:

export HF_ENDPOINT=https://<your-host>/server/api/hf

from huggingface_hub import snapshot_download

snapshot_download(repo_id="<handle-prefix>/<handle-suffix>")

The repo_id is simply the item's handle, e.g. 11234/1-12345. Anything that resolves models via huggingface_hub — including transformers.AutoModel.from_pretrained(repo_id) — works the same way, since it goes through the same client library.

Endpoints

All paths below are relative to /server/api/hf (the controller itself is mounted under /api/hf, with {handle} short for {prefix}/{suffix}, e.g. 11234/1-12345).

Method	Path	Description
GET	/api/models/{handle}	Model info at the implicit main revision
GET	/api/models/{handle}/revision/{revision}	Model info at a specific revision
GET	/api/models/{handle}/tree/{revision}	Flat file tree (one entry per ORIGINAL-bundle bitstream)
GET, HEAD	/{handle}/resolve/{revision}/{filename}	Download (GET) or probe (HEAD) a single file
GET	/api/models?search=&limit=	List exposed items ("models"); search matches the title substring (case-insensitive), limit defaults to 20, capped at 100
GET	/api/whoami-v2	Always 401 Invalid user token — placeholder; no Bearer-token support yet

revision accepts either the literal main or the item's current sha: a 40-character hex string returned as sha in the model info and as the X-Repo-Commit header on every response. The sha is a SHA-1 digest computed from the item's UUID, its last-modified timestamp, and the filename→MD5 map of its ORIGINAL bitstreams, so it changes automatically whenever a file or the item's metadata changes. Any other revision string yields a 404 with X-Error-Code: RevisionNotFound. This is what makes client-side caching (huggingface_hub's local cache keys downloads by repo + revision) behave correctly — a stale sha simply misses the cache and re-downloads.

Other error responses follow the same shape: a 404 with X-Error-Code: RepoNotFound when the handle doesn't resolve to an exposed item, and 404/X-Error-Code: EntryNotFound when the requested file doesn't exist in the item's ORIGINAL bundle. All error bodies are {"error": "..."}.

Successful resolve responses (HEAD, streamed GET, and the S3-redirect case) all carry X-Repo-Commit, ETag, X-Linked-Etag, Content-Length, X-Linked-Size, Accept-Ranges: bytes, Content-Type and Last-Modified. GET additionally supports HTTP Range requests for partial downloads.

Configuration

Disabled by default. Set these in local.cfg / clarin-dspace.cfg:

Property	Default	Description
hf.api.enabled	false	Master switch for the whole facade. When false, every endpoint returns 404.
hf.api.filter.metadata-field	dc.type	Metadata field checked against hf.api.filter.value to decide whether an item is exposed.
hf.api.filter.value	machineLearningModel	Value that hf.api.filter.metadata-field must equal for the item to be exposed.
hf.api.exposed.collections	(empty)	Comma-separated list of collection handles whose items are exposed regardless of the metadata filter.

Exposure is fail-closed: an item is only ever returned by the facade if it is archived, not withdrawn, and either (a) its hf.api.filter.metadata-field metadata contains hf.api.filter.value, or (b) its owning collection's handle is listed in hf.api.exposed.collections. If neither the filter nor the collection allow-list is configured, no item is exposed at all (a one-time warning is logged).

Scope & limitations

• Anonymous/public access only in this phase. The resolve (file download) and model-info/tree endpoints run the item's/bitstream's normal DSpace READ authorization check. If a bitstream sits behind a CLARIN license confirmation or other access restriction, the request is rejected with 401 (anonymous user) or 403 (authenticated but not authorized), and the error message points at the item's landing page to request access. There is no Bearer/PAT support yet — GET /api/whoami-v2 always returns 401.
• Downloads are counted the same way as regular bitstream downloads: a UsageEvent is fired and the download is tracked in Matomo, for any resolve request without a Range header (mirroring how a client's first, range-probing request is distinguished from the real download).
• S3 direct download: if s3.download.direct.enabled (together with assetstore.s3.enabled) is on and the bitstream lives in the ORIGINAL bundle, GET resolve requests are answered with a 302 redirect to a presigned S3 URL instead of being streamed through DSpace — the same pattern huggingface.co itself uses for its CDN. HuggingFace clients follow redirects transparently, so this is invisible to snapshot_download / hf_hub_download. HEAD requests are always answered locally (never redirected), so clients can read the metadata headers without following the presigned URL.

Optional: a vanity HF_ENDPOINT host

To let users set HF_ENDPOINT=https://hf.example.org instead of https://<your-host>/server/api/hf, add an nginx server block that rewrites the vanity host onto the real API path:

server {
    listen 443 ssl;
    server_name hf.example.org;

    location / {
        rewrite ^/(.*)$ /server/api/hf/$1 break;
        proxy_pass http://localhost:8080;
        proxy_set_header Host <your-host>;
    }
}

Known client-side limitation

LM Studio's in-app model browser ignores HF_ENDPOINT and always talks to huggingface.co directly — this is an upstream LM Studio limitation, not something this facade can work around. See lmstudio-ai/lms#104. Command-line tools built on huggingface_hub (hf download, hf_hub_download, snapshot_download, transformers.from_pretrained) are unaffected.

See also

• All Features § HuggingFace-compatible API
• Configuration