Add pagination support to list_models() API for systematic model discovery

[darwich6]

Is your feature request related to a problem? Please describe.
The current list_models() API only supports a limit parameter without true pagination support. This makes it impossible to systematically discover models beyond the initial limit. For example, when fetching most downloaded models:

models = hf.list_models(
filter="text-generation",
sort="downloads",
direction=-1,
limit=100
)

This will always return the same top 100 models unless a new model has overtaken the top 100 model downloads. There's no way to get models 101-200, unless you change the limit to 200 and grab the 1-200 records. This is problematic for services that need to:

• Discover new models systematically
• Process models in smaller batches
• Index or monitor the full model ecosystem

Describe the solution you'd like

Add proper pagination support to the API by either:

1. Adding an offset parameter:

python
models = hf.list_models(
filter="text-generation",
sort="downloads",
limit=100,
offset=100 # Get next 100 models
)

2. Or exposing the internal cursor-based pagination that's already used by paginate():

python
response = hf.list_models(
filter="text-generation",
limit=100,
cursor="next_page_token" # From previous response
)
next_cursor = response.next_cursor

Describe alternatives you've considered

Current workarounds we've tried:

1. Fetching very large batches (1000+ models) and filtering locally
2. Using different sort criteria to try to get different models
3. Using the search parameter with different queries

None of these provide a reliable way to systematically discover all models and ensure we are getting different models with each call.

Additional context
Looking at the source code, the API already uses internal pagination via paginate():

items = paginate(path, params=params, headers=headers)
if limit is not None:
items = islice(items, limit)

Exposing this functionality would align with common API practices and enable better tooling around the Hub's model ecosystem.

[hanouticelina]

Hi @darwich6,
sorry for the late answer! thanks for this feature request. cursor-based pagination is already implemented server-side in the /api/models endpoint, but we haven't exposed it yet in HfApi.list_models(). we'll prioritize adding this feature.

in the meantime, here's a small script that leverage the cursor-based pagination:

from typing import Literal, Optional
from urllib.parse import parse_qs, urlparse

from huggingface_hub.utils import get_session, hf_raise_for_status


class ModelIterator:
    def __init__(self, items, next_cursor: Optional[str] = None):
        self.items = items
        self.next_cursor = next_cursor

    def __iter__(self):
        yield from self.items


def list_models_with_cursor(
    *,
    filter: Optional[str] = None,
    limit: int = 100,
    direction: Optional[Literal[-1]] = None,
    cursor: Optional[str] = None,
) -> ModelIterator:
    """
    List models with cursor-based pagination.
    """
    url = "https://huggingface.co/api/models"
    params = {
        "filter": filter,
        "limit": limit,
        "direction": direction,
        "cursor": cursor,
    }

    response = get_session().get(url, params=params)
    hf_raise_for_status(response)
    next_url = response.links.get("next", {}).get("url")
    next_cursor = None
    if next_url:
        next_cursor = parse_qs(urlparse(next_url).query)["cursor"][0]

    return ModelIterator(response.json(), next_cursor)


response = list_models_with_cursor(filter="text-generation", limit=100)
if response.next_cursor:
    next_response = list_models_with_cursor(filter="text-generation", limit=100, cursor=response.next_cursor)

[darwich6]

Thank you for your prioritization and response! Looking forward to the feature!

[anirbanbasu]

@hanouticelina is the cursor parameter in the link URL in the response header constant for a given set of parameters (and the page), or is it session-based?

I tried paging using curl following the link in the response header setting the limit parameter. It worked fine. Then, I randomly tried one of those links on a different browser on a different device on a different network. I still got the same results as I expected for that page. That led me to thinking that it is not a session-based cursor?

Also, has anyone worked on this issue yet?

[Wauplin]

It is not session-based indeed. Information is contained in the cursor parameter.

Also, has anyone worked on this issue yet?

Not really no as it is not something we want to explore to add to the official package. @hanouticelina provided the correct snippet for power users to use it but it would be an unnecessary addition so far.

[anirbanbasu]

@Wauplin thanks!

Not really no as it is not something we want to explore to add to the official package. @hanouticelina provided the correct snippet for power users to use it but it would be an unnecessary addition so far.

The snippet from @hanouticelina is excellent -- thanks for that. However, it feels that if one has to implement the entire list_models functionality with the paging through the cursor then it would be a repetition of the list_models method in the library.

Also, I wonder what the following lines are for, in the huggingface_hub/utils/_pagination.py:paginate.

    # Follow pages
    # Next link already contains query params
    next_page = _get_next_page(r)
    while next_page is not None:
        logger.debug(f"Pagination detected. Requesting next page: {next_page}")
        r = http_backoff("GET", next_page, max_retries=20, retry_on_status_codes=429, headers=headers)
        hf_raise_for_status(r)
        yield from r.json()
        next_page = _get_next_page(r)

Digging further with curl, running curl -I "https://huggingface.co/api/models?apps=ollama", I get (in the headers)

link: <https://huggingface.co/api/models?apps=ollama&sort=trendingScore&cursor=eyIkb3IiOlt7InRyZW5kaW5nU2NvcmUiOjAuMiwiX2lkIjp7IiRndCI6IjY4YjRmZTdlMWZlMzgwZTc2MDExMTc4ZSJ9fSx7InRyZW5kaW5nU2NvcmUiOnsiJGx0IjowLjJ9fSx7InRyZW5kaW5nU2NvcmUiOm51bGx9XX0%3D>; rel="next"

While, calling with https://huggingface.co/api/models?apps=ollama&limit=10, I get:

link: <https://huggingface.co/api/models?apps=ollama&limit=10&sort=trendingScore&cursor=eyIkb3IiOlt7InRyZW5kaW5nU2NvcmUiOjEyLCJfaWQiOnsiJGd0IjoiNjhhZGRiYmI5NTk3NWU0OTI3Njc3OTVjIn19LHsidHJlbmRpbmdTY29yZSI6eyIkbHQiOjEyfX0seyJ0cmVuZGluZ1Njb3JlIjpudWxsfV19>; rel="next"

Why should I get a next pointer in the headers in the first case when I had not limited the results, which should perhaps not trigger pagination?

Basically, I am trying to understand the need for those lines in huggingface_hub/utils/_pagination.py:paginate when the following lines precede it and seem to be sufficient to generate the response?

    session = get_session()
    r = session.get(path, params=params, headers=headers)
    hf_raise_for_status(r)
    yield from r.json()

I think I am missing something!

[anirbanbasu]

@Wauplin, I think I figured out what is going on in huggingface_hub/utils/_pagination.py:paginate.

Without the limit in the query parameters, the default page size is 1,000 results with a next page cursor to move forward if necessary. With the limit parameter less than 1,000, there exists a link URL in the response headers but a next page is not requested.

However, if the limit parameter is not set or set to something higher than 1,000 then I get the following error when the huggingface_hub/utils/_pagination.py:paginate function is invoked by, say, list_models:

Error in listing models. 504 Server Error: Gateway Timeout for url:                                                                                 
         https://huggingface.co/api/modelsapps=ollama&gated=False&sort=trendingScore&cursor=eyIkb3IiOlt7InRyZW5kaW5nU2NvcmUiOjAuMiwiX2lkIjp7IiRndCI6IjY4YjM5YjBlNGRhZGZjMGVlZmYyZjJhZSJ9fSx7InRyZW5kaW5nU2NvcmUiOnsiJGx0IjowLjJ9fSx7InRyZW5kaW5nU2NvcmUiOm51bGx9XX0%3D (Request ID: Root=1-68b76d0e-74e05eda197d7df30832183d;0d464d83-b878-4a46-9fe5-131ed33c571b)                                                                      
                                                                                                                                                             
         The request is taking longer than expected, please try again later.

With respect to the original point raised in this issue, an offset parameter would be nice but with the current cursor implementation in the API, it will be less straightforward to implement it. The cursor encodes something like a greater_than condition? ($gt) with a model identifier, such as that for the cursor eyIkb3IiOlt7InRyZW5kaW5nU2NvcmUiOjAuMiwiX2lkIjp7IiRndCI6IjY4YjM5YjBlNGRhZGZjMGVlZmYyZjJhZSJ9fSx7InRyZW5kaW5nU2NvcmUiOnsiJGx0IjowLjJ9fSx7InRyZW5kaW5nU2NvcmUiOm51bGx9XX0=:

{"$or":[{"trendingScore":0.2,"_id":{"$gt":"68b39b0e4dadfc0eeff2f2ae"}},{"trendingScore":{"$lt":0.2}},{"trendingScore":null}]}

[Wauplin]

Hi @anirbanbasu you are absolutely right about what's going on here. Pagination is based on the next link header. Pagination is not an option, i.e. you cannot chose to return all models at once. The error you are getting when setting a high "limit" is due to a timeout when fetching them. This is a technical hard limitation without much workarounds.

The snippet from @hanouticelina is excellent -- thanks for that. However, it feels that if one has to implement the entire list_models functionality with the paging through the cursor then it would be a repetition of the list_models method in the library.

It is definitely a power user use case to be able to list models with an offset so I'm not shocked if one has to re-implement it by themselves.

Regarding the cursor content, it is a server-side generated parameter tightly bind to the search system we use. We do not guarantee it will keep the same pattern over time so I would not try to build or parse it by yourself.

[anirbanbasu]

Hi @Wauplin, without reverse-engineering the content of the cursor parameter, here's a plan of paging that I implemented.

api_url = f"https://huggingface.co/api/models?apps=ollama&gated=false&limit={page_size}&sort=trendingScore"
next_page_url = api_url
with self.get_httpx_client() as client:
        while next_page < page and next_page_url:
            models_head = client.head(next_page_url)
            models_head.raise_for_status()
            next_page_url = models_head.links.get("next", {}).get("url")
            next_page += 1
        if next_page_url:
            if next_page > 1:
                logger.info(f"Requesting page {next_page} from {next_page_url}")
            models_response = client.get(next_page_url)
            models_response.raise_for_status()
       # We are done. See `model_response.json()`.

Now, setting the page_size to 100, the next_page initially to 1 and the page to 9 (i.e., the requested page), I get the models from 900 to 1000. However, I get the following timeout error if I set the page to 10 or higher.

Server error '504 Gateway Timeout' for url                                                                                 
         'https://huggingface.co/api/models?apps=ollama&gated=false&limit=100&sort=trendingScore&cursor=eyIkb3IiOlt7InRyZW5kaW5nU2NvcmUiOjEsIl9pZCI6eyIkZ3QiOiI2OGI2NzkxZDY4YjZkNzRkMmE4ZDhhNmIifX0seyJ0cmVuZGluZ1Njb3JlIjp7IiRsdCI6MX19LHsidHJlbmRpbmdTY29yZSI6bnVsbH1dfQ%3D%3D'

The error you are getting when setting a high "limit" is due to a timeout when fetching them. This is a technical hard limitation without much workarounds.

Does this mean that it is technically impossible to request from the API a set of models beyond the first 1,000? Note that for the pages that I skip, I am only invoking a HTTP HEAD to obtain the next page URL. I do not invoke a GET to save on bandwidth.

[leven87]

hello, I simply request https://huggingface.co/api/models?limit=1&cursor=3 in the browser, but get error : {"error":"Error parsing pagination cursor"} . I don't know what I was missing.

Hi @darwich6, sorry for the late answer! thanks for this feature request. cursor-based pagination is already implemented server-side in the /api/models endpoint, but we haven't exposed it yet in HfApi.list_models(). we'll prioritize adding this feature.

in the meantime, here's a small script that leverage the cursor-based pagination:

from typing import Literal, Optional
from urllib.parse import parse_qs, urlparse

from huggingface_hub.utils import get_session, hf_raise_for_status

class ModelIterator:
def init(self, items, next_cursor: Optional[str] = None):
self.items = items
self.next_cursor = next_cursor

def __iter__(self):
    yield from self.items

def list_models_with_cursor(
*,
filter: Optional[str] = None,
limit: int = 100,
direction: Optional[Literal[-1]] = None,
cursor: Optional[str] = None,
) -> ModelIterator:
"""
List models with cursor-based pagination.
"""
url = "https://huggingface.co/api/models"
params = {
"filter": filter,
"limit": limit,
"direction": direction,
"cursor": cursor,
}

response = get_session().get(url, params=params)
hf_raise_for_status(response)
next_url = response.links.get("next", {}).get("url")
next_cursor = None
if next_url:
    next_cursor = parse_qs(urlparse(next_url).query)["cursor"][0]

return ModelIterator(response.json(), next_cursor)

response = list_models_with_cursor(filter="text-generation", limit=100)
if response.next_cursor:
next_response = list_models_with_cursor(filter="text-generation", limit=100, cursor=response.next_cursor)

[Wauplin]

Cursor does not correspond to a page so cursor=3 is not recognized by the server. In fact, cursor is a base64-encoded value returned by the server on the first call. For example cursor=eyIkb3IiOlt7InRyZW5kaW5nU2NvcmUiOjAuMiwiX2lkIjp7IiRndCI6IjY4YjRmZTdlMWZlMzgwZTc2MDExMTc4ZSJ9fSx7InRyZW5kaW5nU2NvcmUiOnsiJGx0IjowLjJ9fSx7InRyZW5kaW5nU2NvcmUiOm51bGx9XX0%3D in my initial comment is the base 64 for {"$or":[{"trendingScore":0.2,"_id":{"$gt":"68b4fe7e1fe380e76011178e"}},{"trendingScore":{"$lt":0.2}},{"trendingScore":null}]}. I would highly recommend not building a cursor value manually.

As described above, you must read the "link" header from the first call in order to get the cursor for second page, etc.