Running inference API can sometimes be very slow or fail

[BenjaminBossan]

Following up on this slack discussion.

The problem is that the skops CI sometimes times out after we added tests for the inference API (here is a failed run). We did some further investigation and this is what we can say for now:

1. The call to the inference API is the slow step
2. It's only the first call that is slow, repeated calls are fast thanks to warm start (but our tests don't benefit from that).
3. The more processes there are in parallel, the more likely it is for the call to be slow or to time out, but there is a lot of variance
4. Even 3 parallel processes can already cause slowdown
5. This happens even without the retry decorator we use
6. If a call does not finish within 3 min, in the vast majority of times, it times out after 7 min (there is no timeout set from the client side).
7. Occasionally, we also get an internal server error

ping @adrinjalali

[adrinjalali]

Would be helpful to have a simple script to reproduce the issue here. Then I guess @Narsil could try to figure out the issue.

[BenjaminBossan]

I was using pytest to derive those results, so had no snippet ready. But I created one:

# content of foo.py
import datetime as dt
from functools import partial
import pickle
import os
import shutil
import sys
import time
import tempfile
from contextlib import contextmanager
from multiprocessing import Pool
from pathlib import Path
from uuid import uuid4

import numpy as np
from huggingface_hub import HfApi, InferenceApi
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression

from skops import card
from skops.hub_utils import init, push
from skops.utils.fixes import metadata

HF_HUB_TOKEN = os.environ["HF_HUB_TOKEN"]


@contextmanager
def timed(name):
    tic = time.perf_counter()
    print(f"Started  {name} at {dt.datetime.now()}")
    yield
    toc = time.perf_counter()
    print(f"Finished {name} in {toc-tic:.0f} sec at {dt.datetime.now()}")


def get_model():
    diabetes = load_diabetes(as_frame=True)
    X, y = diabetes.data, diabetes.target
    model = LinearRegression().fit(X, y)
    return model, X, y


def run(proc_num, model, X, y):
    repo_path = Path(tempfile.mkdtemp())
    dst_path = Path(tempfile.mkdtemp())
    try:
        test_inference(proc_num, model, X, y, repo_path, dst_path)
    finally:
        # cleanup
        shutil.rmtree(repo_path)
        shutil.rmtree(dst_path)


def test_inference(proc_num, model, X, y, repo_path, dst_path):
    client = HfApi()
    model_path = repo_path / "model.pickle"
    with open(model_path, "wb") as f:
        pickle.dump(model, f)

    version = metadata.version("scikit-learn")
    init(
        model=model_path,
        requirements=[f'scikit-learn="{version}"'],
        dst=dst_path,
        task="tabular-regression",
        data=X,
    )

    # a model card is needed for inference engine to work.
    model_card = card.Card(
        model, metadata=card.metadata_from_config(Path(dst_path))
    )
    model_card.save(Path(dst_path) / "README.md")

    user = client.whoami(token=HF_HUB_TOKEN)["name"]
    repo_id = f"{user}/test-{uuid4()}"

    push(
        repo_id=repo_id,
        source=dst_path,
        token=HF_HUB_TOKEN,
        commit_message="test message",
        create_remote=True,
    )

    model_info = HfApi().model_info(repo_id=repo_id, token=HF_HUB_TOKEN)
    api = InferenceApi(repo_id=repo_id, task=model_info.pipeline_tag, token=HF_HUB_TOKEN)
    inputs = {"data": X.head().to_dict(orient="list")}
    y_pred = model.predict(X.head())

    # CRITICAL STEP
    with timed(f"Proc {proc_num:2.0f}: inference call"):
        res = api(inputs=inputs)
    if not isinstance(res, list):
        raise RuntimeError(f"Error during inference: {res}")

    assert np.allclose(y_pred, np.array(res))
    client.delete_repo(repo_id=repo_id, token=HF_HUB_TOKEN)


def main(n_task, n_parallel):
    model, X, y = get_model()
    with Pool(n_parallel) as pool:
        pool.map(partial(run, model=model, X=X, y=y), range(n_task))


if __name__ == '__main__':
    n_task, n_parallel = int(sys.argv[1]), int(sys.argv[2])
    main(n_task, n_parallel)

It requires a Python environment with skops installed (pip install skops). Then you can run X tasks with Y being run in parallel like this:

HF_HUB_TOKEN=<token> python foo.py <X> <Y>

The more tasks in parallel, the more likely the error is triggered (e.g. try 20 tasks with 10 parallel).

[adrinjalali]

cc @Narsil

[Narsil]

This is actually relatively understandable:

My biggest question, is why do you need to test all models on every single commit ? These are integration tests, and even the API itself doesn't strain itself that much for tests. For integration tests, we have a few selected models, which are pinned (meaning always up) that we test against. Tests are not meant to put burden on production workloads.
I would be happy to pin a few select models that you could test against, but testing all of them all the time seems too much IMO.

Then the explanation:

When you cold start a model (especially when new files/weigths are there) then basically the whole mamba install has to be start over which takes whichever amount of time.

These values might need to be updated @OlivierDehaene @co42 https://github.com/huggingface/api-inference/blob/main/chart/templates/master-rs/deployment.yaml#L51-L54 to increase the amount of allowed dynamic models.

There's a limit on how many temp models are loaded at a given time, so the more you load in parallel the more chance it gets killed and swapped for another (even during its load). It's an LRU based approach so it depends on what other traffic is the API receiving, but I've started to see kill before launch events (which should never happen) and just tells me some users are trying too many different models at once, for which we can increase the 60 limit and or prevent that sort of abuse (if it really abused, which I don't really know right now)

Then the API itself has a timeout (of 2mn) on any given request. This can be changed but in my experience it's not great since it's easier to retry than to keep working for clients which have already dropped the call (happens regularly when users visit hf.co click on run and just bail, or don't even click, some links start the API by default).

If warm starts are always faster, it seems to point that the biggest of slowdown could be the mamba install part (which doesn't really suprises me), so we could also look into that, but I doubt we can really do anything about it.

[BenjaminBossan]

Thank you very much for explaining the reason for the issues we observed.

As to possible solutions: How about we not run the integration tests for the whole test matrix. We could instead set up a separate action that only tests one or two settings.

[Narsil]

Seems better IMHO. api-inference-community will test the docker itself on any commit (Not for all models, but all tasks).
Those are closer to your code meaning you're getting more confidence that your code is valid without depending on the network or that service X is up.

That being said, a full fleged integration test with the API seems reasonable, and pinning a few models to test against regularly is both perfectly reasonable and desirable !

[adrinjalali]

At this point we're still changing and adding tasks and touching the way we save configuration files. So it kinda makes sense to have the integration tests more often. What we could do, is to disable the inference tests by default (most PRs don't touch them), and only run them on main and on commits which have a specific tag, like [inference] rest of the commit message.

[BenjaminBossan]

Resolved via skops-dev/skops#212