Skip to content

AbstractCore v2.19.1

Choose a tag to compare

@github-actions github-actions released this 29 Sep 17:18
· 408 commits to main since this release

This release makes the gpu setting work on Linux + NVIDIA and lets abstractcore[gpu] install on
Windows x86_64 with wheels only. Run on a Quadro RTX 5000 (driver 595, CUDA 13): LM Studio text,
llama.cpp GGUF on CUDA, faster-whisper on CUDA, Supertonic speech and Diffusers FLUX.2 klein
images. vLLM installs (0.22, PyTorch 2.11) but was not validated: its first start needs a C
compiler (build-essential). The Windows changes are not yet validated on Windows hardware.

Dependencies

  • openai>=1.0.0,<3.0.0 (was <2.0.0). The old cap held vLLM at 0.11.0 in abstractcore[gpu],
    which does not start with Transformers 5; on Linux the gpu setting now resolves a current vLLM
    (0.22 with torch 2.11 on CUDA 13 at release time). The OpenAI API calls AbstractCore makes are unchanged in openai 2.x.
  • Floors: abstractvision>=0.3.32 (its gpu settings no longer install MLX-Gen, about 2.1 GB less
    on Linux, and leave stable-diffusion.cpp out on Windows) and abstractvoice>=0.13.1
    (faster-whisper on CUDA with CUDA 12 cuBLAS; speech-to-text without TTS credentials), in the
    light install and in the apple and gpu settings.
  • abstractcore[gpu] installs on Windows x86_64 with wheels only:
    vllm; sys_platform == 'linux' (vLLM is Linux only upstream) and
    llama-cpp-python; sys_platform != 'win32' (a source build on PyPI; the AbstractFramework
    installer adds llama.cpp's prebuilt CUDA / Vulkan / CPU build on Windows). The deprecated
    all-gpu and vllm aliases still map to gpu.

Changed

  • Install hints name only the three settings. Every error, CLI message, engine install plan and
    doc says pip install -U abstractcore for a light dependency, abstractcore[apple] or
    abstractcore[gpu] for a local engine, and, on a host with no local-engine setting (Intel Mac,
    Windows on ARM), that the capability is not available there. No message advises a bare package
    or a plugin's own extra.
  • The Engines screen installs abstractcore[apple] / abstractcore[gpu] for MLX, llama.cpp, vLLM
    and Hugging Face. abstractcore --download-vision-model no longer pip-installs
    transformers/torch on its own.
  • Windows x86_64 has the gpu setting in install hints and the Engines screen. The llama.cpp hint
    and engine row there say that the AbstractFramework installer adds llama.cpp's prebuilt GPU
    build and that, with plain pip, llama.cpp is not part of abstractcore[gpu] on Windows.
  • The camera endpoints and capability hint say the camera plugin (abstractcamera) is outside the
    install settings instead of printing a pip command.
  • Recommendations on NVIDIA GPUs: output.image is diffusers /
    black-forest-labs/FLUX.2-klein-4B (AbstractVision's Diffusers backend on CUDA, float16).
    With AbstractVision 0.3.32's automatic model CPU offload it generates 768x768 in about 17 s with
    an 8.3 GiB GPU peak on a 16 GB card (about 15 GiB of system RAM holds the idle weights). It is
    fit-gated with that measured peak: an 8 GB card is reported unavailable with a GPU-memory next
    step. Apple silicon keeps MLX-Gen; hosts without a GPU stay "not available". The catalog row is
    now named "FLUX.2 [klein] 4B".
  • The recommended LM Studio text download is qwen/qwen3.5-9b@q4_k_m (was @4bit, an MLX variant
    that LM Studio does not offer off Apple silicon, where LM Studio is the recommendation).
  • apply-recommended, the defaults grid and models download --recommended read the full host
    probe off Apple silicon, so an NVIDIA host gets its image route from apply-recommended.

Fixed

  • Speech-to-text runs on the configured input.voice route. A transcription
    (output={"modality": "text", "task": "transcription"}) with no provider used AbstractVoice's
    default OpenAI engine instead of the configured one (for example local faster-whisper). The
    route now fills a missing provider/model/base_url under the same rules as every other route (a
    provider named on the request, or a call-level stt_provider/stt_model, still wins), also for
    audio given as an artifact reference without a content type.
  • Linux + NVIDIA: llama.cpp's CUDA wheels load without importing torch first.
    prepare_llama_cpp_import() also runs on Linux and, for a CUDA build of llama-cpp-python,
    preloads the NVIDIA wheels' libcudart / libcublasLt / libcublas (CUDA 12 and CUDA 13
    layouts), so GGUF models run on the GPU instead of the CPU.
  • GGUF on CUDA: when llama.cpp cannot allocate a large context on the GPU, the load takes the next
    smaller context on the GPU and falls back to the CPU only after the smallest one fails there too
    (a CPU retry at a 262144-token context could exhaust system memory). Apple silicon keeps its
    immediate CPU retry.
  • Windows: before AbstractCore first imports llama_cpp, it adds PyTorch's torch\lib folder
    (cuBLAS, cudart) to the DLL search (os.add_dll_directory and PATH), so llama.cpp's prebuilt
    CUDA wheel can find those libraries without the CUDA toolkit (not yet validated on Windows
    hardware)
    (abstractcore.utils.windows_dll.prepare_llama_cpp_import; torch is not imported).
  • Windows: the GGUF lane runs llama_backend_init() before asking llama.cpp whether it can offload
    to the GPU (the Windows CUDA / Vulkan wheels ship the GPU backend as a separate library loaded at init; not
    yet validated on Windows hardware).
  • The server's /v1/responses usage reports input_tokens_details.cache_write_tokens (0 when the
    provider reports none), which the Responses usage schema of openai 2.x requires.