Skip to content

TensorFold 0.6.6: --name-priority on the CUDA server

Choose a tag to compare

@ashhart ashhart released this 06 Oct 09:38
  • --name-priority ID=background on the CUDA server. A request that names that served id (--name or an
    --alias) and sends no priority of its own is served as priority: "background", so it yields to foreground
    requests. A request's own priority always wins. This suits clients that can choose a model id but cannot add a
    field to the request, such as batch extractors. On one DGX Spark, a foreground request behind four background ones
    got its first token in 0.2 s instead of 21 s, with replies unchanged (#445). Thanks to @philip-pentatonic.