is this support for Nvidia NIM models api?

[indramal]

is this support for Nvidia NIM models api?

[gold-silver-copper]

There is no direct support for Nvidia NIM, but according to the NIM documentation, they expose a few OpenAI compatible endpoints, so you could try using the openai client builder with a custom url. Something like

  use rig::prelude::*;
  use rig::providers::openai;

  let client = openai::Client::builder()
      .api_key(std::env::var("NVIDIA_API_KEY")?)
      .base_url("http://localhost:8000/v1")
      .build()?
      .completions_api();

  let agent = client
      .agent("meta/llama-3.1-8b-instruct")
      .build();

https://docs.nvidia.com/nim/large-language-models/latest/reference/api-reference.html

[akakarantzas]

Yeah not direct NIM support, but it should work through the OpenAI-compatible API.

Basically use the OpenAI client builder, set the base_url to your NIM endpoint, and pass the NVIDIA API key / model slug NIM expects. For local NIM that's usually something like http://localhost:8000/v1; for hosted NVIDIA endpoints it would be the NVIDIA-hosted URL instead.

So not first-class support, but the compatibility path should be fine.