return usage for all providers - As OpenAI does

[ishaan-jaff]

OpenAI returns usage for requests, but other providers - eg. Cohere does not.

{
  "id": "chatcmpl-7hPIZUYBst5jQA4odnYgxgZ9iPZ51",
  "object": "chat.completion",
  "created": 1690579707,
  "model": "gpt-3.5-turbo-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! I'm an AI language model, so I don't have feelings, but I'm here to help you with any questions or conversations you have. How can I assist you today?"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 13,
    "completion_tokens": 38,
    "total_tokens": 51
  }

Current Cohere return val from litellm - it's missing usage - which I need to calculate costs $

{'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'content': cohere.Generation {
	id: 3369b9a7-5755-4a4b-9da6-1ebab7c7924a
	prompt: Hello, how are you?
	text:  I am doing well, thank you. How can I help you today?
	likelihood: None
	finish_reason: None
	token_likelihoods: None
}, 'role': 'assistant'}}]}

[krrishdholakia]

lol exact conclusion i landed on in #10

[krrishdholakia]

@ishaan-jaff doesn't the cohere response look broken -> it should have just returned the text. seems a bit weird to just nest the cohere object in there.

[krrishdholakia]

For anthropic / replicate i just mapped the text output:

elif model in anthropic_models:
    #anthropic defaults to os.environ.get("ANTHROPIC_API_KEY")
    prompt = f"{HUMAN_PROMPT}" 
    for message in messages:
      if "role" in message:
        if message["role"] == "user":
          prompt += f"{HUMAN_PROMPT}{message['content']}"
        else:
          prompt += f"{AI_PROMPT}{message['content']}"
      else:
        prompt += f"{HUMAN_PROMPT}{message['content']}"
    prompt += f"{AI_PROMPT}"
    anthropic = Anthropic()
    completion = anthropic.completions.create(
        model=model,
        prompt=prompt,
        max_tokens_to_sample=max_tokens
    )
    new_response = {
      "choices": [
        {
          "finish_reason": "stop",
          "index": 0,
          "message": {
              "content": completion.completion,
              "role": "assistant"
          }
        }
      ]
    }
    print(f"new response: {new_response}")
    response = new_response

[krrishdholakia]

version 0.1.341 now returns token usage across all providers. Where possible it uses the providers own tokenizer (E.g. anthropic) else it defaults to tiktoken.

[krrishdholakia]

commit: 7575d7e