Support new GPT-3.5 Turbo model gpt-3.5-turbo-1106

[onerinas]

There is a new model gpt-3.5-turbo-1106 which replaces old GPT-3.5 Turbo models.

This PR adds the token limit for gpt-3.5-turbo-1106

https://platform.openai.com/docs/models/gpt-3-5

Currently we get the below error if we try to use this model:

lib/langchain/utils/token_length/base_validator.rb:22:in `validate_max_tokens!': undefined method `-' for nil:NilClass (NoMethodError)

          leftover_tokens = token_limit(model_name) - text_token_length```

[onerinas]

Seeing error while using the above:

max_tokens is too large: 15956. This model supports at most 4096 completion tokens, whereas you provided 15956.",

This make me think if it has a separate chat completion limit like we added recently for gpt-4 https://github.com/andreibondarev/langchainrb/pull/379/files#diff-b751390de4d557f12114162633bf3eedc62391b32e3e4145bf2a1b2f1df2f0afR13

[andreibondarev]

Seeing error while using the above:

max_tokens is too large: 15956. This model supports at most 4096 completion tokens, whereas you provided 15956.",

This make me think if it has a separate chat completion limit like we added recently for gpt-4 https://github.com/andreibondarev/langchainrb/pull/379/files#diff-b751390de4d557f12114162633bf3eedc62391b32e3e4145bf2a1b2f1df2f0afR13

That sounds correct:

GPT-3.5-Turbo-1106 has a max context window of 16,385 tokens and can generate 4,096 output tokens.

Source: https://learn.microsoft.com/en-us/azure/ai-services/openai/whats-new

[onerinas]

@andreibondarev Thanks for the link, this seem to work. Let me know if there is anything else needed in this PR or if we can merge.