Together.ai Support

[minosvasilias]

This PR adds a new TogetherAI model, implementing:
TogetherAICompletion, TogetherAIChat, TogetherAIInstruct

In support of together.ai models.

Together has two library implementations: One is the together library, but it also supports the openai library.
This PR uses the openai implementation.

Because of this, we can largely re-use the existing OpenAI classes, with some adjustments:

• To support the wide range of open source models, we use transformers tokenizers instead of tiktoken
  • GrammarlessEngine now accepts Tokenizer instances as well instead of forcing the use of GrammarlessTokenizer, allowing us to create and use TransformersTokenizer instances instead
  • TransformersTokenizer can now download the tokenizer for a HF model by itself, while TransformersEngine now only downloads the model and leaves tokenizer initialization to the tokenizer instance
  • TransformersTokenizer also gains an ignore_bos_token param, which was necessary to avoid extra tokens inserted into the prompt that are not required for API requests
• base_url is set to 'https://api.together.xyz'
• Instead of OPENAI_API_KEY, we use TOGETHERAI_API_KEY

For TogetherAIInstruct, i am simply utilizing the chat template with a single user role message, since together.ai will be parsing that in the model-appropriate instruct/chat format behind the scenes.
Initially, i was using HF chat templates for the instruction role, however midway through testing the completions endpoint stopped working for models within their chat category, which is why i reverted to above.

I also added a TogetherAI.ipynb example notebook.

[slundberg]

Thanks @minosvasilias ! Just looked through this and it is looking good to me. Could you add a couple unit tests though (one's that get skipped when there is no together account available)? That way we can ensure this continues to work as the code evolves.

Thanks again!

[minosvasilias]

Thanks for reviewing @slundberg !
Added the unit tests, largely mirroring the existing OpenAI ones. Skipped if model fails to load due to missing TOGETHERAI_API_KEY.

[slundberg]

Looks great, thanks again!