Leveraging Llama 2

[tmc]

I don’t see any existing discussion about leveraging Meta’s new Llama 2 model. Curious if you guys have any plans in the making for using this new base model in gorilla.

[TomExMachina]

Don't we have all the training data to just do that on our own? The fine-tuning shouldn't be that hard to get training.

[TomExMachina]

ugh maybe not #46 . I haven't read a self-instruct paper. Isn't it just doing inference to generate more training data? Maybe jsonformer is involved. idk

edit: Okay so no jsonformer. GPT-4 was used for self-instruct:

Instruction Generation Guided by the self-instruct paradigm [42], we employed GPT-4 to generate
synthetic instruction data. We provided three in-context examples, along with a reference API
documentation, and tasked the model with generating real-world use cases that call upon the API.
We specifically instructed the model to refrain from using any API names or hints when creating
instructions. We constructed six examples (Instruction-API pairs) for each of the three model hubs.
These 18 points, were the only hand-generated or modified data. For each of our 1,645 API datapoints,
we sample 3 of 6 corresponding instruction examples to generate a total of 10 instruction-api pairs as
demonstrated in Figure 3. We would like to highlight that we only need to employ GPT-4 to generate
the instructions and this can be swapped with open-source alternatives such as LLaMA, Alpaca, etc.

Maybe this code will be shared? It should be relatively trivial (Thanks to the nuances described in the paper) with some tinkering anyway.

[ShishirPatil]

Hey @TomExMachina all the training data is at https://github.com/ShishirPatil/gorilla/tree/main/data/apibench All files with the _train.json suffix!

[ShishirPatil]

Yes! We will release a LLaMA v2 as soon as we can get our hands on some compute!

[tonxxd]

Hi @ShishirPatil is the training code being released as well? Thanks!

[yordis]

Is there any How-To guide to fine-tune/training for those unfamiliar with the topic but would like to contribute?

[ShishirPatil]

@tonxxd There is a community contributed PR in the works here #59 Thanks for your interest @yordis ! If you are interested in contributing APIs, we have a README https://github.com/ShishirPatil/gorilla/tree/main/data#how-to-contribute Let me know if you have any follow up questions!