[Question]: Can a locally stored TensorFlow model be used with llamaindex?

[iriye]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

Can a locally stored TensorFlow model be used with llamaindex?

[dosubot]

🤖

Yes, a locally stored TensorFlow model can be used with LlamaIndex. There are a few ways to do this based on the similar issues I found in the LlamaIndex repository.

1. You can load the model and tokenizer directly as kwargs if you can load them outside of the huggingface LLM. Alternatively, you can implement a custom LLM class. More information on how to do this can be found here. This solution was suggested in the issue Querying Index using local models with no access to download from the hub(s).

2. LlamaIndex can be used with locally hosted model services that simulate OpenAI's API. You can customize your LLM model as per the example provided in the documentation at https://gpt-index.readthedocs.io/en/latest/how_to/customization/custom_llms.html#example-using-a-custom-llm-model. The code pipeline = pipeline("text-generation", model=model_name, device="cuda:0", model_kwargs={"torch_dtype":torch.bfloat16}) is used to create a pipeline for text generation using the specified model, and it runs on the CUDA device with the specified torch data type. This solution was suggested in the issue run llama_index with locally llm that have compatible OpenAI's API.

3. You can use langchain as the underlying LLM abstraction under the hood. You should follow the langchain docs for how to add a custom llm. This solution was suggested in the issue Can I use llama-index based on a locally deployed model?. However, the provided link to the documentation is not working (404 error), so you will need to find the correct documentation or contact the creators for the correct link.

Please let me know if you need further assistance.

Sources

• Querying Index using local models with no access to download from the hub(s)
• run llama_index with locally llm that have compatible OpenAI's API
• Can I use llama-index based on a locally deployed model?

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

@iriye yes, but you'd have to implement it as a customLLM class

https://docs.llamaindex.ai/en/stable/core_modules/model_modules/llms/usage_custom.html#example-using-a-custom-llm-model-advanced