LMEngine

Overview

Welcome to LMEngine. We are pleased to introduce you to our versatile and user-friendly library, designed to facilitate local LLM inference across various programming languages. Whether you are a seasoned developer or just beginning your journey, LMEngine offers a straightforward solution to integrate advanced language model capabilities into your applications with ease.

The library maintains a minimal overhead of approximately 2.5MB. The API is highly flexible, enabling inference execution through the LME_RunInference function, which completes the process and returns the results. Users can define callbacks for customization.

LMEngine provides out-of-the-box bindings for Pascal and C/C++, featuring a straightforward procedural API that facilitates the creation of bindings for additional languages.

This library efficiently loads LLMs in GGUF format into CPU or GPU memory, utilizing a Vulkan backend for enhanced processing speed.

Installation

• Download the LMEngine repo, unzip to a desired location.

• Acquire a GGUF model. All vetted models compatible with LMEngine can be downloaded from our Hugging Face account.

• The library utilizes Vulkan for enhanced performance on supported GPUs. You can perform inference solely on the GPU or distribute the workload between the CPU and GPU to accommodate scenarios with limited VRAM (this split functionality will be available in a future update). Ensure the model size does not exceed the available system resources, considering the requisite memory.

• Consult the installdir\examples directory for demonstrations on integrating LMEngine with your programming language.

• Include the following DLLs in your project distribution: LMEngine.dll. See VirusTotal Report report.

• LMEngine API supports integration across programming languages that accommodate Win64 and Unicode, with out-of-the-box support for Pascal and C/C++.

• Ship-ready DLLs are included in the repository; however, if there is a need to rebuild the LMEngine.dll, RAD Studio 12.1 is required.

• This project is developed using RAD Studio 12.1, on Windows 11, powered by an Intel Core i5-12400F at 2500 MHz with 6 cores (12 logical), equipped with 36GB RAM and an NVIDIA RTX 3060 GPU with 12GB RAM.

• We encourage testing and welcome pull requests.

• If you find this project beneficial, please consider starring the repository, sponsoring, or promoting it. Your support is invaluable and highly appreciated.

Examples

Pascal example:

uses
  SysUtils,
  LMEngine;

begin
  // init
  LME_InitConfigt('C:/LLM/gguf', -1);
  
  // define model
  LME_DefineModel('phi-3-mini-4k-instruct.Q4_K_M.gguf',
    'phi-3-mini-4k-instruct.Q4_K_M', 4000, '<|{role}|>{content}<|end|>',
    '<|assistant|>');  
  
  // add message
  LME_AddMessage(LME_ROLE_SYSTEM, 'You are a helpful AI assistant');
  LME_AddMessage(LME_ROLE_USER, 'What is AI?');
    
  if not LME_LoadModel('phi-3-mini-4k-instruct.Q4_K_M') then
  begin
    LME_PrintLn('Error: %s', LME_FG_RED, LME_GetError());
    Exit;
  end;    
    
  // do inference
  if LME_RunInference('phi-3-mini-4k-instruct.Q4_K_M', 1024) then
    begin
      // success
    end
  else
    begin
      // error
    end;

  // unload model  
  LME_UnloadModel();

end.

C/CPP Example

#include <LMEngine.h>

int main()
{
    // init
    LME_InitConfig(L"C:/LLM/gguf", -1);
    
    // define model
    LME_DefineModel(L"phi-3-mini-4k-instruct.Q4_K_M.gguf",
      L"phi-3-mini-4k-instruct.Q4_K_M", 4000, L"<|{role}|>{content}<|end|>",
      L"<|assistant|>");  
    
    // add message
    LME_AddMessage(LME_ROLE_SYSTEM, L"You are a helpful AI assistant");
    LME_AddMessage(LME_ROLE_USER, L"What is AI?");
    
    if (!LME_LoadModel(L"phi-3-mini-4k-instruct.Q4_K_M"))
    {
        LME_PrintLn(L"Error: %s", LME_FG_RED, LME_GetError());
        return 1;
    }
    
    // do inference
    if (LME_RunInference(L'phi-3-mini-4k-instruct.Q4_K_M', 1024))
    {
        // success
    }
    else
    {
        // error
    }
    
    // unload model  
    LME_UnloadModel();

    return 0;
}

Media

LMEngine in C/C++

LMEngine04.mp4

LMEngine in Pascal

LMEngine05.mp4

Support

Our development motto:

• We will not release products that are buggy, incomplete, adding new features over not fixing underlying issues.
• We will strive to fix issues found with our products in a timely manner.
• We will maintain an attitude of quality over quantity for our products.
• We will establish a great rapport with users/customers, with communication, transparency and respect, always encouragingng feedback to help shape the direction of our products.
• We will be decent, fair, remain humble and committed to the craft.

Links

• Issues
• Discussions
• Facebook Group
• Discord
• Reddit
• YouTube
• X (Twitter)
• tinyBigGAMES

License

LMEngine is a community-driven project created by tinyBigGAMES LLC.

BSD-3-Clause license - Core developers:

• Jarrod Davis

Acknowledgments

LMEngine couldn't have been built without the help of wonderful people and great software already available from the community. Thank you!

Software

• llama.cpp
• SQLite
• sqlite-vec
• sqlite-lembed

People

• John Claw
• Robert Jalarvo

Contributors