Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Rust convenience macros for loading #452

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

EricLBuehler
Copy link
Owner

@EricLBuehler EricLBuehler commented Jun 20, 2024

use mistralrs::{device, load_normal_model};
fn main() -> anyhow::Result<()> {
    let dev = Device::cuda_if_available(0)?;
    let runner = load_normal_model!(
        id = "mistralai/Mistral-7B-Instruct-v0.1".to_string(),
        kind = Mistral,
        device = dev,
        use_flash_attn = false
    );
    Ok(())
}

Copy link

Code Metrics Report
  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Dockerfile              1           34           25            0            9
 Happy                   1          442          369            0           73
 JSON                    9           21           21            0            0
 Python                 32         1256         1075           37          144
 TOML                   16          444          403            1           40
-------------------------------------------------------------------------------
 Jupyter Notebooks       1            0            0            0            0
 |- Markdown             1           60           30           22            8
 |- Python               1           96           87            1            8
 (Total)                            156          117           23           16
-------------------------------------------------------------------------------
 Markdown               17         1297            0          965          332
 |- BASH                 5          100           97            0            3
 |- Python               6          122          110            0           12
 |- Rust                 3          151          135            6           10
 (Total)                           1670          342          971          357
-------------------------------------------------------------------------------
 Rust                  119        36038        32599          623         2816
 |- Markdown            59          658           13          608           37
 (Total)                          36696        32612         1231         2853
===============================================================================
 Total                 197        39532        34492         1626         3414
===============================================================================
  

@EricLBuehler EricLBuehler added documentation Improvements or additions to documentation new feature New feature or request labels Jun 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation new feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant