Running Gemma 3 Locally - Complete Guide

This document contains the optimized Python script for Gemma 3, along with a complete installation guide. This document will explain how to run Gemma 3 locally. It was tested with RTX 4080.

📋 Table of Contents

1. Usage & Setup

2. Optimized Python Script

3. Script Usage

---

🚀 Installation and Setup Guide

Prerequisites

• Python 3.11 or higher
• pip installed
• Internet connection
• Hugging Face account (free)

Step 1: Install Hugging Face Tools

Install Hugging Face CLI

Open your terminal or command prompt and run:

pip install huggingface_hub

Verify Installation

Check if the installation was successful:

huggingface-cli --help

To also install the full tools:

pip install transformers accelerate bitsandbytes
# To install torch for cuda (see install_pytorch.txt)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

If the command works, the installation is successful.

Step 2: Create and Configure Hugging Face Token

Create a Hugging Face Account

1. Go to https://huggingface.co
2. Click on "Sign Up"
3. Create your account with your email

Generate Access Token

1. Once logged in, click on your avatar in the top right
2. Select "Settings"
3. In the left menu, click on "Access Tokens"
4. Click on "New token"
5. Give your token a name (e.g., "gemma-local")
6. Select "Read" type (sufficient for downloading models)
7. Click on "Generate a token"
8. ⚠️ IMPORTANT: Immediately copy the token and save it in a secure place

Authenticate with CLI

Authenticate with CLI

In your terminal, run:

huggingface-cli login

Paste your token when prompted. You should see a confirmation message.

Alternative: You can also authenticate directly with:

huggingface-cli login --token YOUR_TOKEN_HERE

Step 3: Accept Gemma's Terms of Use

Access the Model Page

1. Go to https://huggingface.co/google/gemma-3-4b-it
2. You must be logged into your Hugging Face account

Accept Terms

1. On the model page, you will see a "Gated model" section
2. Read Google's terms of use for Gemma carefully
3. Check the box indicating that you accept the terms
4. Click on "Request access"
5. ⏳ Waiting: Approval may take a few minutes to several hours

Verify Access

Once access is approved, you will receive a confirmation email. You can also verify by returning to the model page – it should now display the model files instead of the request form.

Step 4: Test the Configuration

Simple Test with Python

Create a test file with this code:

from transformers import AutoTokenizer

try:
    tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-4b-it")
    print("✅ Configuration successful! You can now use Gemma 3.")
except Exception as e:
    print(f"❌ Error : {e}")

If the test is successful, your configuration is complete!

---

🧠 Python Script

Breakdown of run_gemma.py script:

Libraries

import torch
import time
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

Function Summaries

• get_gpu_memory_info(): This function retrieves and returns the total, allocated, and free VRAM (in GB) on a CUDA-enabled GPU. It returns None if no GPU is detected, providing essential insights into available GPU resources.

def get_gpu_memory_info():
    """Returns GPU memory information"""

    # Check if CUDA is available
    if not torch.cuda.is_available():
        return None
    
    # Get GPU memory info
    # To calculate total memory, you need to divide by 1024 exponent 3 to convert bytes to GB
    total = torch.cuda.get_device_properties(0).total_memory / 1024**3
    # Get allocated and free memory
    # Memory allocated is the total memory used by tensors
    allocated = torch.cuda.memory_allocated(0) / 1024**3
    free = total - allocated
    
    return {
        'total': total,
        'allocated': allocated,
        'free': free
    }

• setup_pytorch_optimizations(): This function applies several PyTorch optimizations for enhanced performance. It enables TensorFloat-32, sets high precision for matrix multiplications, activates cuDNN benchmarking, and clears the CUDA memory cache. It also includes a fix for potential MSVC errors.

def setup_pytorch_optimizations():
    """Configures PyTorch optimizations"""
    
    # FIX FOR MSVC ERROR
    # That will skip compilation of dynamo graphs
    # This is a workaround for the MSVC error that can occur with PyTorch 2.0
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True
    torch._dynamo.config.disable = True
    
    # TensorFloat-32 Optimizations
    # Enable TensorFloat-32 (TF32) for matrix multiplications and convolutions
    # This can improve performance on Ampere GPUs and later
    torch.backends.cuda.matmul.allow_tf32 = True
    torch.backends.cudnn.allow_tf32 = True
    
    # Matmul precision
    # Set the precision for matrix multiplications to 'high'
    # This can improve performance for large matrix operations
    torch.set_float32_matmul_precision('high')
    
    # cuDNN Optimization
    # Enable cuDNN optimizations for faster convolutions
    # This can significantly speed up convolutional operations
    torch.backends.cudnn.enabled = True
    torch.backends.cudnn.benchmark = True
    
    # Memory cleanup
    torch.cuda.empty_cache()
    
    print("✅ PyTorch optimizations enabled")

• get_quantization_config(available_vram_gb, model_size="12b"): This function dynamically determines the best quantization configuration (BFloat16, 8-bit, or 4-bit NF4) for the model. It bases its decision on the available VRAM and the specified model size, aiming to balance performance with memory efficiency.

def get_quantization_config(available_vram_gb, model_size="4b"):
    """Determines the best quantization configuration"""
    
    # Estimated memory required per model
    memory_requirements = {
        "4b": {"fp32": 16, "bf16": 8, "8bit": 4, "4bit": 2},
        "12b": {"fp32": 24, "bf16": 12, "8bit": 6, "4bit": 3},
        "27b": {"fp32": 54, "bf16": 27, "8bit": 14, "4bit": 7}
    }
    
    req = memory_requirements.get(model_size, memory_requirements[model_size])
    # Check available VRAM against requirements
    if available_vram_gb >= req["bf16"]:
        print(f"🎯 Configuration: BFloat16 (Required VRAM: {req['bf16']}GB)")
        return {
            "torch_dtype": torch.bfloat16, # BFloat16 configuration
            "quantization_config": None, # No quantization config needed
            "config_name": "BFloat16" # Configuration name
        }
    elif available_vram_gb >= req["8bit"]:
        print(f"🎯 Configuration: 8-bit (Required VRAM: {req['8bit']}GB)")
        return {
            "load_in_8bit": True, # Enable 8-bit quantization
            "torch_dtype": torch.float16, # Use float16 for 8-bit models
            "quantization_config": None, # No specific quantization config needed
            "config_name": "8-bit" # Configuration name
        }
    else:
        print(f"🎯 Configuration: 4-bit NF4 (Required VRAM: {req['4bit']}GB)")
        return {
            "quantization_config": BitsAndBytesConfig(
                load_in_4bit=True, # Enable 4-bit quantization
                bnb_4bit_quant_type="nf4", # Use NF4 quantization
                bnb_4bit_compute_dtype=torch.bfloat16, # Use BFloat16 for computation
                bnb_4bit_use_double_quant=True, # Enable double quantization
                bnb_4bit_quant_storage=torch.uint8 # Use uint8 for storage
            ),
            "config_name": "4-bit NF4"
        }

• load_model(model_id): This function handles the loading of the Gemma 3 model and its tokenizer from Hugging Face. It automatically applies the PyTorch optimizations and selects the appropriate quantization based on available VRAM, including a fallback to CPU if GPU loading fails.

def load_model(model_id):
    """Loads a model with automatic optimizations"""
    
    # Setup optimizations
    setup_pytorch_optimizations()
    
    # GPU Info
    gpu_info = get_gpu_memory_info()
    if gpu_info:
        print(f"🖥️  GPU: {torch.cuda.get_device_name(0)}")
        print(f"💾 VRAM: {gpu_info['free']:.1f}GB free out of {gpu_info['total']:.1f}GB")
        available_vram = gpu_info['free']
    else:
        print("⚠️  No GPU detected, using CPU")
        available_vram = 0
    
    # Quantization configuration
    config = get_quantization_config(available_vram)
    
    # Load tokenizer
    print(f"📥 Loading tokenizer...")
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    
    # Load model
    print(f"🧠 Loading model {model_id}...")
    start_time = time.time()
    
    # Prepare parameters
    model_params = {
        "device_map": "auto",
        "trust_remote_code": True
    }
    
    # Add specific configuration
    if config.get("quantization_config"):
        model_params["quantization_config"] = config["quantization_config"]
    elif config.get("load_in_8bit"):
        model_params["load_in_8bit"] = True
        model_params["torch_dtype"] = config["torch_dtype"]
    else:
        model_params["torch_dtype"] = config["torch_dtype"]
    
    try:
        # Load the model with the specified parameters
        # Using AutoModelForCausalLM for causal language models
        # Note: This may require the transformers library to be updated
        model = AutoModelForCausalLM.from_pretrained(model_id, **model_params)
        
        load_time = time.time() - start_time
        print(f"✅ Model loaded in {load_time:.1f}s with {config['config_name']} configuration")
        
        # Post-load optimizations
        model.eval()  # Inference mode
        
        # Display final info
        if gpu_info:
            final_info = get_gpu_memory_info()
            used_vram = final_info['allocated'] - gpu_info['allocated']
            print(f"📊 VRAM used: {used_vram:.1f}GB")
        
        return model, tokenizer, config['config_name']
        
    except Exception as e:
        print(f"❌ Error during loading: {e}")
        print("Attempting with fallback configuration...")
        
        # Fallback configuration (CPU)
        model = AutoModelForCausalLM.from_pretrained(
            model_id,
            torch_dtype=torch.float32
        )
        print("⚠️  Model loaded on CPU. Reduced performance.")
        return model, tokenizer, "CPU Fallback"

• generate(model, tokenizer, prompt, **kwargs): This function generates text responses from the loaded Gemma 3 model. It formats the user's prompt, tokenizes it, and uses optimized generation parameters. It also attempts to leverage Flash Attention for faster generation and decodes the model's output into a readable response.

def generate(model, tokenizer, prompt, **kwargs):
    """Generation with memory management"""
    
    # Default template for chat
    chat = [{"role": "user", "content": prompt}]
    formatted_prompt = tokenizer.apply_chat_template(
        chat, 
        tokenize=False, 
        add_generation_prompt=True
    )
    
    # Tokenization and device management
    print("🔍 Tokenizing input...")
    inputs = tokenizer(formatted_prompt, return_tensors="pt")
    # Move inputs to the appropriate device
    if torch.cuda.is_available() and hasattr(model, 'device'):
        inputs = inputs.to(model.device)
    
    # Default parameters
    default_params = {
        "max_new_tokens": 2000, # Increased for longer responses
        "temperature": 0.7, # Adjusted for better quality
        "top_k": 50, # Reduced to allow more diverse outputs
        "top_p": 0.95, # Adjusted for better quality
        "do_sample": True, # Enable sampling for more diverse outputs
        "pad_token_id": tokenizer.eos_token_id, # Use EOS token for padding
        "use_cache": True  # Enable KV cache
    }
    
    # Merge parameters
    generation_params = {**default_params, **kwargs}
    
    # Generation with memory optimizations
    with torch.no_grad():
        try:
            # Context manager for Flash Attention if available
            with torch.backends.cuda.sdpa_kernel(
                enable_flash=True, # Enable/Disable Flash Attention
                enable_math=True, # Enable/Disable math optimizations
                enable_mem_efficient=True # Enable/Disable memory-efficient mode
            ):
                # Generate with Flash Attention
                print
                outputs = model.generate(**inputs, **generation_params)
        except:
            # Fallback without Flash Attention
            print("⚠️ Flash Attention not available, using standard generation.")
            outputs = model.generate(**inputs, **generation_params)
    
    # Decode the output
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    
    return response

• print_menu(): This function displays the interactive main menu of the script to the user. It presents options to interact with Gemma, view system information, or exit the program, ensuring that the user provides valid input.

def print_menu():
    """Displays the main menu"""
    choice = 0
    while choice not in ['1', '2', '0']:
        print("\n" + "="*50)
        print("🤖 Gemma 3 - Main Menu")
        print("="*50)
        print("1. 💬 Prompt Gemma")
        print("2. 📊 System Information")
        print("0. ❌ Exit")
        print("="*50)
        print("Choose an option (1, 2 or 0): ", end="")
        choice = input().strip()
        if choice not in ['1', '2', '0']:
            print("❌ Invalid option. Please choose 1, 2 or 0.")
    return choice

• show_system_info(config_name): This function provides a detailed overview of the system's hardware and software. It displays GPU specifics (name, VRAM usage), the active model configuration, and the versions of PyTorch and CUDA, which is helpful for monitoring and troubleshooting.

def show_system_info(config_name):
    """Displays system information"""
    print("\n" + "="*50)
    print("📊 System Information")
    print("="*50)
    
    # GPU Info
    if torch.cuda.is_available():
        print(f"🖥️  GPU: {torch.cuda.get_device_name(0)}")
        gpu_info = get_gpu_memory_info()
        if gpu_info:
            print(f"💾 Total VRAM: {gpu_info['total']:.1f}GB")
            print(f"💾 Used VRAM: {gpu_info['allocated']:.1f}GB")
            print(f"💾 Free VRAM: {gpu_info['free']:.1f}GB")
        print(f"🔧 Active configuration: {config_name}")
    else:
        print("⚠️  GPU not available - CPU Mode")
    
    # PyTorch Info
    print(f"🐍 PyTorch: {torch.__version__}")
    print(f"🔥 CUDA available: {'Yes' if torch.cuda.is_available() else 'No'}")
    if torch.cuda.is_available():
        print(f"🔥 CUDA Version: {torch.version.cuda}")
    
    print("="*50)

• main(model, tokenizer, config_name): This function orchestrates the main interactive loop of the application. It allows users to navigate through the menu options, input prompts for text generation, displays the model's responses along with generation time and approximate tokens per second, and includes basic error handling.

def main(model, tokenizer, config_name):
    """Main function with interactive menu"""
    
    print(f"\n🎉 Model loaded successfully!")
    print(f"🔧 Configuration: {config_name}")
    
    while True:
        choice = print_menu()
        
        if choice == '1':
            # Text generation mode
            prompt = input("\n💭 Enter your prompt (or 'back' to return to menu): ").strip()
            
            if prompt.lower() == 'back':
                continue
            elif not prompt:
                print("❌ Prompt cannot be empty.")
                continue
            
            print(f"\n🤔 Generating response for: '{prompt}'")
            print("⏳ Please wait...")
            
            try:
                # Start timer for generation
                start_time = time.perf_counter()
                # Generation call
                response = generate(model, tokenizer, prompt)
                # End timer for generation
                end_time = time.perf_counter()
                generation_time = end_time - start_time
                
                # Extract new response 
                if "<start_of_turn>model" in response:
                    new_response = response.split("<start_of_turn>model", 1)[-1]
                    if new_response.startswith("\n"):
                        new_response = new_response[1:]
                else:
                    # Fallback: take everything after the original prompt
                    prompt_in_response = response.find(prompt)
                    if prompt_in_response != -1:
                        new_response = response[prompt_in_response + len(prompt):].strip()
                    else:
                        new_response = response
                
                # Display results
                print("\n" + "="*60)
                print("🤖 MODEL RESPONSE")
                print("="*60)
                print(new_response.strip())
                print("="*60)
                print(f"⏱️  Generation time: {generation_time:.2f} seconds")
                print(f"📝 Length: {len(new_response)} characters")
                
                # Approximate tokens/second calculation
                approx_tokens = len(new_response.split())
                if generation_time > 0:
                    tokens_per_sec = approx_tokens / generation_time
                    print(f"🚀 Approximate speed: {tokens_per_sec:.1f} tokens/sec")
                
            except Exception as e:
                print(f"❌ Error during generation: {e}")
                print("💡 Try a shorter prompt or restart the script.")
            
            input("\n👆 Press Enter to continue...")
            
        elif choice == '2':
            # Display system information
            show_system_info(config_name)
            input("\n👆 Press Enter to continue...")
            
        elif choice == '0':
            # Exit
            print("\n👋 Goodbye!")
            break

• run_gemma(): This is the primary entry point for the entire script. It defines the Gemma 3 model ID, initiates the optimized model loading process, and then launches the interactive main menu. It also incorporates robust error handling for user interruptions and critical startup failures.

def run_gemma():
    """Main launch function"""
    
    # Model configuration
    model_id = "google/gemma-3-4b-it"
    
    print("🚀 Starting Gemma 3")
    print(f"📋 Model: {model_id}")
    
    try:
        # model loading 
        model, tokenizer, config_name = load_model(model_id)
        
        # Launch main menu
        main(model, tokenizer, config_name)
        
    except KeyboardInterrupt:
        print("\n⚠️  Script interrupted by user.")
    except Exception as e:
        print(f"❌ Fatal error: {e}")
        print("💡 Check your internet connection and available disk space.")

---

🎯 Script Usage

Launch

Once the configuration is complete, run the script:

python run_gemma.py

Features

Main Menu :

1. 💬 Prompt Gemma - Interact with the model
2. 📊 System Information - Displays GPU/CPU details
3. ❌ Exit - Closes the program

Automatic Optimizations :

• Automatic VRAM detection
• Optimal quantization configuration (BFloat16 → 8-bit → 4-bit NF4)
• PyTorch optimizations (TF32, cuDNN benchmark, Flash Attention)
• Intelligent memory management

Required Configuration per Model Size

Model	BFloat16	8-bit	4-bit NF4
4B	8 GB	4 GB	2 GB
12B	12 GB	6 GB	3 GB
27B	27 GB	14 GB	7 GB

---

🔧 Troubleshooting

"Repository not found" Error

• Verify that you are logged in:huggingface-cli whoami
• Make sure you have accepted the terms on the model page

Token Error

• Regenerate a new token on Hugging Face
• Re-login: huggingface-cli login

Permissions Error

• Check that your token has "Read" permissions
• Wait for approval of your Gemma access request

Memory Issues

• Check that your token has "Read" permissions
• Wait for approval of your Gemma access request

Triton library issue

• if you accounter issues with triton library try install it

pip install triton

• Window :

  pip install https://huggingface.co/madbuda/triton-windows-builds/resolve/main/triton-3.0.0-cp311-cp311-win_amd64.whl```

---

✨ Congratulations! Your environment is now ready to use Gemma 3 with the optimized script.

📄 License

This document and the accompanying script are provided "as is" for educational and demonstration purposes only. This project is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) License.

You are free to:

Share — copy and redistribute the material in any medium or format.

Adapt — remix, transform, and build upon the material.

Under the following terms:

Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

NonCommercial — You may not use the material for commercial purposes.

ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.

For the full license text -> license