Memory management

[Serizao]

Describe the bug

A have a A770 16GB graphical card. I try to run finunning for a model . This is python code :

import csv
import intel_extension_for_pytorch as ipex
import torch
from random import randint
from tqdm import tqdm
from itertools import islice, zip_longest

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, DataCollatorForSeq2Seq
from transformers import Trainer, TrainingArguments

import warnings

warnings.filterwarnings("ignore", category=UserWarning, module="intel_extension_for_pytorch")
warnings.filterwarnings("ignore", category=UserWarning, module="torchvision.io.image", lineno=13)


torch.xpu.empty_cache()
DEVICE = torch.device("xpu" if torch.xpu.is_available() else "cpu")
print(f"Finetuning on device: {ipex.xpu.get_device_name()}")
def get_device(self):
    if torch.xpu.is_available():
        return DEVICE
    else:
        return self.device

def place_model_on_device(self):
    self.model.to(self.args.device)


deviceCompute = torch.device("xpu" if torch.xpu.is_available() else "cpu")
print(f"Using device: {deviceCompute}")
model_name = "facebook/nllb-200-distilled-600M"
#model_name ="Helsinki-NLP/opus-mt-tc-big-fr-en"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
model.to(deviceCompute)


dataset = load_dataset("json", data_files="dataset-3/unit/data.json")
dataset = dataset["train"].shuffle(seed=42)
dataset = dataset.shard(num_shards=10, index=0)
#TrainingArguments.device = property(get_device, TrainingArguments.device.setter)


def preprocess_function(examples):
    padding = "max_length"

    inputs = [ex for ex in examples["fr"]]
    targets = [ex for ex in examples["en"]]
    model_inputs = tokenizer(inputs, padding=padding, truncation=True)
    labels = tokenizer(targets, padding=padding, truncation=True)


    model_inputs["labels"] = labels["input_ids"]
    return model_inputs


train_dataset = dataset.map(preprocess_function, batched=True, desc="Running tokenizer")
data_collator = DataCollatorForSeq2Seq(
            tokenizer,
            model=model,
            label_pad_token_id=tokenizer.pad_token_id,
            pad_to_multiple_of=64
        )

# Paramètres d'entraînement avec PyTorch
training_args = TrainingArguments(
    gradient_accumulation_steps=2,
    output_dir="./results",
    per_device_train_batch_size=4,
    num_train_epochs=3,
    logging_dir="./logs",
    logging_steps=100,
    save_steps=500,
    bf16=True,  # setting datype to bfloat16
    save_total_limit=2,  # Conservez seulement les 2 derniers checkpoints
    push_to_hub=False,   # Si vous souhaitez sauvegarder le modèle sur Hugging Face Hub##
    use_xpu=True,  # let Trainer use available XPU device (intel GPU namespace)
    xpu_backend="mpi",
    use_ipex=True,  # optimize the model and optimizer using intel extension for pyotrch (optional)
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
)

# Entraînez le modèle
trainer.train()

save_directory = "./dataset-3/models/new-finetune-fb-nllb-600M"
model.save_pretrained(save_directory)
tokenizer.save_pretrained(save_directory)

i have this error:

To test i try with extrem lower value for max_length (tokenizer) but it seem be the same result. So maybe there's a memory error.

Versions

CPU:
Architecture :                          x86_64
Mode(s) opératoire(s) des processeurs : 32-bit, 64-bit
Address sizes:                          42 bits physical, 48 bits virtual
Boutisme :                              Little Endian
Processeur(s) :                         20
Liste de processeur(s) en ligne :       0-19
Identifiant constructeur :              GenuineIntel
BIOS Vendor ID:                         Intel(R) Corporation
Nom de modèle :                         12th Gen Intel(R) Core(TM) i7-12700
BIOS Model name:                        12th Gen Intel(R) Core(TM) i7-12700 To Be Filled By O.E.M. CPU @ 4.4GHz
BIOS CPU family:                        198
Famille de processeur :                 6
Modèle :                                151
Thread(s) par cœur :                    2
Cœur(s) par socket :                    12
Socket(s) :                             1
Révision :                              2
CPU(s) scaling MHz:                     48%
Vitesse maximale du processeur en MHz : 4900,0000
Vitesse minimale du processeur en MHz : 800,0000
BogoMIPS :                              4224,00
Drapaux :                               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi umip pku ospke waitpkg gfni vaes vpclmulqdq tme rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr ibt flush_l1d arch_capabilities
Virtualisation :                        VT-x
Cache L1d :                             512 KiB (12 instances)
Cache L1i :                             512 KiB (12 instances)
Cache L2 :                              12 MiB (9 instances)
Cache L3 :                              25 MiB (1 instance)
Nœud(s) NUMA :                          1
Nœud NUMA 0 de processeur(s) :          0-19
Vulnerability Gather data sampling:     Not affected
Vulnerability Itlb multihit:            Not affected
Vulnerability L1tf:                     Not affected
Vulnerability Mds:                      Not affected
Vulnerability Meltdown:                 Not affected
Vulnerability Mmio stale data:          Not affected
Vulnerability Retbleed:                 Not affected
Vulnerability Spec store bypass:        Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:               Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:               Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                    Not affected
Vulnerability Tsx async abort:          Not affected

Versions of relevant libraries:
[pip3] intel-extension-for-pytorch==2.0.110+xpu
[pip3] numpy==1.25.2
[pip3] torch==2.0.1a0+cxx11.abi
[pip3] torchvision==0.15.2a0+cxx11.abi
[conda] N/A

[BA8F0D39]

@Serizao
You can only allocate 4GB on the A770 16 GB

#325

[Serizao]

A sad news 😢 but do you think my modele take 4GB ? On my memory monitory i don't see that