OpenGVLab/InternViT-6B-448px-V1-5 as Zero Shot Image Classification.

[iavinas]

Hi,

Thanks for sharing the model and code with us.

I am trying to using Vision Foundation Model for a zero shot classification problem.

It is possible with OpenGVLab/InternVL-14B-224px but I am not able to do with OpenGVLab/InternViT-6B-448px-V1-5.

model = AutoModel.from_pretrained('OpenGVLab/InternViT-6B-448px-V1-5', torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, trust_remote_code=True).cuda().eval()

tokenizer = AutoTokenizer.from_pretrained('OpenGVLab/InternViT-6B-448px-V1-5', use_fast=False, add_eos_token=True, trust_remote_code=True)

Is there anyway to get the tokenizer for OpenGVLab/InternViT-6B-448px-V1-5?

[czczup]

Hi, the OpenGVLab/InternViT-6B-448px-V1-5 is a vision encoder extracted from the pretraining stage of the multimodal large language model (MLLM), OpenGVLab/InternVL-Chat-V1-5, which is trained to be a specialized vision encoder for MLLM and cannot be used directly for zero-shot classification tasks.

[karthika2094-manju]

is anyone tried to finetune witht he mathvista dataset and has any problem with forward pass. ? import torch
from torch.optim import AdamW
from transformers import get_linear_schedule_with_warmup

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

Assuming model, tokenizer, traindataloader are already loaded/setup

model = model.to(device)
model.train()

Prepare optimizer and scheduler

optimizer = AdamW(model.parameters(), lr=2e-4, weight_decay=0.01)
num_training_steps = len(traindataloader) * 1 # 1 epoch example
num_warmup_steps = int(0.1 * num_training_steps)
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps, num_training_steps)

grad_accum_steps = 4
running_loss = 0.0

for epoch in range(1): # Single epoch example
for step, batch in enumerate(traindataloader, start=1):

    # Move inputs to device
    inputids = batch["inputids"].to(device)
    attentionmask = batch["attentionmask"].to(device)
    labels = batch["labels"].to(device)
    pixelvalues = batch["pixelvalues"].to(device).to(model.dtype)

    # Safe handling of imageflags: create zeros if missing
    imageflags = batch.get("imageflags", None)
    if imageflags is None:
        imageflags = torch.zeros((inputids.size(0), 1), dtype=torch.float32, device=device)
    else:
        if hasattr(imageflags, "squeeze"):
            imageflags = imageflags.to(device).squeeze(-1)
        else:
            imageflags = torch.tensor(imageflags, dtype=torch.float32, device=device).squeeze(-1)

    optimizer.zero_grad()

    # Forward pass
    outputs = model(
        input_ids=inputids,
        attention_mask=attentionmask,
        pixel_values=pixelvalues,
        labels=labels,
        imageflags=imageflags,
        return_dict=True,
    )

    # Get loss
    loss = outputs.loss if outputs.loss is not None else torch.tensor(0.0, requires_grad=True).to(device)

    # Backward and step with gradient accumulation
    loss = loss / grad_accum_steps
    loss.backward()

    if step % grad_accum_steps == 0 or step == len(traindataloader):
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
        optimizer.step()
        scheduler.step()
        optimizer.zero_grad()

    running_loss += loss.item() * grad_accum_steps

    if step % 20 == 0:
        avg_loss = running_loss / 20
        print(f"Epoch {epoch+1}, Step {step}, Avg Loss: {avg_loss:.4f}")
        running_loss = 0.0

print("Training completed.")

ERROR: NoneType object has no attribute 'squeeze'
How ever i tried to fix this issues with image flags attribute also i cannot fix this . can any one see this code and suggest me where im wrong.... I WANT TO FINETUNE THE INTERNVL 8B MODEL APPLIED LORA.