CLIPForImageClassification has been added

[Andron00e]

What does this PR do?

This PR adds a new model to the hub. Called CLIPForImage classification.

Details about implementation and pre-trained version on hub can be seen in my repo.

My "New model" issue with describing of idea
Model on hub link
Code link

Tags:
vision models: @amyeroberts

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• [] Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[NielsRogge]

this can be removed

[NielsRogge]

Suggested change

input_ids: Optional[torch.LongTensor] = None,

no text needs to be passed in case of image classification

[Andron00e]

You are right, there is no need in "input_ids" in this case.
But it is a key element when we use clip. for my further enhancement I am going to output not "clip_output.image_embeddings" but "logits_per_image" and here we need those "input_ids".

[NielsRogge]

Suggested change

	attention_mask: Optional[torch.Tensor] = None,
	position_ids: Optional[torch.LongTensor] = None,
	return_loss: Optional[bool] = None,

Same here.

[NielsRogge]

here, the same logic as here needs to be defined:

transformers/src/transformers/models/dinov2/modeling_dinov2.py

Lines 721 to 757 in 4d4febb

	logits = self.classifier(linear_input)
	loss = None
	if labels is not None:
	# move labels to correct device to enable model parallelism
	labels = labels.to(logits.device)
	if self.config.problem_type is None:
	if self.num_labels == 1:
	self.config.problem_type = "regression"
	elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
	self.config.problem_type = "single_label_classification"
	else:
	self.config.problem_type = "multi_label_classification"
	if self.config.problem_type == "regression":
	loss_fct = MSELoss()
	if self.num_labels == 1:
	loss = loss_fct(logits.squeeze(), labels.squeeze())
	else:
	loss = loss_fct(logits, labels)
	elif self.config.problem_type == "single_label_classification":
	loss_fct = CrossEntropyLoss()
	loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
	elif self.config.problem_type == "multi_label_classification":
	loss_fct = BCEWithLogitsLoss()
	loss = loss_fct(logits, labels)
	if not return_dict:
	output = (logits,) + outputs[2:]
	return ((loss,) + output) if loss is not None else output
	return ImageClassifierOutput(
	loss=loss,
	logits=logits,
	hidden_states=outputs.hidden_states,
	attentions=outputs.attentions,
	)

[NielsRogge]

Overall it's a good idea to add a CLIPForImageClassification, but one would need to make sure weights can be loaded using from_pretrained.

[Andron00e]

In my opinion it would be better to save this structure of forward pass for further enhancement.

[Andron00e]

You are right, there is no need in "input_ids" in this case.
But it is a key element when we use clip. for my further enhancement I am going to output not "clip_output.image_embeddings" but "logits_per_image" and here we need those "input_ids".

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.