Skip to content

[pipeline] Add pool option to image feature extraction pipeline#28985

Merged
amyeroberts merged 2 commits intohuggingface:mainfrom
amyeroberts:pooling-image-fe-pipeline
Feb 20, 2024
Merged

[pipeline] Add pool option to image feature extraction pipeline#28985
amyeroberts merged 2 commits intohuggingface:mainfrom
amyeroberts:pooling-image-fe-pipeline

Conversation

@amyeroberts
Copy link
Copy Markdown
Contributor

What does this PR do?

Adds the flag pool which will return the pooled output, rather than the raw hidden states.

Doesn't work for data2vecvision as the model doesn't add the pooling layer by default. At the moment, model kwargs aren't passed to the model constructor, so it's not simple to add passing add_pooling_layer here.

I chose to raise an error when getting the outputs of the model. Although this means we fail quite late, it avoids any complex inspection needed on the model.

From the list of models in #28944:

Models which work with the pool option:

  • beit
  • bit
  • convnext
  • convnextv2
  • deit
  • dinov2
  • dpt
  • efficientnet
  • focalnet
  • levit
  • mobilenet_v1
  • mobilenet_v2
  • mobilevit
  • mobilevitv2
  • nat
  • regnet
  • resnet
  • swin
  • swinv2
  • van
  • vit
  • vit_hybrid
  • vivit
  • yolos

Models that don't have a pooling layer:

  • conditional_detr
  • deformable_detr
  • deta
  • detr
  • dinat
  • efficientformer
  • glpn
  • imagegpt
  • poolformer
  • pvt
  • segformer
  • siglip_vision_model
  • swiftformer
  • swin2sr
  • table
  • timesformer
  • timm_backbone
  • videomae
  • vit_msn
  • vitdet
  • vit_mae

Not working

  • data2vec-vision.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts amyeroberts marked this pull request as ready for review February 13, 2024 11:10
Copy link
Copy Markdown
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, let's maybe add a pipeline test were we make sure the outputs slices are alright (instead of juste the shape?)


if pool:
if "pooler_output" not in model_outputs:
raise ValueError("The model needs to have a `pooler` layer to use the `pool` option.")
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it more like the pooler_outpus were not returned since we don't check if there is a pooler layer layer or not 😉

@amyeroberts
Copy link
Copy Markdown
Contributor Author

@ArthurZucker Updated the error message and added tests for exact outputs in b94d08c

Copy link
Copy Markdown
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Feel free to merge 🤗

@amyeroberts amyeroberts force-pushed the pooling-image-fe-pipeline branch from 37563a5 to 33378e7 Compare February 20, 2024 20:00
@amyeroberts amyeroberts merged commit e770f03 into huggingface:main Feb 20, 2024
@amyeroberts amyeroberts deleted the pooling-image-fe-pipeline branch February 20, 2024 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants