Skip to content
This repository was archived by the owner on Nov 21, 2025. It is now read-only.
This repository was archived by the owner on Nov 21, 2025. It is now read-only.

ImageInferenceEngine can only consume a very specific ImageModel class #802

@corcra

Description

@corcra

I want to use the ImageInferenceEngine from hi-ml-multimodal with an image model that can't cleanly be fit into the existing ImageModel class.

I'd like to propose something like:

  • We define a more abstract class for ImageModels that just requires the methods actually used by ImageInferenceEngine / ImageTextInferenceEngine (from what I can see, this is forward, get_patchwise_projected_embeddings, and the training attribute). The inference engine can then just expect something from this class as an input.
  • The existing ImageModel class can subclass this, so no expected change to current behaviour.
  • I can write a custom subclass for my special model and be able to use it in the inference engine.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthi-ml-multimodalIssues related to the hi-ml-multimodal package

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions