Support custom_pipeline argument for Diffuser T2V Community Pipelines

[ParagEkbote]

What API design would you like to have changed or added to the library? Why?

In the diffusers community pipelines, the T2V pipelines such as Spatiotemporal Skip Guidance (STG) or CogVideoX DDIM Inversion Pipeline do not yet support the custom_pipeline argument since it is only supported for T2I pipelines like:

pipeline = DiffusionPipeline.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    custom_pipeline="pipeline_flux_with_cfg"
)

Due to this, I cannot contribute colab notebooks for the pipelines. This is further referenced here.

What use case would this enable or better enable? Can you give us a code example?

Ideally, I would like to load the CogVideoX DDIM Inversion Pipeline like:

import torch

pipeline = CogVideoXPipelineForDDIMInversion.from_pretrained(
    "THUDM/CogVideoX1.5-5B",
    torch_dtype=torch.bfloat16,
    custom_pipeline="cogvideox_ddim_inversion"
).to("cuda")

output = pipeline_for_inversion(
    prompt="prompt that describes the edited video",
    video_path="path/to/input.mp4",
    guidance_scale=6.0,
    num_inference_steps=50,
    skip_frames_start=0,
    skip_frames_end=0,
    frame_sample_step=None,
    max_num_frames=81,
    width=720,
    height=480,
    seed=42,
)
pipeline.export_latents_to_video(output.inverse_latents[-1], "path/to/inverse_video.mp4", fps=8)
pipeline.export_latents_to_video(output.recon_latents[-1], "path/to/recon_video.mp4", fps=8)

Could you please let me know if this feasible or not?

[tolgacangoz]

Hi @ParagEkbote! Could you try DiffusionPipeline.from_pretrained(..., custom_pipeline="cogvideox_ddim_inversion") with diffusers installed from the source?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.