Inference with a pretrained model

[IssamLaradji]

Hiya, is there a simple inference script that uses a specific checkpoint to load a fully trained model and generate images?

It is not clear how to use this repo without finetuning the model.

[lisadunlap]

^^

[karry-z]

I am trying to write simple scripts with DiffusionPriorNetwork and DiffusionPrior class, and load the pretrained model from huggingface.

I do the code below as the params in conditioned-prior / vit-l-14 / aesthetic / prior_config.json:

clip = OpenAIClipAdapter("ViT-L/14")

prior_network = DiffusionPriorNetwork(
    dim = 768,
    depth = 12,
    dim_head = 64,
    heads = 12
)

diffusion_prior = DiffusionPrior(
    net = prior_network,
    clip = clip,
    timesteps = 1000,
    cond_drop_prob = 0.1
)

when loading the checkpoints with diffusion_prior.load_state_dict(torch.load(prior_path, map_location='cpu'))

I got the error below:

RuntimeError: Error(s) in loading state_dict for DiffusionPrior:
	Missing key(s) in state_dict: "net.to_time_embeds.0.1.net.0.0.weight", "net.to_time_embeds.0.1.net.0.0.bias", "net.to_time_embeds.0.1.net.1.0.weight", "net.to_time_embeds.0.1.net.1.0.bias", "net.to_time_embeds.0.1.net.2.weight", "net.to_time_embeds.0.1.net.2.bias", "net.causal_transformer.layers.0.0.norm.gamma", "net.causal_transformer.layers.0.0.norm.beta", "net.causal_transformer.layers.0.0.to_out.1.gamma", "net.causal_transformer.layers.0.0.to_out.1.beta", "net.causal_transformer.layers.0.1.0.gamma", "net.causal_transformer.layers.0.1.0.beta", "net.causal_transformer.layers.1.0.norm.gamma", "net.causal_transformer.layers.1.0.norm.beta", "net.causal_transformer.layers.1.0.to_out.1.gamma", "net.causal_transformer.layers.1.0.to_out.1.beta", "net.causal_transformer.layers.1.1.0.gamma", "net.causal_transformer.layers.1.1.0.beta", "net.causal_transformer.layers.2.0.norm.gamma", "net.causal_transformer.layers.2.0.norm.beta", "net.causal_transformer.layers.2.0.to_out.1.gamma", "net.causal_transformer.layers.2.0.to_out.1.beta", "net.causal_transformer.layers.2.1.0.gamma", "net.causal_transformer.layers.2.1.0.beta", "net.causal_transformer.layers.3.0.norm.gamma", "net.causal_transformer.layers.3.0.norm.beta", "net.causal_transformer.layers.3.0.to_out.1.gamma", "net.causal_transformer.layers.3.0.to_out.1.beta", "net.causal_transformer.layers.3.1.0.gamma", "net.causal_transformer.layers.3.1.0.beta", "net.causal_transformer.layers.4.0.norm.gamma", "net.causal_transformer.layers.4.0.norm.beta", "net.causal_transformer.layers.4.0.to_out.1.gamma", "net.causal_transformer.layers.4.0.to_out.1.beta", "net.causal_transformer.layers.4.1.0.gamma", "net.causal_transformer.layers.4.1.0.beta", "net.causal_transformer.layers.5.0.norm.gamma", "net.causal_transformer.layers.5.0.norm.beta", "net.causal_transformer.layers.5.0.to_out.1.gamma", "net.causal_transformer.layers.5.0.to_out.1.beta", "net.causal_transformer.layers.5.1.0.gamma", "net.causal_transformer.layers.5.1.0.beta", "net.causal_transformer.layers.6.0.norm.gamma", "net.causal_transformer.layers.6.0.norm.beta", "net.causal_transformer.layers.6.0.to_out.1.gamma", "net.causal_transformer.layers.6.0.to_out.1.beta", "net.causal_transformer.layers.6.1.0.gamma", "net.causal_transformer.layers.6.1.0.beta", "net.causal_transformer.layers.7.0.norm.gamma", "net.causal_transformer.layers.7.0.norm.beta", "net.causal_transformer.layers.7.0.to_out.1.gamma", "net.causal_transformer.layers.7.0.to_out.1.beta", "net.causal_transformer.layers.7.1.0.gamma", "net.causal_transformer.layers.7.1.0.beta", "net.causal_transformer.layers.8.0.norm.gamma", "net.causal_transformer.layers.8.0.norm.beta", "net.causal_transformer.layers.8.0.to_out.1.gamma", "net.causal_transformer.layers.8.0.to_out.1.beta", "net.causal_transformer.layers.8.1.0.gamma", "net.causal_transformer.layers.8.1.0.beta", "net.causal_transformer.layers.9.0.norm.gamma", "net.causal_transformer.layers.9.0.norm.beta", "net.causal_transformer.layers.9.0.to_out.1.gamma", "net.causal_transformer.layers.9.0.to_out.1.beta", "net.causal_transformer.layers.9.1.0.gamma", "net.causal_transformer.layers.9.1.0.beta", "net.causal_transformer.layers.10.0.norm.gamma", "net.causal_transformer.layers.10.0.norm.beta", "net.causal_transformer.layers.10.0.to_out.1.gamma", "net.causal_transformer.layers.10.0.to_out.1.beta", "net.causal_transformer.layers.10.1.0.gamma", "net.causal_transformer.layers.10.1.0.beta", "net.causal_transformer.layers.11.0.norm.gamma", "net.causal_transformer.layers.11.0.norm.beta", "net.causal_transformer.layers.11.0.to_out.1.gamma", "net.causal_transformer.layers.11.0.to_out.1.beta", "net.causal_transformer.layers.11.1.0.gamma", "net.causal_transformer.layers.11.1.0.beta", "net.causal_transformer.norm.gamma", "net.causal_transformer.norm.beta". 
	Unexpected key(s) in state_dict: "net.to_time_embeds.0.weight", "net.causal_transformer.layers.0.0.norm.g", "net.causal_transformer.layers.0.0.to_out.1.g", "net.causal_transformer.layers.0.1.0.g", "net.causal_transformer.layers.0.1.3.g", "net.causal_transformer.layers.1.0.norm.g", "net.causal_transformer.layers.1.0.to_out.1.g", "net.causal_transformer.layers.1.1.0.g", "net.causal_transformer.layers.1.1.3.g", "net.causal_transformer.layers.2.0.norm.g", "net.causal_transformer.layers.2.0.to_out.1.g", "net.causal_transformer.layers.2.1.0.g", "net.causal_transformer.layers.2.1.3.g", "net.causal_transformer.layers.3.0.norm.g", "net.causal_transformer.layers.3.0.to_out.1.g", "net.causal_transformer.layers.3.1.0.g", "net.causal_transformer.layers.3.1.3.g", "net.causal_transformer.layers.4.0.norm.g", "net.causal_transformer.layers.4.0.to_out.1.g", "net.causal_transformer.layers.4.1.0.g", "net.causal_transformer.layers.4.1.3.g", "net.causal_transformer.layers.5.0.norm.g", "net.causal_transformer.layers.5.0.to_out.1.g", "net.causal_transformer.layers.5.1.0.g", "net.causal_transformer.layers.5.1.3.g", "net.causal_transformer.layers.6.0.norm.g", "net.causal_transformer.layers.6.0.to_out.1.g", "net.causal_transformer.layers.6.1.0.g", "net.causal_transformer.layers.6.1.3.g", "net.causal_transformer.layers.7.0.norm.g", "net.causal_transformer.layers.7.0.to_out.1.g", "net.causal_transformer.layers.7.1.0.g", "net.causal_transformer.layers.7.1.3.g", "net.causal_transformer.layers.8.0.norm.g", "net.causal_transformer.layers.8.0.to_out.1.g", "net.causal_transformer.layers.8.1.0.g", "net.causal_transformer.layers.8.1.3.g", "net.causal_transformer.layers.9.0.norm.g", "net.causal_transformer.layers.9.0.to_out.1.g", "net.causal_transformer.layers.9.1.0.g", "net.causal_transformer.layers.9.1.3.g", "net.causal_transformer.layers.10.0.norm.g", "net.causal_transformer.layers.10.0.to_out.1.g", "net.causal_transformer.layers.10.1.0.g", "net.causal_transformer.layers.10.1.3.g", "net.causal_transformer.layers.11.0.norm.g", "net.causal_transformer.layers.11.0.to_out.1.g", "net.causal_transformer.layers.11.1.0.g", "net.causal_transformer.layers.11.1.3.g", "net.causal_transformer.norm.g".

does anyone have some idea?

[FrankieDong]

I am trying to write simple scripts with DiffusionPriorNetwork and DiffusionPrior class, and load the pretrained model from huggingface.

I do the code below as the params in conditioned-prior / vit-l-14 / aesthetic / prior_config.json:

clip = OpenAIClipAdapter("ViT-L/14")

prior_network = DiffusionPriorNetwork(
    dim = 768,
    depth = 12,
    dim_head = 64,
    heads = 12
)

diffusion_prior = DiffusionPrior(
    net = prior_network,
    clip = clip,
    timesteps = 1000,
    cond_drop_prob = 0.1
)

when loading the checkpoints with diffusion_prior.load_state_dict(torch.load(prior_path, map_location='cpu'))

I got the error below:

RuntimeError: Error(s) in loading state_dict for DiffusionPrior:
	Missing key(s) in state_dict: "net.to_time_embeds.0.1.net.0.0.weight", "net.to_time_embeds.0.1.net.0.0.bias", "net.to_time_embeds.0.1.net.1.0.weight", "net.to_time_embeds.0.1.net.1.0.bias", "net.to_time_embeds.0.1.net.2.weight", "net.to_time_embeds.0.1.net.2.bias", "net.causal_transformer.layers.0.0.norm.gamma", "net.causal_transformer.layers.0.0.norm.beta", "net.causal_transformer.layers.0.0.to_out.1.gamma", "net.causal_transformer.layers.0.0.to_out.1.beta", "net.causal_transformer.layers.0.1.0.gamma", "net.causal_transformer.layers.0.1.0.beta", "net.causal_transformer.layers.1.0.norm.gamma", "net.causal_transformer.layers.1.0.norm.beta", "net.causal_transformer.layers.1.0.to_out.1.gamma", "net.causal_transformer.layers.1.0.to_out.1.beta", "net.causal_transformer.layers.1.1.0.gamma", "net.causal_transformer.layers.1.1.0.beta", "net.causal_transformer.layers.2.0.norm.gamma", "net.causal_transformer.layers.2.0.norm.beta", "net.causal_transformer.layers.2.0.to_out.1.gamma", "net.causal_transformer.layers.2.0.to_out.1.beta", "net.causal_transformer.layers.2.1.0.gamma", "net.causal_transformer.layers.2.1.0.beta", "net.causal_transformer.layers.3.0.norm.gamma", "net.causal_transformer.layers.3.0.norm.beta", "net.causal_transformer.layers.3.0.to_out.1.gamma", "net.causal_transformer.layers.3.0.to_out.1.beta", "net.causal_transformer.layers.3.1.0.gamma", "net.causal_transformer.layers.3.1.0.beta", "net.causal_transformer.layers.4.0.norm.gamma", "net.causal_transformer.layers.4.0.norm.beta", "net.causal_transformer.layers.4.0.to_out.1.gamma", "net.causal_transformer.layers.4.0.to_out.1.beta", "net.causal_transformer.layers.4.1.0.gamma", "net.causal_transformer.layers.4.1.0.beta", "net.causal_transformer.layers.5.0.norm.gamma", "net.causal_transformer.layers.5.0.norm.beta", "net.causal_transformer.layers.5.0.to_out.1.gamma", "net.causal_transformer.layers.5.0.to_out.1.beta", "net.causal_transformer.layers.5.1.0.gamma", "net.causal_transformer.layers.5.1.0.beta", "net.causal_transformer.layers.6.0.norm.gamma", "net.causal_transformer.layers.6.0.norm.beta", "net.causal_transformer.layers.6.0.to_out.1.gamma", "net.causal_transformer.layers.6.0.to_out.1.beta", "net.causal_transformer.layers.6.1.0.gamma", "net.causal_transformer.layers.6.1.0.beta", "net.causal_transformer.layers.7.0.norm.gamma", "net.causal_transformer.layers.7.0.norm.beta", "net.causal_transformer.layers.7.0.to_out.1.gamma", "net.causal_transformer.layers.7.0.to_out.1.beta", "net.causal_transformer.layers.7.1.0.gamma", "net.causal_transformer.layers.7.1.0.beta", "net.causal_transformer.layers.8.0.norm.gamma", "net.causal_transformer.layers.8.0.norm.beta", "net.causal_transformer.layers.8.0.to_out.1.gamma", "net.causal_transformer.layers.8.0.to_out.1.beta", "net.causal_transformer.layers.8.1.0.gamma", "net.causal_transformer.layers.8.1.0.beta", "net.causal_transformer.layers.9.0.norm.gamma", "net.causal_transformer.layers.9.0.norm.beta", "net.causal_transformer.layers.9.0.to_out.1.gamma", "net.causal_transformer.layers.9.0.to_out.1.beta", "net.causal_transformer.layers.9.1.0.gamma", "net.causal_transformer.layers.9.1.0.beta", "net.causal_transformer.layers.10.0.norm.gamma", "net.causal_transformer.layers.10.0.norm.beta", "net.causal_transformer.layers.10.0.to_out.1.gamma", "net.causal_transformer.layers.10.0.to_out.1.beta", "net.causal_transformer.layers.10.1.0.gamma", "net.causal_transformer.layers.10.1.0.beta", "net.causal_transformer.layers.11.0.norm.gamma", "net.causal_transformer.layers.11.0.norm.beta", "net.causal_transformer.layers.11.0.to_out.1.gamma", "net.causal_transformer.layers.11.0.to_out.1.beta", "net.causal_transformer.layers.11.1.0.gamma", "net.causal_transformer.layers.11.1.0.beta", "net.causal_transformer.norm.gamma", "net.causal_transformer.norm.beta". 
	Unexpected key(s) in state_dict: "net.to_time_embeds.0.weight", "net.causal_transformer.layers.0.0.norm.g", "net.causal_transformer.layers.0.0.to_out.1.g", "net.causal_transformer.layers.0.1.0.g", "net.causal_transformer.layers.0.1.3.g", "net.causal_transformer.layers.1.0.norm.g", "net.causal_transformer.layers.1.0.to_out.1.g", "net.causal_transformer.layers.1.1.0.g", "net.causal_transformer.layers.1.1.3.g", "net.causal_transformer.layers.2.0.norm.g", "net.causal_transformer.layers.2.0.to_out.1.g", "net.causal_transformer.layers.2.1.0.g", "net.causal_transformer.layers.2.1.3.g", "net.causal_transformer.layers.3.0.norm.g", "net.causal_transformer.layers.3.0.to_out.1.g", "net.causal_transformer.layers.3.1.0.g", "net.causal_transformer.layers.3.1.3.g", "net.causal_transformer.layers.4.0.norm.g", "net.causal_transformer.layers.4.0.to_out.1.g", "net.causal_transformer.layers.4.1.0.g", "net.causal_transformer.layers.4.1.3.g", "net.causal_transformer.layers.5.0.norm.g", "net.causal_transformer.layers.5.0.to_out.1.g", "net.causal_transformer.layers.5.1.0.g", "net.causal_transformer.layers.5.1.3.g", "net.causal_transformer.layers.6.0.norm.g", "net.causal_transformer.layers.6.0.to_out.1.g", "net.causal_transformer.layers.6.1.0.g", "net.causal_transformer.layers.6.1.3.g", "net.causal_transformer.layers.7.0.norm.g", "net.causal_transformer.layers.7.0.to_out.1.g", "net.causal_transformer.layers.7.1.0.g", "net.causal_transformer.layers.7.1.3.g", "net.causal_transformer.layers.8.0.norm.g", "net.causal_transformer.layers.8.0.to_out.1.g", "net.causal_transformer.layers.8.1.0.g", "net.causal_transformer.layers.8.1.3.g", "net.causal_transformer.layers.9.0.norm.g", "net.causal_transformer.layers.9.0.to_out.1.g", "net.causal_transformer.layers.9.1.0.g", "net.causal_transformer.layers.9.1.3.g", "net.causal_transformer.layers.10.0.norm.g", "net.causal_transformer.layers.10.0.to_out.1.g", "net.causal_transformer.layers.10.1.0.g", "net.causal_transformer.layers.10.1.3.g", "net.causal_transformer.layers.11.0.norm.g", "net.causal_transformer.layers.11.0.to_out.1.g", "net.causal_transformer.layers.11.1.0.g", "net.causal_transformer.layers.11.1.3.g", "net.causal_transformer.norm.g".

does anyone have some idea?

same problem with me, i am finding solution now

[FrankieDong]

you can try set strict to false, maybe is ok

[cest-andre]

#282

See my response here. This may help you too.