Skip to content

Conversation

lstein
Copy link
Collaborator

@lstein lstein commented Feb 23, 2023

  • Discord member @marcus.llewellyn reported that some civitai 2.1-derived checkpoints were not converting properly (probably dreambooth-generated): https://discord.com/channels/1020123559063990373/1078386197589655582/1078387806122025070

  • @blessedcoolant tracked this down to a missing key that was used to derive vector length of the CLIP model used by fetching the second dimension of the tensor at "cond_stage_model.model.text_projection".

  • On inspection, I found that the same second dimension can be recovered from key 'cond_stage_model.model.ln_final.bias', and use that instead. I hope this is correct; tested on multiple v1, v2 and inpainting models and they converted correctly.

  • While debugging this, I found and fixed several other issues:

    • model download script was not pre-downloading the OpenCLIP text_encoder or text_tokenizer. This is fixed.
    • got rid of legacy code in ckpt_to_diffuser.py and replaced with calls into model_manager
    • more consistent status reporting in the CLI.

- Discord member @marcus.llewellyn reported that some civitai 2.1-derived checkpoints were
  not converting properly (probably dreambooth-generated):
  https://discord.com/channels/1020123559063990373/1078386197589655582/1078387806122025070

- @blessedcoolant tracked this down to a missing key that was used to
  derive vector length of the CLIP model used by fetching the second
  dimension of the tensor at "cond_stage_model.model.text_projection".
  His proposed solution was to hardcode a value of 1024.

- On inspection, I found that the same second dimension can be
  recovered from key 'cond_stage_model.model.ln_final.bias', and use
  that instead. I hope this is correct; tested on multiple v1, v2 and
  inpainting models and they converted correctly.

- While debugging this, I found and fixed several other issues:

  - model download script was not pre-downloading the OpenCLIP
    text_encoder or text_tokenizer. This is fixed.
  - got rid of legacy code in `ckpt_to_diffuser.py` and replaced
    with calls into `model_manager`
  - more consistent status reporting in the CLI.
@blessedcoolant
Copy link
Collaborator

LGTM. If any exceptions pop up with edge case models, we can see if they need a hot fix or not. For now, this might work. Hard to say without trying with a bunch of varied range models. But the code reads good.

@lstein lstein enabled auto-merge February 24, 2023 02:28
@lstein lstein merged commit c69fcb1 into v2.3 Feb 24, 2023
@lstein lstein deleted the bugfix/v2-model-conversion branch February 24, 2023 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants