Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(wiki) Issue converting Stable Diffusion models to Core ML ... global_step key not found in model #261

Closed
1 task done
designosis opened this issue Jun 10, 2023 · 21 comments
Closed
1 task done

(wiki) Issue converting Stable Diffusion models to Core ML ... global_step key not found in model #261

designosis opened this issue Jun 10, 2023 · 21 comments
Labels
bug Something isn't working

Comments

@designosis
Copy link

Running Latest Version

  • I am running the latest version

Processor

M1 Max (or later)

Intel processor

No response

Memory

32GB

What happened?

Following the instructions on https://github.com/godly-devotion/MochiDiffusion/wiki/How-to-convert-Stable-Diffusion-models-to-Core-ML, I download ANY safetensor file from civitai.com (e.g. https://civitai.com/models/71745/conceptliquid-clothesliquid-dress or https://civitai.com/models/63655/tarot-cards-rider-waite), and run the SD model → Diffusers conversion ...

python convert_original_stable_diffusion_to_diffusers.py --checkpoint_path LiquidClothesV1fixed.safetensors --from_safetensors --device cpu --extract_ema --dump_path LiquidClothesV1fixed_diffusers

... and no matter which file (out of ~20 random ones I tried), I receive an error starting with global_step key not found in model, for example:

global_step key not found in model
Traceback (most recent call last):
  File "convert_original_stable_diffusion_to_diffusers.py", line 131, in <module>
    pipe = download_from_original_stable_diffusion_ckpt(
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 1214, in download_from_original_stable_diffusion_ckpt
    converted_unet_checkpoint = convert_ldm_unet_checkpoint(
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 385, in convert_ldm_unet_checkpoint
    new_checkpoint["time_embedding.linear_1.weight"] = unet_state_dict["time_embed.0.weight"]
KeyError: 'time_embed.0.weight'

On rare occasions, a download conversion will spit out that key not found error, followed by a ton more, yet it creates the _diffusers directory. For example https://civitai.com/models/4468/counterfeit-v30 does this. On these rare occasions, if I try to run the Diffusers → MLMODELC process, e.g. ...

python -m python_coreml_stable_diffusion.torch2coreml --compute-unit CPU_AND_GPU --convert-vae-decoder --convert-vae-encoder --convert-unet --unet-support-controlnet --convert-text-encoder --model-version CounterfeitV30_v30_diffusers --bundle-resources-for-swift-cli --attention-implementation ORIGINAL -o CounterfeitV30_v30_original && python -m python_coreml_stable_diffusion.torch2coreml --compute-unit CPU_AND_GPU --convert-unet --model-version CounterfeitV30_v30_diffusers --bundle-resources-for-swift-cli --attention-implementation ORIGINAL -o CounterfeitV30_v30_original

... I get the following error:

Torch version 2.0.1 has not been tested with coremltools. You may run into unexpected errors. Torch 2.0.0 is the most recent version that has been tested.
INFO:__main__:Initializing StableDiffusionPipeline with CounterfeitV30_v30_diffusers..
/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/transformers/models/clip/feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead.
  warnings.warn(
INFO:__main__:Done.
INFO:__main__:Attention implementation in effect: AttentionImplementations.ORIGINAL
INFO:__main__:Converting vae_decoder
/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/diffusers/models/resnet.py:138: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert hidden_states.shape[1] == self.channels
/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/diffusers/models/resnet.py:151: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if hidden_states.shape[0] >= 64:
INFO:__main__:Converting vae_decoder to CoreML..
Converting PyTorch Frontend ==> MIL Ops:  22%|████████                             | 80/369 [00:00<00:00, 1939.07 ops/s]
Traceback (most recent call last):
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/n/StableDiffusion/ml-stable-diffusion/python_coreml_stable_diffusion/torch2coreml.py", line 1311, in <module>
    main(args)
  File "/Users/n/StableDiffusion/ml-stable-diffusion/python_coreml_stable_diffusion/torch2coreml.py", line 1161, in main
    convert_vae_decoder(pipe, args)
  File "/Users/n/StableDiffusion/ml-stable-diffusion/python_coreml_stable_diffusion/torch2coreml.py", line 454, in convert_vae_decoder
    coreml_vae_decoder, out_path = _convert_to_coreml(
  File "/Users/n/StableDiffusion/ml-stable-diffusion/python_coreml_stable_diffusion/torch2coreml.py", line 125, in _convert_to_coreml
    coreml_model = ct.convert(
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/_converters_entry.py", line 492, in convert
    mlmodel = mil_convert(
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 188, in mil_convert
    return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 212, in _mil_convert
    proto, mil_program = mil_convert_to_proto(
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 285, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 108, in __call__
    return load(*args, **kwargs)
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 63, in load
    return _perform_torch_convert(converter, debug)
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 110, in _perform_torch_convert
    raise e
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 102, in _perform_torch_convert
    prog = converter.convert()
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 284, in convert
    convert_nodes(self.context, self.graph)
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 83, in convert_nodes
    raise RuntimeError(
RuntimeError: PyTorch convert function for op 'scaled_dot_product_attention' not implemented.

Help? :)

Crash Log

No response
@designosis designosis added the bug Something isn't working label Jun 10, 2023
@jrittvo
Copy link
Contributor

jrittvo commented Jun 10, 2023

I get the global_step key not found in model warning on every conversion I have done for months. It doesn't seem to prevent successful conversion, ever.

The other errors may be coming from a recent update to a python package. Sometimes an updated package breaks things. I just updated all my packages, and I'll see if I get either of the 2 long error you are getting. In the meantime, tell us when you set up your miniconda environment, and if you updated it after setting it up, when did you last update it. I know for sure that the diffusers package was updated in the past 24 hours, and prior update broke other packages until they were also updated.

@jrittvo

This comment was marked as off-topic.

Sign in to view
@jrittvo

This comment was marked as off-topic.

Sign in to view
@jrittvo

This comment was marked as off-topic.

Sign in to view
@jrittvo

This comment was marked as off-topic.

Sign in to view
@jrittvo

This comment was marked as off-topic.

Sign in to view
@designosis
Copy link
Author

@jrittvo Thank you for the datapoints!

I did a fresh install yesterday, following the guide @ https://github.com/godly-devotion/MochiDiffusion/wiki/How-to-convert-Stable-Diffusion-models-to-Core-ML to the letter ...

  1. Already had homebrew, and update it regularly.
  2. Already had the latest wget
  3. Downloaded + installed Xcode
  4. Downloaded + installed miniconda (via the pkg)
  5. Updated miniconda on the spot with conda update conda
  6. Ran the listed commands in their order (also ran conda init zsh for safe measure since I'm using zsh (via oh-my-zsh)
  7. Downloaded the linked python script into same dir as my .safetensors file, and cd'd to it
  8. Ran conda activate coreml_stable_diffusion

I have python 3.11.4 installed (via brew), but which -a python only yields ~/miniconda3/bin/python, and python --version shows 3.10.11, so I'm guessing miniconda3 entirely overrides the PATH, which is good I think.

I saw your post above about commenting out a while section at lines 1086 and 1087. What's odd to me is that my ~/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py file has no while section on those lines, it has ...

    if pipeline_class is None:
        pipeline_class = StableDiffusionPipeline

Later down on lines 1123-4 and 1404-5, there's this ...

    while "state_dict" in checkpoint:
        checkpoint = checkpoint["state_dict"]

... but commenting it out makes no difference.

For purposes of testing, I'll use the https://civitai.com/models/22922/lyriel (a base model) and https://civitai.com/models/55543/dragonscaleai (a style).

Test 1 - lyriel

Running python convert_original_stable_diffusion_to_diffusers.py --checkpoint_path lyriel_v16.safetensors --from_safetensors --device cpu --extract_ema --dump_path lyriel_v16_diffusers reports global_step key not found in model spins for half a minute, and then spits out a few hundred lines of output. It does create a lyriel_v16_diffusers directory.

Running python -m python_coreml_stable_diffusion.torch2coreml --compute-unit CPU_AND_GPU --convert-vae-decoder --convert-vae-encoder --convert-unet --unet-support-controlnet --convert-text-encoder --model-version lyriel_v16_diffusers --bundle-resources-for-swift-cli --attention-implementation ORIGINAL -o lyriel_v16_original && python -m python_coreml_stable_diffusion.torch2coreml --compute-unit CPU_AND_GPU --convert-unet --model-version lyriel_v16_diffusers --bundle-resources-for-swift-cli --attention-implementation ORIGINAL -o lyriel_v16_original runs for 20 seconds, then spits out this runtime error ...

Converting PyTorch Frontend ==> MIL Ops:  22%|████████                             | 80/369 [00:00<00:00, 1929.76 ops/s]
Traceback (most recent call last):
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/n/StableDiffusion/ml-stable-diffusion/python_coreml_stable_diffusion/torch2coreml.py", line 1311, in <module>
    main(args)
  File "/Users/n/StableDiffusion/ml-stable-diffusion/python_coreml_stable_diffusion/torch2coreml.py", line 1161, in main
    convert_vae_decoder(pipe, args)
  File "/Users/n/StableDiffusion/ml-stable-diffusion/python_coreml_stable_diffusion/torch2coreml.py", line 454, in convert_vae_decoder
    coreml_vae_decoder, out_path = _convert_to_coreml(
  File "/Users/n/StableDiffusion/ml-stable-diffusion/python_coreml_stable_diffusion/torch2coreml.py", line 125, in _convert_to_coreml
    coreml_model = ct.convert(
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/_converters_entry.py", line 492, in convert
    mlmodel = mil_convert(
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 188, in mil_convert
    return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 212, in _mil_convert
    proto, mil_program = mil_convert_to_proto(
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 285, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 108, in __call__
    return load(*args, **kwargs)
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 63, in load
    return _perform_torch_convert(converter, debug)
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 110, in _perform_torch_convert
    raise e
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 102, in _perform_torch_convert
    prog = converter.convert()
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 284, in convert
    convert_nodes(self.context, self.graph)
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 83, in convert_nodes
    raise RuntimeError(
RuntimeError: PyTorch convert function for op 'scaled_dot_product_attention' not implemented.

The lyriel_v16_original directory exists, but is empty.

Test 2 - dragonscaleai

Running python convert_original_stable_diffusion_to_diffusers.py --checkpoint_path Dra9onScaleAI.safetensors --from_safetensors --device cpu --extract_ema --dump_path Dra9onScaleAI_diffusers runs for a few seconds then spits out ...

global_step key not found in model
Traceback (most recent call last):
  File "convert_original_stable_diffusion_to_diffusers.py", line 131, in <module>
    pipe = download_from_original_stable_diffusion_ckpt(
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 1214, in download_from_original_stable_diffusion_ckpt
    converted_unet_checkpoint = convert_ldm_unet_checkpoint(
  File "/Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 385, in convert_ldm_unet_checkpoint
    new_checkpoint["time_embedding.linear_1.weight"] = unet_state_dict["time_embed.0.weight"]
KeyError: 'time_embed.0.weight'

No directory was created.

--

The real question I think is how my convert_from_ckpt.py has different lines 1086 and 1087 from yours :)
Any thoughts or advice?

@jrittvo
Copy link
Contributor

jrittvo commented Jun 11, 2023

Yes, that script is a good place to start. It comes from your diffusers python package.

Hugging Face updated difusers from 0.16.1 to 0.17.0 three days ago. As I mentioned in my first reply above, updates to diffusers have created problems in the past. Your diffusers folder is in /Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/. (You were in one of its subfolders when you found the script file.) Right next to the difusers folder there will be a diffusers-0.xx.x.dist-info folder with the version number instead of the 0.xx.x part. I just grabbed the new version and it will take me about an hour to swap it out and see if that is the culprit. While I experiment, please post back what that version number is, and we'll go from there.

You are running python 3.8.x when you do conversions, by the way. That python version is the one in your coreml_stable_diffusion environment. python 3.10.11 is what miniconda uses for its own base environment, and version 3.11.4 is, as you say, from brew, so that is the version you have outside of miniconda. The one that matters here is the 3.8.x and that is the version you want for this, so you are good there. I can explain more about the different versions and how they get called up if you want later on. Let's stick with the diffusers package issues for now though. Tell me the version you find that you have.

@designosis
Copy link
Author

Here are some more datapoints ...

A git pull in the ml-stable-diffusion directory shows Already up to date..

My conda info output ...

     active environment : base
    active env location : /Users/n/miniconda3
            shell level : 1
       user config file : /Users/n/.condarc
 populated config files :
          conda version : 23.5.0
    conda-build version : not installed
         python version : 3.10.11.final.0
       virtual packages : __archspec=1=arm64
                          __osx=13.3.1=0
                          __unix=0=0
       base environment : /Users/n/miniconda3  (writable)
      conda av data dir : /Users/n/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/n/miniconda3/pkgs
                          /Users/n/.conda/pkgs
       envs directories : /Users/n/miniconda3/envs
                          /Users/n/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.5.0 requests/2.29.0 CPython/3.10.11 Darwin/22.4.0 OSX/13.3.1
                UID:GID : 501:20
             netrc file : None
           offline mode : False

Here's my conda list output:

# Name                    Version                   Build  Channel
boltons                   23.0.0          py310hca03da5_0
brotlipy                  0.7.0           py310h1a28f6b_1002
bzip2                     1.0.8                h620ffc9_4
ca-certificates           2023.05.30           hca03da5_0
certifi                   2023.5.7        py310hca03da5_0
cffi                      1.15.1          py310h80987f9_3
charset-normalizer        2.0.4              pyhd3eb1b0_0
conda                     23.5.0          py310hca03da5_0
conda-content-trust       0.1.3           py310hca03da5_0
conda-package-handling    2.1.0           py310hca03da5_0
conda-package-streaming   0.8.0           py310hca03da5_0
cryptography              39.0.1          py310h834c97f_1
idna                      3.4             py310hca03da5_0
jsonpatch                 1.32               pyhd3eb1b0_0
jsonpointer               2.1                pyhd3eb1b0_0
libffi                    3.4.4                hca03da5_0
ncurses                   6.4                  h313beb8_0
openssl                   3.0.8                h1a28f6b_0
packaging                 23.0            py310hca03da5_0
pip                       23.0.1          py310hca03da5_0
pluggy                    1.0.0           py310hca03da5_1
pycosat                   0.6.4           py310h1a28f6b_0
pycparser                 2.21               pyhd3eb1b0_0
pyopenssl                 23.0.0          py310hca03da5_0
pysocks                   1.7.1           py310hca03da5_0
python                    3.10.11              hb885b13_3
python.app                3               py310h1a28f6b_0
readline                  8.2                  h1a28f6b_0
requests                  2.29.0          py310hca03da5_0
ruamel.yaml               0.17.21         py310h1a28f6b_0
ruamel.yaml.clib          0.2.6           py310h1a28f6b_1
setuptools                65.6.3          py310hca03da5_0
six                       1.16.0             pyhd3eb1b0_1
sqlite                    3.41.2               h80987f9_0
tk                        8.6.12               hb8d0fd4_0
toolz                     0.12.0          py310hca03da5_0
tqdm                      4.65.0          py310h33ce5c2_0
tzdata                    2023c                h04d1e81_0
urllib3                   1.26.16         py310hca03da5_0
wheel                     0.38.4          py310hca03da5_0
xz                        5.4.2                h80987f9_0
zlib                      1.2.13               h5a0b063_0
zstandard                 0.19.0          py310h80987f9_0

@designosis
Copy link
Author

@jrittvo Oh! Ok let's see. Running ls -lA /Users/n/miniconda3/envs/coreml_stable_diffusion/lib/python3.8/site-packages/ |grep diffusers shows diffusers-0.17.0.dist-info in the output. So version 0.17.0.

@jrittvo

This comment was marked as off-topic.

Sign in to view
@jrittvo

This comment was marked as off-topic.

Sign in to view
@jrittvo
Copy link
Contributor

jrittvo commented Jun 11, 2023

Bingo. I get your error using diffusers 0.17.0:

RuntimeError: PyTorch convert function for op 'scaled_dot_product_attention' not implemented.

Do you have about an hour now? If you can redo some of your setup, I will give you a file to use so that your miniconda environment uses package versions that have been working for people for a while. The new diffusers version may not be the only one that breaks things.

@designosis
Copy link
Author

Yes, the files above are https://civitai.com/models/22922/lyriel (a base model) and https://civitai.com/models/55543/dragonscaleai (a style).

Running convert_original_stable_diffusion_to_diffusers.py on Lyriel (base model) did produce content in the output folder, where Dragonscale (style) did not.

Running python_coreml_stable_diffusion.torch2coreml on that Lyriel model caused the runtime error listed above.

@designosis
Copy link
Author

designosis commented Jun 11, 2023 •
edited

Yes! I'd love to help. Chat somewhere?

@jrittvo
Copy link
Contributor

jrittvo commented Jun 11, 2023

Do you use the Mochi Diffusion discord?

@jrittvo
Copy link
Contributor

jrittvo commented Jun 11, 2023

This is not going to be that complicated. You know your way around this stuff I see. We just need to delete the miniconda coreml_stable_diffusion env folder and re-do that part with specific package versions. You'll need to add like 5 numbers to one file and then go back to following the same instructions from the wiki for a third time ;)

@designosis
Copy link
Author

I'm there now, in the #SUPPORT chat.

@jrittvo
Copy link
Contributor

jrittvo commented Jun 11, 2023

Ok. Let me go there. I'm jrrjrr there.

@jrittvo
Copy link
Contributor

jrittvo commented Jun 12, 2023

So the problem is diffusers version 0.17.0. Drop back to version 0.16.1 and the instructions in the wiki work again.

If anyone reads this and needs help in downgrading their diffusers version, ask over on the Mochi discord where there are a few people who can help you out.

@jrittvo
Copy link
Contributor

jrittvo commented Jun 15, 2023

With the releases of Apple's new ml-stable-diffusion 1.0.0 and their coremltools 7.0b1, and Hugging Face's diffusers 0.17.1, all packages that get pulled into a new (or fully updated) miniconda environment for doing model conversions per the Wiki are working together again.

@pgainullin pgainullin mentioned this issue Jun 30, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@designosis @godly-devotion @jrittvo