Issues on the step "Converting Models from PyTorch to Core ML"

[HaskDev0]

While going through the steps listed in the description, the "Converting Models from PyTorch to Core ML" section fails on the Step 2, i.e., when I type:

python -m tests.torch2coreml.test_mmdit --sd3-ckpt-path ../sd3_medium_incl_clips.safetensors --model-version '8b' -o .../generatedResponsesWithStableDiffusion3Medium --latent-size 128

I get the following error:

======================================================================
ERROR: setUpClass (__main__.TestSD3MMDiT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../DiffusionKit-main/tests/torch2coreml/test_mmdit.py", line 69, in setUpClass
    _load_mmdit_weights(cls.test_torch_model, TEST_SD3_CKPT_PATH)
  File ".../DiffusionKit-main/python/src/torch/model_io.py", line 83, in _load_mmdit_weights
    raise ValueError(
ValueError: Total number of parameters in state_dict (2084877376) does not match the number of parameters in the module (8146086208)

----------------------------------------------------------------------
Ran 0 tests in 24.105s

FAILED (errors=1)

I see that there is a discrepancy in the number of module parameters, but the file I downloaded from HuggingFace is "sd3_medium_incl_clips.safetensors".

I'm using macOS Sonoma 14.3.1, Python 3.9.19

Are there any workarounds and suggestions for the issue? Thank you in advance.

[atiorh]

Hi @HaskDev0, please use the file without incl_clips: https://huggingface.co/stabilityai/stable-diffusion-3-medium/blob/main/sd3_medium.safetensors.

[HaskDev0]

Hi @HaskDev0, please use the file without incl_clips: https://huggingface.co/stabilityai/stable-diffusion-3-medium/blob/main/sd3_medium.safetensors.

Hi @atiorh , thanks for suggestion, but I tried the model without the _incl_clips in the meantime and still getting the same error...

[ZachNagengast]

@HaskDev0 I believe the issue is targeting 8b instead of 2b (medium) - here is a script that should work better

python -m tests.torch2coreml.test_mmdit --sd3-ckpt-path stabilityai/stable-diffusion-3-medium --model-version '2b' -o .../generatedResponsesWithStableDiffusion3Medium --latent-size 128

[HaskDev0]

@ZachNagengast , thanks for the suggestion, it does pass the error that I had before! Although it raises other errors (and one question)...

1. Do I understand right that there is no 8b model version available?

2. Now the error I get is the following, which specifically says my numpy version is the wrong one:

.../DiffusionKit-main/python/src/torch/mmdit.py:206: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  -torch.log(torch.tensor(self.config.max_period).to(t.dtype))
.../DiffusionKit-main/python/src/torch/mmdit.py:160: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert h <= self.max_hw and w <= self.max_hw
E
======================================================================
ERROR: setUpClass (__main__.TestSD3MMDiT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../DiffusionKit-main/tests/torch2coreml/test_mmdit.py", line 80, in setUpClass
    super().setUpClass()
  File ".../lib/python3.9/site-packages/argmaxtools/test_utils.py", line 55, in setUpClass
    cls.test_coreml_model = _create_coreml_model(
  File ".../lib/python3.9/site-packages/argmaxtools/test_utils.py", line 411, in _create_coreml_model
    inputs=[
  File ".../lib/python3.9/site-packages/argmaxtools/test_utils.py", line 412, in <listcomp>
    ct.TensorType(k, shape=v.shape, dtype=v.dtype)
  File ".../lib/python3.9/site-packages/coremltools/converters/mil/input_types.py", line 229, in __init__
    self.dtype = numpy_type_to_builtin_type(dtype)
  File ".../lib/python3.9/site-packages/coremltools/converters/mil/mil/types/type_mapping.py", line 363, in numpy_type_to_builtin_type
    if np.issubclass_(type(nptype), np.dtype):
  File ".../lib/python3.9/site-packages/numpy/__init__.py", line 397, in __getattr__
    raise AttributeError(
AttributeError: `np.issubclass_` was removed in the NumPy 2.0 release. Use `issubclass` builtin instead.

Should I reinstall numpy version or is there a long-term solution for future versions?

[ZachNagengast]

Yes this sounds like a dependency issue. Are you using the conda environment setup via https://github.com/argmaxinc/DiffusionKit?tab=readme-ov-file#python-environment-setup?

[HaskDev0]

@ZachNagengast , I was using venv for creating virtual environments, but I have resolved the issue by downgrading numpy version.

Finally, I was able to convert PyTorch models co CoreML, but as soon as I use diffusionkit-cli afterwards, it raises an error regarding HF access... Why is it trying to do so? I have the model locally and converted everything to CoreML through the steps provided?

Thank you in advance.

[ZachNagengast]

Ah great! The error is likely due to the gated access due to the StabilityAI license. All you need to do is go to the model page here https://huggingface.co/stabilityai/stable-diffusion-3-medium and accept the terms.

[HaskDev0]

@ZachNagengast , hmm, but if I already accepted the license, downloaded the .safetensors model, used it for converting to CoreML, why is it still trying to access HF? All the models are local now. Or how can I use CoreML generated models?

[ZachNagengast]

Yes just noticed something - if you are using a fine-grained token, you will need to also add the permission to read gated repos on the tokens page: https://huggingface.co/settings/tokens

Just updated the readme as well. You can also use --local-ckpt <path> for the local sd3_medium.safetensors

[HaskDev0]

@ZachNagengast , thank you! I recloned the repository to update the files, now with --local-ckpt argument it loads indeed (although it still in process), but I am confused a bit since it just redownloads all the .safetensors files from HF. Why does it do so? And where does it save all these big files? I cannot find them...

[ZachNagengast]

Tagging @arda-argmax who may be able to answer better

[arda-argmax]

@HaskDev0 , default save directory for huggingface is ~/.cache/huggingface/hub/. You can probably find the saved .safetensors file in ~/.cache/huggingface/hub/models--stabilityai--stable-diffusion-3-medium/. Can you write the full cli command that triggers the redownload?