Is it possible to train LoRa on model merged by ModelMergeSimple?

[EbisusBaka]

I recently merged 2 finetuned models, on both which I am able to train LoRas (with Kohya for example). But it's not possible to train LoRas on merged model. My workflow:

Error I'm getting during LoRa training on merged model:

  File "/content/kohya-trainer/train_network_xl_wrapper.py", line 10, in <module>
    trainer.train(args)
  File "/content/kohya-trainer/train_network.py", line 213, in train
    model_version, text_encoder, vae, unet = self.load_target_model(args, weight_dtype, accelerator)
  File "/content/kohya-trainer/sdxl_train_network.py", line 34, in load_target_model
    ) = sdxl_train_util.load_target_model(args, accelerator, sdxl_model_util.MODEL_VERSION_SDXL_BASE_V0_9, weight_dtype)
  File "/content/kohya-trainer/library/sdxl_train_util.py", line 33, in load_target_model
    ) = _load_target_model(
  File "/content/kohya-trainer/library/sdxl_train_util.py", line 70, in _load_target_model
    ) = sdxl_model_util.load_models_from_sdxl_checkpoint(model_version, name_or_path, device, weight_dtype)
  File "/content/kohya-trainer/library/sdxl_model_util.py", line 260, in load_models_from_sdxl_checkpoint
    info1 = text_model1.load_state_dict(te1_sd)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 2189, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(

RuntimeError: Error(s) in loading state_dict for CLIPTextModel:
	Missing key(s) in state_dict: "text_model.embeddings.position_ids". 

Is there any solution for this or you just simply can't train LoRas on model merges?

[mcmonkey4eva]

This error should be reported to kohya