Beginner Question: Merge 2 7B models to make 13B model

[fakerybakery]

Hi,
This is a beginner question, as I'm completely new to model merging. Is it possible to merge 2 7B models to create a larger 13B model?
Thank you

[cg123]

Yes, it is possible - people have had some pretty cool successes with this sort of model stacking lately.

For example, here's the config Undi95 used to make Mistral-11B-OpenOrcaPlatypus (one of the steps in his Mistral-11B-OmniMix):

slices:
  - sources:
    - model: Open-Orca/Mistral-7B-OpenOrca
      layer_range: [0, 24]
  - sources:
    - model: akjindal53244/Mistral-7B-v0.1-Open-Platypus
      layer_range: [8, 32]
merge_method: passthrough
dtype: bfloat16

[fakerybakery]

Great, thank you!