NanoChat Distilled learning

[dustinwloring1988]

Has anyone attempted distillation with the larger of the model that @karpathy released and maybe using it when training a mode with the depth of say 12 or a nice power of 2 number like 8? Before someone says that I want a ChatGPT clone from doing that I am not that naive and know that if should only be a capability of the teacher model. If someone has or would like to work together on this please let me know. I think the results will be nice to share on here for people to learn from either way