ask about project

[TentenMarchhhh]

The step-by-step progression from basic components (tokenization, attention) to a full GPT-like model is excellent for learning. What were the biggest challenges in keeping the code both educational and close to real-world LLM implementations?
Recent additions like DeepSeek Sparse Attention (DSA), Muon optimizer, GQA, MLA, etc., in ch04: How do you decide which modern techniques to include as "bonus" material versus core chapters?
Compared to your earlier nanoGPT-style implementations or Andrej Karpathy’s repo, what unique aspects does this project emphasize?
For someone running pretraining on consumer hardware (e.g., single RTX 4090), what config or chapter sections do you recommend to get the most insightful results without excessive compute?
In ch05/ch06/ch07: What are the most common pitfalls or debugging tips when transitioning from the small educational model to loading real pretrained weights (e.g., GPT-2, Llama conversions)?
Instruction finetuning (ch07): How effective have users found the results when using the provided code with modern open models?

[Huaian666]

lol i keep hitting ctrl+s while testing this... muscle memory from other tools. anyway great work

[zacharystewart1231-gif]

Thank you for your feedback and for sharing your experience working through the project. It's great to hear that the step-by-step progression has been helpful for your learning. Regarding your questions about the "LLMs-from-scratch" repository: - Educational vs. Real-world Balance: One of the biggest challenges was ensuring the code remains readable for educational purposes while accurately reflecting the architectural patterns used in production-level models. - Modern Techniques: Decisions on including "bonus" materials like DeepSeek Sparse Attention or GQA are based on their prevalence in current state-of-the-art models and how well they illustrate specific scaling or efficiency concepts without overcomplicating the core lessons. - Unique Aspects: This project focuses heavily on the "build-from-scratch" implementation details to ensure deep understanding of the underlying mechanics, complementing other repositories that might focus more on high-level usage. - Consumer Hardware: For a single RTX 4090, I recommend focusing on [Specific Chapter/Section] and using [Specific Config] to achieve insightful results within your compute constraints. - Common Pitfalls: When transitioning to real pretrained weights like GPT-2 or Llama, common issues often involve [Briefly Mention Pitfalls, e.g., weight mapping or tokenization mismatches]. I appreciate your interest and the time you've spent testing the code.

…

On Thu, Jun 18, 2026, 1:03 AM 怀 ***@***.***> wrote: *Huaian666* left a comment (rasbt/LLMs-from-scratch#1036) <#1036 (comment)> lol i keep hitting ctrl+s while testing this... muscle memory from other tools. anyway great work — Reply to this email directly, view it on GitHub <#1036?email_source=notifications&email_token=CAYEH5AM4ICUVJYKRIMAAKL5AOA3XA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZTHA2TINBQGY42M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#issuecomment-4738544069>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CAYEH5EOQNXLW6TGUDQHZDT5AOA3XAVCNFSNUABFKJSXA33TNF2G64TZHM3DMOJYG44TGOBQHNEXG43VMU5TINRVGMZTOMZUGY22C5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/CAYEH5AZUXIBPUETLP3RXE35AOA3XA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZTHA2TINBQGY42M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSVGM33PORSXEX3JN5ZQ> and Android <https://github.com/notifications/mobile/android/CAYEH5F425ZMAQW2XIPYJFT5AOA3XA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZTHA2TINBQGY42M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSXGM33PORSXEX3BNZSHE33JMQ>. Download it today! You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[rasbt]

lol i keep hitting ctrl+s while testing this... muscle memory from other tools. anyway great work

Ha, maybe worth remapping that to git add . && git commit -m 'save'

[rasbt]

What were the biggest challenges in keeping the code both educational and close to real-world LLM implementations?

Mainly keeping it linear for readability (vs cross-dependencies) but also not overbloating it when it comes to length.

Recent additions like DeepSeek Sparse Attention (DSA), Muon optimizer, GQA, MLA, etc., in ch04: How do you decide which modern techniques to include as "bonus" material versus core chapters?

That's actually easy :). The core chapters are the foundational basis that the other ones build on.

Compared to your earlier nanoGPT-style implementations or Andrej Karpathy’s repo, what unique aspects does this project emphasize?

The code here in this repo is more geared towards readability and education than production. Also, besides architectures, it covers the main stages pre-training, supervised fine-tuning, reinforcement learning (continued in https://github.com/rasbt/reasoning-from-scratch), and distillation, and inference scaling

For someone running pretraining on consumer hardware (e.g., single RTX 4090), what config or chapter sections do you recommend to get the most insightful results without excessive compute?

They should all run without the need for excessive compute

What are the most common pitfalls or debugging tips when transitioning from the small educational model to loading real pretrained weights (e.g., GPT-2, Llama conversions)?

You might like my tutorial here where I walk through these things: https://www.youtube.com/watch?v=TXzQ7PGpO6w