Hello! I am Kaiyue Wen. I am a senior-year undergraduate student in Yao's pilot class studying computer science and engineering at Tsinghua University. Here are my CV and Publications.
My research interest spreads broadly in machine learning, including theory and applications. I delve deeper into language models, exploring both macroscopic and microscopic attributes.
- Macroscopic Level. I am interested in better utilizing large language models, including but not limited to, improving interpretability, controllability, and reasoning ability, through building systems around LLMs through first-principle analysis and theoretical thinking.
- MIcroscopic Level. I am interested in understanding the training dynamics of large language models, including but not limited to, the generalization ability, the implicit bias, and the optimization dynamics of pretraining, through theoretical analysis and empirical study.
I believe that the two levels are closely related and mutually beneficial. I am applying for a PhD position starting in 2024. Please contact me through email if you are interested in my research!
- Sep, 2023 2 papers (Sharpness&Generalization, (Un)interpretability of Transformers) accepted at Neurips 2024! Sharpness&Generalization is received as oral.
- Sep, 2023 Receive silver medal for Yao Award (Top 4 in Yao's pilot class).
- Aug, 2023 Return to China for my senior year in Tsinghua.
- Jul, 2023 Visit Hawaii for ICML 2023! Always great to see old friends.
- Jun, 2023 Residual Permutation Test receive Major Revision from AoS.
- Jun, 2023 Visiting Tengyu Ma at Stanford!
- May, 2023 Visit Rwanda for ICLR 2023!
- Mar, 2023 Reviewing ICML for the first time!
- Mar, 2023 New preprint Solving LPN with Neural Networks on arxiv!
- Feb, 2023 Visiting Andrej Risteski at CMU!
- Jan, 2023 2 papers (Understanding SAM, Not Benign Overfitting) accepted at ICLR 2023!
- Dec, 2022 New preprint Residual Permutation Test on arxiv!
- Dec, 2022 New preprint Understanding SAM on arxiv!
- Oct, 2022 1 paper (Skill Neurons) accepted at EMNLP 2022.
- Jun, 2022 New preprint Not Benign Overfitting on arxiv!
I keep a firm faith in analytical thinking, hard work, and consistent self-improvement. Any advice or feedback is welcome. You can use this Anonymous Form or discuss with me in person.