Skip to content

Latest commit

 

History

History
325 lines (294 loc) · 45.2 KB

README.md

File metadata and controls

325 lines (294 loc) · 45.2 KB

NLP, CV, Deep Learning Paper Reading Table

I read these papers that are related to NLP and Deep Learning. Here are various papers from basic to advanced. 😊 In addition, you can check my Korean paper reviews by clicking the link attached to the table. 😉

You can see more paper reviews, code implementation, and mathematics descriptions in my blog <- click here

My Insight 🧐

I write several articles to explain in detail some Deep Learning technologies. These articles can be found in the table below.

Title Blog link
How has scaling law developed in NLP? 🤔 https://cartinoe5930.tistory.com/entry/How-has-scaling-law-developed-in-NLP-%F0%9F%A4%94-NLP%EC%97%90%EC%84%9C-scaling-law%EB%8A%94-%EC%96%B4%EB%96%BB%EA%B2%8C-%EB%B0%9C%EC%A0%84%EB%90%98%EC%97%88%EC%9D%84%EA%B9%8C
Closed-source🔒? Open-source🔓? What is that?? 🤨🤔 https://cartinoe5930.tistory.com/entry/The-hopes-of-researchers-Open-source-%F0%9F%A4%97-%EC%97%B0%EA%B5%AC%EC%9E%90%EB%93%A4%EC%9D%98-%ED%9D%AC%EB%A7%9D-Open-source-%F0%9F%A4%97
Context window of LM, should it be long? Should it be short? 📏🤨 https://cartinoe5930.tistory.com/entry/LM%EC%9D%98-context-window-%EA%B8%B8%EC%96%B4%EC%95%BC-%ED%95%A0%EA%B9%8C-%EC%A7%A7%EC%95%84%EC%95%BC-%ED%95%A0%EA%B9%8C-%F0%9F%93%8F%F0%9F%A4%A8
What is the most optimal way to evaluate LM? 😎 https://cartinoe5930.tistory.com/entry/LM%EC%9D%84-%EA%B0%80%EC%9E%A5-%EC%B5%9C%EC%A0%81%EC%9C%BC%EB%A1%9C-%ED%8F%89%EA%B0%80%ED%95%A0-%EC%88%98-%EC%9E%88%EB%8A%94-%EB%B0%A9%EB%B2%95%EC%9D%80-%EB%AC%B4%EC%97%87%EC%9D%BC%EA%B9%8C-%F0%9F%98%8E
The performance of ChatGPT is getting worse?!?!? 😲😲 https://cartinoe5930.tistory.com/entry/ChatGPT%EC%9D%98-%EC%84%B1%EB%8A%A5%EC%9D%B4-%EC%95%88-%EC%A2%8B%EC%95%84%EC%A7%80%EA%B3%A0-%EC%9E%88%EB%8B%A4%EA%B5%AC-%F0%9F%98%B2%F0%9F%98%B2
You can fine-tune too! with PEFT 🤗 https://cartinoe5930.tistory.com/entry/%EB%8B%B9%EC%8B%A0%EB%8F%84-Fine-tuning-%ED%95%A0-%EC%88%98-%EC%9E%88%EC%8A%B5%EB%8B%88%EB%8B%A4-with-PEFT-%F0%9F%A4%97
Let's think step by step like humans! 🧠🤔 https://cartinoe5930.tistory.com/entry/%ED%95%9C-%EB%8B%A8%EA%B3%84-%ED%95%9C-%EB%8B%A8%EA%B3%84%EC%94%A9-%EC%9D%B8%EA%B0%84%EC%B2%98%EB%9F%BC-%EC%83%9D%EA%B0%81%ED%95%B4%EB%B3%B4%EC%9E%90-%F0%9F%A7%A0%F0%9F%A4%94
Development process of fine-tuning method!! From fine-tuning to RLHF 🦖➡️🧑 https://cartinoe5930.tistory.com/entry/Fine-tuning-method%EC%9D%98-%EC%A7%84%ED%99%94-%EA%B3%BC%EC%A0%95-%F0%9F%A6%96%E2%9E%A1%EF%B8%8F%F0%9F%A7%91
It's time to fine-tune ChatGPT!! ⏰ https://cartinoe5930.tistory.com/entry/%EC%9D%B4%EC%A0%9C%EB%8A%94-ChatGPT%EB%A5%BC-fine-tuning-%ED%95%A0-%EC%8B%9C%EA%B0%84-%E2%8F%B0
Noise makes LLM better! - NEFTune 😉 https://cartinoe5930.tistory.com/entry/Noise-makes-LLM-better-NEFTune-%F0%9F%98%89

Natural Language Processing

Word Embedding & Neural Networks

Paper Title Paper or reference site Link Paper Review
Embedding Matrix https://wikidocs.net/book/2155 https://cartinoe5930.tistory.com/entry/Embedding-Matrix-%ED%95%99%EC%8A%B5
LSTM: Long-Short Term Memory https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://cartinoe5930.tistory.com/entry/%EC%95%8C%EA%B8%B0-%EC%89%BD%EA%B2%8C-LSTM-networks-%EC%9D%B4%ED%95%B4%ED%95%98%EA%B8%B0
GRU: Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation https://arxiv.org/abs/1406.1078 https://cartinoe5930.tistory.com/entry/GRU-Empirical-Evaluation-of-Gated-Recurrent-Neural-Networks-on-Sequence-Modeling-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
LSTM vs. GRU: Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling https://arxiv.org/abs/1412.3555 https://cartinoe5930.tistory.com/entry/LSTM-vs-GRU-%EB%AD%90%EA%B0%80-%EB%8D%94-%EB%82%98%EC%9D%84%EA%B9%8C-Empirical-Evaluation-of-Gated-Recurrent-Neural-Networks-on-Sequence-Modeling-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0

Language Models🤖

Basic📖

Paper Title Paper or reference site Link Paper Review
Transformer: Attention Is All You Need https://arxiv.org/abs/1706.03762 https://cartinoe5930.tistory.com/entry/Transformer-Attention-Is-All-You-Need-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
ELMo: Deep contextualized word representations https://arxiv.org/abs/1802.05365 https://cartinoe5930.tistory.com/entry/Pre-trained-Language-Modeling-paper-reading1-ELMo-Deep-contextualized-word-representations
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding https://arxiv.org/abs/1810.04805 https://cartinoe5930.tistory.com/entry/Pre-trained-Language-Modeling-paper-reading2-BERT-Pre-training-of-Deep-Bidirectional-Transformers-for-Language-Understanding
GPT-1: Improving Language Understanding by Generative Pre-Training https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf https://cartinoe5930.tistory.com/entry/Pre-trained-Language-Modeling-paper-reading3-GPT-1-Improving-Language-Understanding-by-Generative-Pre-Training
GPT-2: Language Models are Unsupervised Multitask Learners https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://cartinoe5930.tistory.com/entry/GPT-2-Language-Models-are-Unsupervised-Multitask-Learners-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
GPT-3: Language Models are Few-Shot Learners https://cartinoe5930.tistory.com/entry/GPT-3-Language-Models-are-Few-Shot-Learners-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0 https://cartinoe5930.tistory.com/entry/GPT-3-Language-Models-are-Few-Shot-Learners-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context https://arxiv.org/abs/1901.02860 https://cartinoe5930.tistory.com/entry/Transformer-XL-Attentive-Language-Models-Beyond-a-Fixed-Length-Context-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Sparse Transformers: Generating Long Sequences with Sparse Transformers https://arxiv.org/abs/1904.10509 https://cartinoe5930.tistory.com/entry/Sparse-Transformers-Generating-Long-Sequence-with-Sparse-Transformers-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
XLNET: Generalized Autoregressive Pretraining for Language Understanding https://arxiv.org/abs/1906.08237 https://cartinoe5930.tistory.com/entry/XLNet-Generalized-Autoregressive-Pretraining-for-Language-Understanding-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
SpanBERT: Improving Pre-training by Representing and Predicting Spans https://arxiv.org/abs/1907.10529 https://cartinoe5930.tistory.com/entry/SpanBERT-Improving-Pre-training-by-Representing-and-Predicting-Spans-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
RoBERTa: A Robustly Optimized BERT Pre-training Approach https://arxiv.org/abs/1907.11692 https://cartinoe5930.tistory.com/entry/RoBERTa-A-Robustly-Optimized-BERT-Pretraining-Approach-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks https://arxiv.org/abs/1908.10084 https://cartinoe5930.tistory.com/entry/Sentence-BERT-Sentence-Embeddings-using-Siamese-BERT-Networks-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations https://arxiv.org/abs/1909.11942 https://cartinoe5930.tistory.com/entry/ALBERT-A-Lite-BERT-for-Self-supervised-Learning-of-Language-Representations-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension https://arxiv.org/abs/1910.13461 https://cartinoe5930.tistory.com/entry/BART-Denoising-Sequence-to-Sequence-Pre-training-for-Natural-Language-Generation-Translation-and-Comprehension-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Pre-LN Transformer: On Layer Normalization in the Transformer Architecture https://arxiv.org/abs/2002.04745 https://cartinoe5930.tistory.com/entry/Pre-LN-Transformer-On-Layer-Normalization-in-the-Transformer-Architecture-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
ELECTRA: Pre-training Text Encoders as Discriminators rather than Generators https://arxiv.org/abs/2003.10555 https://cartinoe5930.tistory.com/entry/ELECTRA-Pre-training-Text-Encoders-as-Discriminators-rather-than-Generators
Longformer: The Long-Document Transformer https://arxiv.org/abs/2004.05150 https://cartinoe5930.tistory.com/entry/Longformer-The-Long-Document-Transformer-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
BigBird: Transformers for Longer Sequences https://arxiv.org/abs/2007.14062 https://cartinoe5930.tistory.com/entry/BigBird-Transformers-for-Longer-Sequences-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
WebGPT: Browser-assisted question-answering with human feedback https://arxiv.org/abs/2112.09332 https://cartinoe5930.tistory.com/entry/WebGPT-Browser-assisted-question-answering-with-human-feedback-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
OPT: Open Pre-trained Transformer Language Models https://arxiv.org/abs/2205.01068 https://cartinoe5930.tistory.com/entry/OPT-Open-Pre-trained-Transformer-Language-Models-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Mamba: Linear-Time Sequence Modeling with Selective State Spaces https://arxiv.org/abs/2312.00752 No plan!

Efficient Models💸

Paper Title Paper or reference site Link Paper Review
TinyBERT: Distilling BERT for Natural Language Understanding https://arxiv.org/abs/1909.10351 https://cartinoe5930.tistory.com/entry/TinyBERT-Distilling-BERT-for-Natural-Language-Understanding-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
DistilBERT: a distilled version of BERT https://arxiv.org/abs/1910.01108 https://cartinoe5930.tistory.com/entry/DistilBERT-a-distilled-version-of-BERT-smaller-faster-cheaper-and-lighter-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
It's Not Just Size That Matters: Small Language Models are Also Few-Shot Learners(PET 응용) https://arxiv.org/abs/2009.07118 https://cartinoe5930.tistory.com/entry/Its-Not-Just-Size-That-Matters-Small-Language-Models-Are-Also-Few-Shot-Learners-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0

Open-source Language Model(Scaling law)🤗

Paper Title Paper or reference site Link Paper Review
Chinchilla: Training Compute-Optimal Large Language Models https://arxiv.org/abs/2203.15556 https://cartinoe5930.tistory.com/entry/%EC%A7%80%EA%B8%88-%EA%B9%8C%EC%A7%80%EC%9D%98-LM-Scaling-Law%EC%97%90%EB%8A%94-%EB%AC%B8%EC%A0%9C%EC%A0%90%EC%9D%B4-%EC%9E%88%EB%8B%A4-%F0%9F%98%B6%E2%80%8D%F0%9F%8C%AB%EF%B8%8F-Chinchilla-Training-Compute-Optimal-Large-Language-Models-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling https://arxiv.org/abs/2304.01373 No plan!
LIMA: Less Is More for Alignment https://arxiv.org/abs/2305.11206 https://cartinoe5930.tistory.com/entry/LIMA-Less-Is-More-for-Alignment-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
LLaMA: Open and Efficient Foundation Language Models https://arxiv.org/abs/2302.13971 https://cartinoe5930.tistory.com/entry/LLaMA-Open-and-Efficient-Foundation-Language-Models-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
WizardLM: Empowering Large Language Models to Follow Complex Instructions https://arxiv.org/abs/2304.12244 https://cartinoe5930.tistory.com/entry/Open-domain-instruction%EC%9D%98-%ED%9A%A8%EA%B3%BC-%F0%9F%AA%84-WizardLM-Empowering-Large-Language-Models-to-Follow-Complex-Instructions-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
WizardCoder: Empowering Code Large Language Models with Evol-Instruct https://arxiv.org/abs/2306.08568 https://huggingface.co/WizardLM/WizardCoder-15B-V1.0
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct https://arxiv.org/abs/2308.09583 https://huggingface.co/WizardLM/WizardMath-70B-V1.0
Alpaca: A Strong, Replicable Instruction-Following Model https://crfm.stanford.edu/2023/03/13/alpaca.html https://cartinoe5930.tistory.com/entry/Alpaca-A-Strong-Replicable-Instruction-Following-Model-%EB%A6%AC%EB%B7%B0
Vicuna: An Open-Source Chatbot Impressing GPT-4 https://lmsys.org/blog/2023-03-30-vicuna/ https://cartinoe5930.tistory.com/entry/Vicuna-An-Open-Source-Chatbot-Impressing-GPT-4-%EB%A6%AC%EB%B7%B0
Koala: A Dialogue Model for Academic Research https://bair.berkeley.edu/blog/2023/04/03/koala/ https://cartinoe5930.tistory.com/entry/%EC%A4%91%EC%9A%94%ED%95%9C-%EA%B1%B4-%EA%BA%BE%EC%9D%B4%EC%A7%80-%EC%95%8A%EB%8A%94-high-quality-data-Koala%F0%9F%90%A8-A-Dialogue-Model-for-Academic-Researc
Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data https://arxiv.org/abs/2304.01196 https://cartinoe5930.tistory.com/entry/%F0%9F%90%B2Baize-An-Open-Source-Chat-Model-with-Parameter-Efficient-Tuning-on-Self-Chat-Data-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Scaling Data-Constrained Language Models https://arxiv.org/abs/2305.16264 https://www.youtube.com/watch?v=TK0-sitkCMw&pp=ygUgaHR0cHM6Ly9hcnhpdi5vcmcvYWJzLzIzMDUuMTYyNjQ%3D
Falcon & RefinedWeb https://arxiv.org/abs/2306.01116 https://cartinoe5930.tistory.com/entry/Open-LLM-Leaderboard%EB%A5%BC-%ED%9C%A9%EC%93%B4-Falcon%F0%9F%A6%85-LLM-Falcon-RefinedWeb
Orca: Progressive Learning from Complex Explanation Traces of GPT-4 https://arxiv.org/pdf/2306.02707 https://cartinoe5930.tistory.com/entry/%F0%9F%90%ACOrca-Progressive-Learning-from-Complex-Explanation-Traces-of-GPT-4-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
phi-1: Text Books Are All You Need https://arxiv.org/abs/2306.11644 https://cartinoe5930.tistory.com/entry/%ED%95%84%EC%9A%94%ED%95%9C-%EA%B1%B4-%EC%98%A4%EC%A7%81-%EA%B5%90%EA%B3%BC%EC%84%9C-%EC%88%98%EC%A4%80%EC%9D%98-%EB%8D%B0%EC%9D%B4%ED%84%B0%EB%BF%90-%F0%9F%93%96-phi-1-Textbooks-Are-All-You-Need-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
AlpaGasus: Training a Better Alpaca with Fewer Data https://arxiv.org/abs/2307.08701 Will be uploaded later!
Llama 2: Open Foundation and Fine-Tuned Chat Models https://arxiv.org/abs/2307.09288 https://cartinoe5930.tistory.com/entry/The-hopes-of-researchers-Open-source-%F0%9F%A4%97-%EC%97%B0%EA%B5%AC%EC%9E%90%EB%93%A4%EC%9D%98-%ED%9D%AC%EB%A7%9D-Open-source-%F0%9F%A4%97
Platypus: Quick, Cheap, and Powerful Refinement of LLMs https://arxiv.org/abs/2308.07317 Will be uploaded later!
Code Llama: Open Foundation Models for Code https://arxiv.org/abs/2308.12950 No plan
FLM-101B: An Open LLM and How to Train It with $100K Budget https://arxiv.org/pdf/2309.03852 No plan!
Textbooks are All You Need II: phi-1.5 technical report https://arxiv.org/abs/2309.05463 https://huggingface.co/microsoft/phi-1_5
OpenChat: Advancing Open-Source Language Models with Mixed-Quality Data https://arxiv.org/abs/2309.11235 https://github.com/imoneoi/openchat
Mistral 7B https://arxiv.org/abs/2310.06825 https://mistral.ai/news/announcing-mistral-7b/
Prometheus: Inducing Fine-grained Evaluation Capability in Language Models https://arxiv.org/abs/2310.08491 https://huggingface.co/papers/2310.08491#652a8e7f30355beba68c1be6
Zephyr: Direct Distillation of LM Alignment https://arxiv.org/abs/2310.16944 https://www.youtube.com/watch?v=TkZBg3mKsIo
Orca2: Teaching Small Language Models How to Reason https://arxiv.org/abs/2311.11045 https://www.microsoft.com/en-us/research/blog/orca-2-teaching-small-language-models-how-to-reason/
The Falcon Series of Open Language Models https://arxiv.org/abs/2311.16867 No plan!
SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling https://arxiv.org/abs/2312.15166 No plan!

Large Language Models(LLMs)💣

Paper Title Paper or reference site Link Paper Review
LaMDA: Language Models for Dialog Applications blog: https://ai.googleblog.com/2022/01/lamda-towards-safe-grounded-and-high.html, paper: https://arxiv.org/abs/2201.08239 https://cartinoe5930.tistory.com/entry/%EA%B5%AC%EA%B8%80%EC%9D%98-%EC%B5%9C%EA%B0%95-%EC%B1%97%EB%B4%87-LaMDA%EC%97%90-%EB%8C%80%ED%95%B4-%EC%95%8C%EC%95%84%EB%B3%B4%EC%9E%90-Language-Models-for-Dialog-Applications-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
PaLM: Scaling Language Modeling with Pathways blog: https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html, paper: https://arxiv.org/abs/2204.02311 1: https://cartinoe5930.tistory.com/entry/LaMDA%EC%9D%98-%EB%92%A4%EB%A5%BC-%EC%9E%87%EB%8A%94-Pathways%EB%A5%BC-%ED%99%9C%EC%9A%A9%ED%95%9C-%EC%B4%88%EA%B1%B0%EB%8C%80-%EC%96%B8%EC%96%B4-%EB%AA%A8%EB%8D%B8-PaLM-%EB%A6%AC%EB%B7%B0, 2: https://cartinoe5930.tistory.com/entry/LaMDA%EC%9D%98-%EB%92%A4%EB%A5%BC-%EC%9E%87%EB%8A%94-Pathways%EB%A5%BC-%EC%82%AC%EC%9A%A9%ED%95%9C-%EC%B4%88%EA%B1%B0%EB%8C%80-%EC%96%B8%EC%96%B4-%EB%AA%A8%EB%8D%B8-PaLM-%EB%A6%AC%EB%B7%B02
GPT-4: Technical Review blog: https://openai.com/research/gpt-4, paper: https://arxiv.org/abs/2303.08774 https://cartinoe5930.tistory.com/entry/GPT-4-Techinal-Report-Review
Gemini: A Family of Highly Capable Multimodal Models https://arxiv.org/abs/2312.11805 No plan!
AlphaCode 2 Technical Report https://storage.googleapis.com/deepmind-media/AlphaCode2/AlphaCode2_Tech_Report.pdf No plan!

Fine-tuning

Instruction-tuning🧑‍🏫

Paper Title Paper or reference site Link Paper Review
FLAN: Fine-tuned Language Models are Zero-shot Learners https://arxiv.org/abs/2109.01652 https://cartinoe5930.tistory.com/entry/FLAN-Fine-tuned-Language-Models-are-Zero-shot-Learners-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
T0: Multitask Prompted Training Enables Zero-shot Task Generalization https://arxiv.org/abs/2110.08207 https://cartinoe5930.tistory.com/entry/T0-Multitask-Prompted-Training-Enables-Zero-shot-Task-Generalization-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Super-Natural Instructions: Generalization via Declarative Instructions on 1600+ NLP Tasks https://arxiv.org/abs/2204.07705 https://cartinoe5930.tistory.com/entry/Super-Natural-Instructions-Generalization-via-Declarative-Instructions-on-1600-NLP-Tasks-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Unnatural Instructions: Tuning Language Models with (Almost) Not Human Labor https://arxiv.org/abs/2212.09689 Will be uploaded later!
Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-shot Learners https://arxiv.org/abs/2210.02969 https://cartinoe5930.tistory.com/entry/Guess-the-Instruction-Flipped-Learning-Makes-Language-Models-Stronger-Zero-shot-Learners-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Scaling Instruction-Finetuned Language Models https://arxiv.org/abs/2210.11416 https://cartinoe5930.tistory.com/entry/Scaling-Instruction-Finetuned-Language-Models-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Exploring the Benefits of Training Expert Language Models over Instruction Tuning https://arxiv.org/abs/2302.03202 https://cartinoe5930.tistory.com/entry/Exploring-the-Benefits-of-Training-Expert-Language-Models-over-Instruction-Tuning-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
ICIL: In-Context Instruction Learning https://arxiv.org/abs/2302.14691 https://cartinoe5930.tistory.com/entry/ICIL-In-Context-Instruction-Learning-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Instruction tuning with GPT-4 https://arxiv.org/abs/2304.03277 https://cartinoe5930.tistory.com/entry/Instruction-Tuning-with-GPT-4-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
FIP: Fixed Input Parameterization for Efficient Prompting https://aclanthology.org/2023.findings-acl.533.pdf Will be uploaded later!
FlaCuna: unleashin the Problem Solving Power of Vicuna using FLAN Fine-tuning https://arxiv.org/abs/2307.02053 Will be uploaded later!
Maybe Only 0.5% Data Is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning https://arxiv.org/abs/2305.09246 Will be uploaded later!
Becoming self-instruct: introducing early stopping criteria for minimal instruct tuning https://arxiv.org/abs/2307.03692 Will be uploaded later!

Reinforcement Learning from Human Feedback(RLHF)👥

Paper Title Paper or reference site Link Paper Review
RLHF(Reinforcement Learning from Human Feedback) https://huggingface.co/blog/rlhf https://cartinoe5930.tistory.com/entry/%EC%82%AC%EB%9E%8C%EC%9D%98-%ED%94%BC%EB%93%9C%EB%B0%B1%EC%9D%84-%ED%86%B5%ED%95%9C-%EA%B0%95%ED%99%94%ED%95%99%EC%8A%B5-Reinforcement-Learning-from-Human-Feedback-RLHF
Red Teaming Language Models with Language Models https://arxiv.org/abs/2202.03286 https://cartinoe5930.tistory.com/entry/Red-Teaming-Language-Models-with-Language-Models-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
InstructGPT: Training language models to follow instructions with human feedback https://arxiv.org/abs/2203.02155 https://cartinoe5930.tistory.com/entry/InstructGPT-Training-language-models-to-follow-instructions-with-human-feedback-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Training a helpful and harmless assistant with reinforcement learning from human feedback https://arxiv.org/abs/2204.05862 https://cartinoe5930.tistory.com/entry/Training-a-helpful-and-harmless-assistant-with-reinforcement-learning-from-human-feedback-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback https://arxiv.org/abs/2305.14387 Will be uploaded later!
ALMoST: Aligning Large Language Models through Synthetic Feedback https://arxiv.org/abs/2305.13735 https://cartinoe5930.tistory.com/entry/Aligning-Large-Language-Models-through-Synthetic-Feedback-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback https://arxiv.org/abs/2307.15217 Will be uploaded later!
RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback https://arxiv.org/abs/2309.00267 No plan!
SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF https://arxiv.org/abs/2310.05344 No plan!
HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM https://arxiv.org/abs/2311.09528 No plan!

Efficient-tuning ✨

Paper Title Paper or reference site Link Paper Review
Adapter: Parameter-Efficient learning for NLP https://arxiv.org/abs/1902.00751 https://cartinoe5930.tistory.com/entry/%EB%8B%B9%EC%8B%A0%EB%8F%84-Fine-tuning-%ED%95%A0-%EC%88%98-%EC%9E%88%EC%8A%B5%EB%8B%88%EB%8B%A4-with-PEFT-%F0%9F%A4%97
Prefix-Tuning: Optimizing Continuous Prompts for Generation https://arxiv.org/abs/2101.00190 https://cartinoe5930.tistory.com/entry/%EB%8B%B9%EC%8B%A0%EB%8F%84-Fine-tuning-%ED%95%A0-%EC%88%98-%EC%9E%88%EC%8A%B5%EB%8B%88%EB%8B%A4-with-PEFT-%F0%9F%A4%97
LoRA: Low-Rank Adaptation of Large Language Models https://arxiv.org/abs/2106.09685 https://cartinoe5930.tistory.com/entry/%EB%8B%B9%EC%8B%A0%EB%8F%84-Fine-tuning-%ED%95%A0-%EC%88%98-%EC%9E%88%EC%8A%B5%EB%8B%88%EB%8B%A4-with-PEFT-%F0%9F%A4%97
Towards a Unified View of Parameter-Efficient Transfer Learning https://arxiv.org/abs/2110.04366 Will be uploaded later!
UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning https://arxiv.org/abs/2110.07577 Will be uploaded later!
(IA)^3: Few-Shot Parameter-Efficient Fine-TUning is Better and Cheaper than In-Context Learning https://arxiv.org/abs/2205.05638 Will be uploaded later!
QLoRA: Efficient Fine-tuning of Quantized LLMs https://arxiv.org/abs/2305.14314 Will be uploaded later!
Stack More Layers Differently: High-Rank Training Through Low-Rank Updates https://arxiv.org/abs/2307.05695 Will be uploaded later!
LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition https://arxiv.org/abs/2307.13269 Will be uploaded later!

Dataset 💫

Paper Title Paper or reference site Link Paper Review
Instruction Mining: High-quality Instruction Data Selection for Large Language Models https://arxiv.org/abs/2307.06290 No plan!
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization https://arxiv.org/abs/2212.10465 No plan!
MoDS: Model-oriented Data Selection for Instruction Tuning https://arxiv.org/abs/2311.15653 No plan!
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models https://arxiv.org/abs/2312.06585 No plan!
Magicoder: Source Code Is All You Need https://arxiv.org/abs/2312.02120 No plan!
WaveCoder: Widespread and Versatile Enhanced Instruction Tuning with Refined Data Generation https://arxiv.org/abs/2312.14187 No plan!
What Makes Good Data for Alignment: A Comprehensive Study of Automatic Data Selection in Instruction Tuning https://arxiv.org/abs/2312.15685 No plan!

Prompt Engineering 👨‍🔧

Paper Title Paper or reference site Link Paper Review
What is the 'Prompt Engineering'? See my blog! https://cartinoe5930.tistory.com/entry/Prompt-Engineering%EC%9D%B4-%EB%AC%B4%EC%97%87%EC%9D%BC%EA%B9%8C
CoT: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models blog: https://ai.googleblog.com/2022/05/language-models-perform-reasoning-via.html, paper: https://arxiv.org/abs/2201.11903 https://cartinoe5930.tistory.com/entry/LM%EC%9D%B4-%EC%82%AC%EB%9E%8C%EA%B3%BC-%EC%9C%A0%EC%82%AC%ED%95%9C-%EC%83%9D%EA%B0%81-%ED%94%84%EB%A1%9C%EC%84%B8%EC%8A%A4%EB%A5%BC-%EA%B0%80%EC%A7%80%EA%B2%8C-%EB%90%9C%EB%8B%A4%EB%A9%B4-Chain-of-Thought-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Zero-shot CoT: Large Language Models Are Zero-shot Reasoners https://arxiv.org/abs/2205.11916 https://cartinoe5930.tistory.com/entry/Large-Language-Models-are-Zero-Shot-Reasoners-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Language Models are Multilingual Chain-of-Thought Reasoners https://arxiv.org/abs/2210.03057 Will be uploaded later!
Auto-CoT: Automatic Chain of Thought Prompting in Large Language Models https://arxiv.org/abs/2210.03493 Will be uploaded later!
CoT KD: Teaching Small Language Models to Reason https://arxiv.org/abs/2212.08410 Will be uploaded later!
ToT: Tree of Thoughts: Deliberate Problem Solving with Large Language Models https://arxiv.org/abs/2305.10601 https://cartinoe5930.tistory.com/entry/Tree-of-Thoughts-Deliberate-Problem-Solving-with-Large-Language-Models-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning https://arxiv.org/abs/2305.14045 https://cartinoe5930.tistory.com/entry/CoT-Collection-Improving-Zero-shot-and-Few-shot-Learning-of-Language-Models-via-Chain-of-Thought-Fine-tuning-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Let's verify step-by-step https://arxiv.org/abs/2305.20050 https://cartinoe5930.tistory.com/entry/Lets-verify-step-by-step-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Measuring Faitfulness in Chain-of-Thought Reasoning https://arxiv.org/abs/2307.13702 Will be uploaded later!
SoT: Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding https://arxiv.org/abs/2307.15337 Will be uploaded later!
Graph of Thoughts: Solving Elaborate Problems with Large Language Models https://arxiv.org/abs/2308.09687 Will be uploaded later!
From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting https://arxiv.org/abs/2309.04269 No plan!
Chain-of-Verification Resuces Hallucination in Large Language Models https://arxiv.org/abs/2309.11495 https://www.youtube.com/watch?v=l0zFjwRegog&pp=ygUgaHR0cHM6Ly9hcnhpdi5vcmcvYWJzLzIzMDkuMTE0OTU%3D
Contrastive Chain-of-Thought Prompting https://arxiv.org/abs/2311.09277 No plan!
Thread of Thought Unraveling Chaotic Contexts https://arxiv.org/abs/2311.08734 No plan!
System 2 Attention (Is Something You Might Need Too) https://arxiv.org/abs/2311.11829 No plan!
Chain of Code: Reasoning with a Language Model-Augmented Code Emulator https://arxiv.org/abs/2312.04474 No plan!

Model Efficiency 🧰

Paper Title Paper Paper Review
FlashAttention: Fast and Memory-Efficient Exact Attention https://arxiv.org/abs/2205.14135 https://gordicaleksa.medium.com/eli5-flash-attention-5c44017022ad
Exponentially Faster Language Modeling https://arxiv.org/abs/2311.10770 No plan!
LLM in a flash: Efficient Large Language Model Inference with Limited Memory https://arxiv.org/abs/2312.11514 No plan!

Method 📐

Paper Title Paper or reference site Link Paper Review
Data Augmentations in NLP blogs: https://neptune.ai/blog/data-augmentation-nlp, https://amitness.com/2020/05/data-augmentation-for-nlp/?fbclid=IwAR11MkccCti-2cD93RYftNPHb7Wxdj7AlZG7NNG4EhPaBkmiJkcBPtdl1eo https://cartinoe5930.tistory.com/entry/Data-Augmentation-methods-in-NLP
PET: Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference https://arxiv.org/abs/2001.07676 https://cartinoe5930.tistory.com/entry/PET-Exploiting-Cloze-Questions-for-Few-Shot-Text-Classification-and-Natural-Language-Inference-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Pathways https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/ https://cartinoe5930.tistory.com/entry/%EB%A7%8C%EC%95%BD-%EB%AA%A8%EB%8D%B8%EC%9D%B4-%EC%97%AC%EB%9F%AC-%EA%B0%90%EA%B0%81%EC%9D%84-%EB%8A%90%EB%82%84-%EC%88%98-%EC%9E%88%EA%B2%8C-%EB%90%9C%EB%8B%A4%EB%A9%B4-Pathways-%EB%A6%AC%EB%B7%B0
LMSI: Large Language Models Can Self-Improve https://arxiv.org/abs/2210.11610 https://cartinoe5930.tistory.com/entry/LMSI-Large-Language-Models-can-Self-Improve-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Self-Instruct: Aligning Language Model with Self Generated Instruction https://arxiv.org/abs/2212.10560 https://cartinoe5930.tistory.com/entry/Self-Instruct-Aligning-Language-Model-with-Self-Generated-Instructions-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Reflexion: Language Agents with Verbal Reinforcement Learning https://arxiv.org/abs/2303.11366 https://cartinoe5930.tistory.com/entry/Reflexion-Language-Agents-with-Verbal-Reinforcement-Learning-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Self-Refine: Iterative Refinement with Self-Feedback https://arxiv.org/abs/2303.17651 https://cartinoe5930.tistory.com/entry/Self-Refine-Iterative-Refinement-with-Self-Feedback-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
REFINER: Reasoning Feedback on Intermediate Representations https://arxiv.org/abs/2304.01904 No plan!
SelFee: Iterative Self-Revising LLM Expowered by Self-Feedback Generation https://kaistai.github.io/SelFee/ https://cartinoe5930.tistory.com/entry/SelFee-Iterative-Self-Revising-LLM-Expowered-by-Self-Feedback-Generation-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints https://arxiv.org/abs/2305.13245 https://aliissa99.medium.com/-a596e4d86f79
Shpherd: A Critic for Language Model Generation https://arxiv.org/abs/2308.04592 Will be uploaded later!
Self-Alignment with Instruction Backtranslation https://arxiv.org/pdf/2308.06259 Will be uploaded later!
SCREWS: A Modular Framework for Reasoning with Revisions https://arxiv.org/pdf/2309.13075 No plan!
NEFTune: Noisy Embeddings Improve Instruction Fineuning https://arxiv.org/abs/2310.05914 https://cartinoe5930.tistory.com/entry/Noise-makes-LLM-better-NEFTune-%F0%9F%98%89
Language Models are Super Mario; Absorbing Abilities from Homologous Models as a Free Lunch https://arxiv.org/abs/2311.03099 No plan!
LoRAMoE: Revolutionizing Mixture of Experts for Maintaining World Knowledge in Language Model Alignment https://arxiv.org/abs/2312.09979 No plan!

Retrieval Augmented Generation(RAG) 📚

Paper Title Paper or reference site Link Paper Review
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks https://arxiv.org/abs/2005.11401 No plan!
Self-RAG: Learning to Retrieve, Generate, And Critique Through Self-Reflection https://arxiv.org/abs/2310.11511 No plan!
InstructRetro: Instruction Tuning Post Retrieval-Augmented Pretraining https://arxiv.org/abs/2310.07713 No plan!
Retrieval-Augmented Generation for Large Language Models: A Survey https://arxiv.org/abs/2312.10997 No plan!

Benchmarks 🏆 & Evaluation Metric ⚔️

Paper Title Paper or reference site Link Paper Review
BIG-Bench Hard: Challenging BIG-Bench tasks and whether chain-of-thought can solve tham https://arxiv.org/abs/2210.09261 Will be uploaded later!
Large Language Models are not Fair Evaluators https://arxiv.org/abs/2305.17926 Will be uploaded later!
MT-Bench: Judging LLM-as-a-judge with MT-Bench https://arxiv.org/abs/2306.05685 Will be uploaded later!
InstructEval: Towards Holistic Evaluation of Instruction-Tuned Large Language Models https://arxiv.org/abs/2306.04757 Will be uploaded later!
FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets https://arxiv.org/abs/2307.10928 Will be uploaded later!
GAIA: A Benchmark for General AI Assistants https://arxiv.org/abs/2311.12983 No plan!

Context of LLM 📜

Paper Title Paper or reference site Link Paper Review
A Length-Extrapolatable Transformer https://arxiv.org/abs/2212.10554 No plan!
Extending Context Window of Large Language Models via Positional Interpolation https://arxiv.org/abs/2306.15595 https://cartinoe5930.tistory.com/entry/LM%EC%9D%98-context-window-%EA%B8%B8%EC%96%B4%EC%95%BC-%ED%95%A0%EA%B9%8C-%EC%A7%A7%EC%95%84%EC%95%BC-%ED%95%A0%EA%B9%8C-%F0%9F%93%8F%F0%9F%A4%A8
LongNet: Scaling Transformers to 1,000,000,000 Tokens https://arxiv.org/abs/2307.02486 https://cartinoe5930.tistory.com/entry/LM%EC%9D%98-context-window-%EA%B8%B8%EC%96%B4%EC%95%BC-%ED%95%A0%EA%B9%8C-%EC%A7%A7%EC%95%84%EC%95%BC-%ED%95%A0%EA%B9%8C-%F0%9F%93%8F%F0%9F%A4%A8
Lost in the Middle: How Language Models Use Long Contexts https://arxiv.org/abs/2307.03172 https://cartinoe5930.tistory.com/entry/LM%EC%9D%98-context-window-%EA%B8%B8%EC%96%B4%EC%95%BC-%ED%95%A0%EA%B9%8C-%EC%A7%A7%EC%95%84%EC%95%BC-%ED%95%A0%EA%B9%8C-%F0%9F%93%8F%F0%9F%A4%A8
YaRN: Efficient Context Window Extension of Large Language Models https://arxiv.org/abs/2309.00071 No plan!

Analysis🔬

Paper Title Paper or reference site Link Paper Review
Why can GPT learn in-context? https://arxiv.org/abs/2212.10559 https://cartinoe5930.tistory.com/entry/Why-can-GPT-learn-in-context-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Sparks of Artificial General Intelligence: Early experiments with GPT-4 paper: https://arxiv.org/abs/2303.12712, youtube: https://www.youtube.com/watch?v=Mqg3aTGNxZ0 https://cartinoe5930.tistory.com/entry/Sparks-of-Artificial-General-Intelligence-Early-experiments-with-GPT-4-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
The False Promise of Imitating Proprietary LLMs https://arxiv.org/abs/2305.15717 https://cartinoe5930.tistory.com/entry/%EA%B8%B0%EC%A1%B4-imitation-model%EC%9D%80-%EC%9E%98%EB%AA%BB-%ED%95%99%EC%8A%B5%EB%90%98%EA%B3%A0-%EC%9E%88%EB%8B%A4-%F0%9F%AB%A2-The-False-Promise-of-Imitating-Proprietary-L
TULU: How Far Can Camels Go? Exploring the State of Instructiopn Tuning on Open Resources https://arxiv.org/abs/2306.04751 Will be uploaded later!
How Is ChatGPT's Behavior Changing over Time? https://arxiv.org/abs/2307.09009 https://cartinoe5930.tistory.com/entry/ChatGPT%EC%9D%98-%EC%84%B1%EB%8A%A5%EC%9D%B4-%EC%95%88-%EC%A2%8B%EC%95%84%EC%A7%80%EA%B3%A0-%EC%9E%88%EB%8B%A4%EA%B5%AC-%F0%9F%98%B2%F0%9F%98%B2
Large Language Models Cannot Self-Correct Reasoning Yet https://arxiv.org/abs/2310.01798
How Far Are Large Language Models from Agents with Theory-of-Mind https://arxiv.org/pdf/2310.03051 No plan!
Can LLMs Follow Simple Rules https://arxiv.org/abs/2311.04235 https://www.youtube.com/watch?v=CY6o43037OY
Camels in a Changing Climate; Enhancing LM Adaptation with Tulu 2 https://arxiv.org/abs/2311.10702 No plan!
ChatGPT's One-year Anniversary; Are Open-Source Large Language Models Catching up https://arxiv.org/abs/2311.15653 No plan!
An In-depth Look at Gemini's Language Abilities https://arxiv.org/abs/2312.11444 No plan!

Interesting🫣

Paper Title Paper or reference site Link Paper Review
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature https://arxiv.org/abs/2301.11305 https://cartinoe5930.tistory.com/entry/%EC%9D%B4-%EA%B8%80%EC%9D%B4-LM%EC%9D%B4-%EB%A7%8C%EB%93%A4%EC%96%B4%EB%82%B8-%EA%B8%80%EC%9D%BC%EA%B9%8C-%EB%8F%84%EC%99%80%EC%A4%98-DetectGPT-DetectGPT-Zero-Shot-Machine-Generated-Text-Detection-using-Probability-Curvature-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback https://arxiv.org/abs/2302.12813 https://cartinoe5930.tistory.com/entry/ChatGPT%EC%9D%98-hallucination-%EC%96%B4%EB%96%BB%EA%B2%8C-%ED%95%B4%EA%B2%B0%ED%95%B4%EC%95%BC-%ED%95%A0%EA%B9%8C-Check-Your-Facts-and-Try-Again-Improving-Large-Language-Models-with-External-Knowledge-and-Automated-Feedback
RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text https://arxiv.org/abs/2305.13304 https://cartinoe5930.tistory.com/entry/ChatGPT%EC%97%90-%EB%B0%98%EB%B3%B5-%EB%A9%94%EC%BB%A4%EB%8B%88%EC%A6%98LSTM%EC%9D%84-%EC%82%AC%EC%9A%A9%ED%95%9C%EB%8B%A4%EB%A9%B4-RecurrentGPT-Interactive-Generation-of-Arbitrarily-Long-Text-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Large Language Models as Tool Makers https://arxiv.org/abs/2305.17126 https://cartinoe5930.tistory.com/entry/LM%EC%9D%B4-%EB%8F%84%EA%B5%AC%EB%A5%BC-%EC%82%AC%EC%9A%A9%ED%95%98%EA%B2%8C-%EB%90%9C%EB%8B%A4%EB%A9%B4-%F0%9F%94%AC-Large-Language-Models-as-Tool-Makers-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion https://arxiv.org/abs/2306.02561 No plan!
Knowledge Distillation of Large Language Models https://arxiv.org/abs/2306.08543 https://cartinoe5930.tistory.com/entry/KD%EC%97%90-%EC%82%B4%EC%A7%9D%EC%9D%98-%EB%B3%80%ED%99%94%EB%A5%BC-%EC%A4%98%EB%B3%B4%EC%9E%90-%F0%9F%98%9C-Knowledge-Distillation-of-Large-Language-Models-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Scaling Relationship on Learning Mathematical Reasoning with Large Language Models https://arxiv.org/abs/2308.01825 Will be uploaded later!
ToolLLM: Facilitating Lare Language Models to Master 16000+ Real-World APIs https://arxiv.org/abs/2307.16789 Will be uploaded later!
SelfCheck: Using LLMs to Zero-shot Check Their Own Step-by-Step Reasoning https://arxiv.org/abs/2308.00436 Will be uploaded later!
Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification https://arxiv.org/abs/2308.07921 Will be uploaded later!
Large Language Models as Optimizers https://arxiv.org/abs/2309.03409 No plan!
FIAT: Fusing Learning Paradigms with Instruction-Accelerated Tuning https://arxiv.org/abs/2309.04663 https://www.youtube.com/watch?v=EZsZEcRDte0&pp=ygUgaHR0cHM6Ly9hcnhpdi5vcmcvYWJzLzIzMDkuMDQ2NjM%3D
Contrastive Decoding Improves Reasoning in Large Language Models https://arxiv.org/abs/2309.09117 https://www.youtube.com/watch?v=nMR56TkwC1Q&pp=ygUgaHR0cHM6Ly9hcnhpdi5vcmcvYWJzLzIzMDkuMDkxMTc%3D
Think before you speak: Training Language Models with Pause Tokens https://arxiv.org/abs/2310.02226 https://www.youtube.com/watch?v=MtJ1jacr_yI
Large Language Models Can Learn Rules https://arxiv.org/abs/2310.07064 No plan!
In-context Pretraining: Language Modeling Beyond Document Boundaries https://arxiv.org/abs/2310.10638 https://www.youtube.com/watch?v=GI-0lAaILrU
Learning From Mistakes Makes LLM Better Reasoner https://arxiv.org/abs/2310.20689 No plan!
Language Models can be Logical Solvers https://arxiv.org/abs/2311.06158 No plan!
MART: Improving LLM Safety with Multi-round Automatic Red-Teaming https://arxiv.org/abs/2311.07689 No plan!
Fine-tuning Language Models for Factuality https://arxiv.org/abs/2311.08401 No plan!
Positional Description Matters for Transformers Arithmetic https://arxiv.org/abs/2311.14737 No plan!
Weak-to-Strong Generalization: Eliciting Strong Capabilities with Weak Supervision https://arxiv.org/abs/2312.09390 https://openai.com/research/weak-to-strong-generalization
TinyGSM: achieving higher than 80 percentage on GSM8k with small language models https://arxiv.org/abs/2312.09241 No plan!

Korean LM 🇰🇷

Paper Title Paper or reference site Link Paper Review
Morpheme-aware Subword Tokenizer: An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks https://arxiv.org/abs/2010.02534 Will be uploaded later!
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers https://arxiv.org/abs/2109.04650 Will be uploaded later!

Computer Vision

Paper Title Paper or reference site Link Paper Review
history of CNN LeNet, AlexNet, VGGNet, GoogLeNet, ResNet, ResNeXt, Sception, Mobilenet, DenseNet, EfficientNet, ConvNext https://cartinoe5930.tistory.com/entry/CNN-network%EC%9D%98-%EC%97%AD%EC%82%AC
ViT: An Image Worth 16 x 16 Words: Transformers for Image Recognition at Scale https://arxiv.org/abs/2010.11929 https://cartinoe5930.tistory.com/entry/ViT-An-Image-Worth-16-x-16-Words-Transformers-for-Image-Recognition-at-Scale
Swin Transformer: Hierarchical Vision Transformer using Shifted Winodws https://arxiv.org/abs/2103.14030 https://cartinoe5930.tistory.com/entry/Swin-Transformer-Hierarchical-Vision-Transformer-using-Shifted-Windows-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
CLIP: Learning Transferable Visual Models From Natural Language Supervision https://arxiv.org/abs/2103.00020 https://cartinoe5930.tistory.com/entry/CLIP-Learning-Transferable-Visual-Models-From-Natural-Language-Supervision-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0

Multi-modal Models

Paper Title Paper or reference site Link Paper Review
Let's learn about VLM(Visual-Language Model) https://huggingface.co/blog/vision_language_pretraining#supporting-vision-language-models-in-%F0%9F%A4%97-transformers https://cartinoe5930.tistory.com/entry/VLMVision-Language-Model%EC%97%90-%EB%8C%80%ED%95%B4-%EC%95%8C%EC%95%84%EB%B3%B4%EC%9E%90
VisualBERT: A simple and Performant Baseline for Vision and Language https://arxiv.org/abs/1908.03557 https://cartinoe5930.tistory.com/entry/VisualBERT-A-Simple-and-Performant-Baseline-for-Vision-and-Language-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
ViLBERT: Pre-training Task-Agnostic Visiolinguistic Representations for Visual-and-Language Tasks https://arxiv.org/abs/1908.02265 https://cartinoe5930.tistory.com/entry/ViLBERT-Pretraining-Task-Agnostic-Visiolinguistic-Representations-for-Visual-and-Language-Tasks
LXMERT: Learning Cross-Modality Encoder Representations from Transformers https://arxiv.org/abs/1908.07490 https://cartinoe5930.tistory.com/entry/LXMERT-Learning-Cross-Modality-Encoder-Representations-from-Transformers-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
VL-BERT: Pre-training of Generic Visual-Linguistic Representations https://arxiv.org/abs/1908.08530 https://cartinoe5930.tistory.com/entry/VL-BERT-Pre-training-of-Generic-Visual-Linguistic-Representations-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
VLP: Unified Vision-Language Pre-Training for Image Captioning and VQA https://arxiv.org/abs/1909.11059 https://cartinoe5930.tistory.com/entry/VLP-Unified-Vision-Language-Pre-Traning-for-Image-Captioning-and-VQA-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks https://arxiv.org/abs/2004.06165 https://cartinoe5930.tistory.com/entry/Oscar-Object-Semantics-Aligned-Pre-training-for-Vision-Language-Tasks-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
VinVL: Revisiting Visual Representations in Vision-Language Models https://arxiv.org/abs/2101.00529 https://cartinoe5930.tistory.com/entry/VinVL-Revisiting-Visual-Representations-in-Vision-Language-Models-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision https://arxiv.org/abs/2102.03334 https://cartinoe5930.tistory.com/entry/ViLT-Vision-and-Language-Transformer-Without-Convolution-or-Region-Supervision-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision https://arxiv.org/abs/2102.05918 https://cartinoe5930.tistory.com/entry/ALIGN-Scaling-up-Visual-and-Vision-Language-Representation-with-Noisy-Text-Supervision-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
ALBEF: Vision and Language Representation Learning with Momentum Distillation https://arxiv.org/abs/2107.07651 https://cartinoe5930.tistory.com/entry/ALBEF-Vision-and-Language-Representation-Learning-with-Momentum-Distillation-%EB%85%BC%EB%AC%B8
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision https://arxiv.org/abs/2108.10904 https://cartinoe5930.tistory.com/entry/SimVLM-Simple-Visual-Language-Model-Pre-training-with-Weak-Supervision-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
VLMo: Unified Vision-Language Pre-training with Mixture-of-Modality-Experts https://arxiv.org/abs/2111.02358 https://cartinoe5930.tistory.com/entry/VLMo-Unified-Vision-Language-Pre-training-with-Mixture-of-Modality-Experts-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
LiT🔥 : Zero-Shot Transfer with Locked-image text Tuning https://arxiv.org/abs/2111.07991 https://cartinoe5930.tistory.com/entry/LiT%F0%9F%94%A5-Zero-Shot-Transfer-with-Locked-image-text-Tuning-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
FLAVA: A Foundational Language And Vision Alignment Model https://arxiv.org/abs/2112.04482 https://cartinoe5930.tistory.com/entry/FLAVA-A-Foundational-Language-And-Vision-Alignment-Model-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation https://arxiv.org/abs/2201.12086 https://cartinoe5930.tistory.com/entry/BLIP-Bootstrapping-Language-Image-Pre-training-fro-Unified-Vision-Language-Understanding-and-Generation-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0

Deep Learning Concept

Paper or Posting Title reference site Link Review
Knowledge Distillation: Distilling the Knowledge in a Neural Network https://arxiv.org/abs/1503.02531 https://cartinoe5930.tistory.com/entry/Distilling-the-Knowledge-in-a-Neural-Network-%EB%85%BC%EB%AC%B8-%EB%A6%AC%EB%B7%B0
What is Zero-shot, One-shot, Few-shot Learning? see my blog! https://cartinoe5930.tistory.com/entry/Zero-shot-One-shot-Few-shot-Learning%EC%9D%B4-%EB%AC%B4%EC%97%87%EC%9D%BC%EA%B9%8C