time

Resources on ChatGPT and Large Language Models

Collection of papers and related works for Large Language Models (ChatGPT, GPT-3, Codex etc.).

Contributors

This repository is contributed by the following contributors.

• Organizers: Guilin Qi (漆桂林), Xiaofang Qi (戚晓芳)
• Paper Collectors: Zafar Ali, Sheng Bi (毕胜), Yongrui Chen (陈永锐), Zizhuo Chen (陈孜卓), Xinbang Dai (戴鑫邦), Huan Gao (高桓), Nan Hu (胡楠), Shilong Hu (胡世龙), Jingqi Kang (康婧淇), Jiaqi Li (李嘉琦), Dehai Min (闵德海), Guilin Qi (漆桂林), Yiming Tan (谭亦鸣), Tongtong Wu (吴桐桐), Songlin Zhai (翟松林), Shenyu Zhang (张沈昱), Yuxin Zhang (张裕欣)
• Maintainers: Runzhe Wang (王润哲), Shenyu Zhang (张沈昱)

The automation script of this repo is powered by Auto-Bibfile. If you'd like to commit to this repo, please modify bibtex.bib or related_works.json and re-generate README.md using python scripts/run.py.

This page categorizes the literature by the Last Post

Papers

Outline

• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 

Hyperlinks

• [Overview] -- Homepage
• -- Summary
• -- Author
• -- Techniques
• -- Published Time
• -- Published Venue

2024

• KB-Plugin: A Plug-and-play Framework for Large Language Models to Induce Programs over Low-resourced Knowledge Bases,
  by Jiajie Zhang, Shulin Cao, Linmei Hu, Ling Feng, Lei Hou and Juanzi Li
  
  
• Large Language Model Meets Graph Neural Network in Knowledge Distillation,
  by Shengxiang Hu, Guobing Zou, Song Yang, Yanglan Gan, Bofeng Zhang and Yixin Chen
  
  
• Do Large Language Models Latently Perform Multi-Hop Reasoning?,
  by Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva and Sebastian Riedel
  
  
• StructLM: Towards Building Generalist Models for Structured Knowledge Grounding,
  by Alex Zhuang, Ge Zhang, Tianyu Zheng, Xinrun Du, Junjie Wang, Weiming Ren, Stephen W. Huang, Jie Fu et al.
  
  
• Aligning Large Language Models to a Domain-specific Graph Database,
  by Yuanyuan Liang, Keren Tan, Tingyu Xie, Wenbiao Tao, Siyuan Wang, Yunshi Lan and Weining Qian
  
  
• Topologies of Reasoning: Demystifying Chains, Trees, and Graphs of Thoughts,
  by Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwaśniewski et al.
  
  
• InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integration,
  by Fali Wang, Runxue Bao, Suhang Wang, Wenchao Yu, Yanchi Liu, Wei Cheng and Haifeng Chen
  
  
• Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling,
  by Yang, Linyao, Chen, Hongyang, Li, Zhao, Ding, Xiao and Wu, Xindong
  
  
• GraphRAG: Unlocking LLM discovery on narrative private data,
  by Jonathan Larson, Steven Truitt
  
  
• GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph,
  by Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen and Xinchao Wang
  
  
• Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models,
  by Yubin Wang, Xinyang Jiang, De Cheng, Dongsheng Li and Cairong Zhao
  
  
• Unlocking the Power of Large Language Models for Entity Alignment,
  by Xuhui Jiang, Yinghan Shen, Zhichao Shi, Chengjin Xu, Wei Li, Zixuan Li, Jian Guo, Huawei Shen et al.
  
  
• Two Heads Are Better Than One: Integrating Knowledge from Knowledge Graphs and Large Language Models for Entity Alignment,
  by Linyao Yang, Hongyang Chen, Xiao Wang, Jing Yang, Fei-Yue Wang and Han Liu
  
  
• RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation,
  by Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma and Yitao Liang
  
  
• LLM Augmented LLMs: Expanding Capabilities through Composition,
  by Rachit Bansal, Bidisha Samanta, Siddharth Dalmia, Nitish Gupta, Shikhar Vashishth, Sriram Ganapathy, Abhishek Bapna, Prateek Jain et al.
  
  
• Large Multi-Modal Models (LMMs) as Universal Foundation Models for AI-Native Wireless Systems,
  by Shengzhe Xu, Christo Kurisummoottil Thomas, Omar Hashash, Nikhil Muralidhar, Walid Saad and Naren Ramakrishnan
  
  
• Arcee's MergeKit: A Toolkit for Merging Large Language Models,
  by Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade and Jacob Solawetz
  
  
• LangBridge: Multilingual Reasoning Without Multilingual Supervision,
  by Dongkeun Yoon, Joel Jang, Sungdong Kim, Seungone Kim, Sheikh Shafayat and Minjoon Seo
  
  
• Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration,
  by Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran and Yulia Tsvetkov
  
  
• AUTOACT: Automatic Agent Learning from Scratch via Self-Planning,
  by Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv and Huajun Chen
  
  
• Corrective Retrieval Augmented Generation,
  by Shi-Qi Yan, Jia-Chen Gu, Yun Zhu and Zhen-Hua Ling
  
  
• ULTRA: Unleash LLMs' Potential for Event Argument Extraction through Hierarchical Modeling and Pair-wise Refinement,
  by Xinliang Frederick Zhang, Carter Wood Blum, Temma Choji, Shalin Shah and Alakananda Vempala
  
  
• ProgGen: Generating Named Entity Recognition Datasets Step-by-step with Self-Reflexive Large Language Models,
  by Yuzhao Heng, Chunyuan Deng, Yitong Li, Yue Yu, Yinghao Li, Rongzhi Zhang and Chao Zhang
  
  
• A Simple but Effective Approach to Improve Structured Language Model Output for Information Extraction,
  by Yinghao Li, Rampi Ramprasad and Chao Zhang
  
  
• UrbanKGent: A Unified Large Language Model Agent Framework for Urban Knowledge Graph Construction,
  by Yansong Ning and Hao Liu
  
  
• Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models,
  by Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui and Furu Wei
  
  
• Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models,
  by Anonymous Submission
  
  
• Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language Models,
  by Sijia Chen, Baochun Li and Di Niu
  
  
• Can Language Models Act as Knowledge Bases at Scale?,
  by Qiyuan He, Yizhong Wang and Wenya Wang
  
  
• Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey,
  by Garima Agrawal, Tharindu Kumarage, Zeyad Alghami and Huan Liu
  
  
• Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-Based Retrofitting,
  by Xinyan Guan, Yanjiang Liu, Hongyu Lin, Yaojie Lu, Ben He, Xianpei Han and Le Sun
  
  
• SoFA: Shielded On-the-fly Alignment via Priority Rule Following,
  by Xinyu Lu, Bowen Yu, Yaojie Lu, Hongyu Lin, Haiyang Yu, Le Sun, Xianpei Han and Yongbin Li
  
  
• Learning or Self-aligning? Rethinking Instruction Fine-tuning,
  by Mengjie Ren, Boxi Cao, Hongyu Lin, Cao Liu, Xianpei Han, Ke Zeng, Guanglu Wan, Xunliang Cai et al.
  
  
• Is a Large Language Model a Good Annotator for Event Extraction?,
  by Ruirui Chen, Chengwei Qin, Weifeng Jiang and Dongkyu Choi
  
  
• Code-Style In-Context Learning for Knowledge-Based Question Answering,
  by Zhijie Nie, Richong Zhang, Zhongyuan Wang and Xudong Liu
  
  
• Two Heads Are Better Than One: Integrating Knowledge from Knowledge Graphs and Large Language Models for Entity Alignment,
  by Linyao Yang, Hongyang Chen, Xiao Wang, Jing Yang, Fei-Yue Wang and Han Liu
  
  
• Is it Possible to Edit Large Language Models Robustly?,
  by Xinbei Ma, Tianjie Ju, Jiyang Qiu, Zhuosheng Zhang, Hai Zhao, Lifeng Liu and Yulong Wang
  
  
• An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaboration,
  by Yihao Li, Ru Zhang, Jianyi Liu and Gongshen Liu
  
  
• LLMs Instruct LLMs: An Extraction and Editing Method,
  by Xin Zhang, Tianjie Ju, Huijia Liang, Ying Fu and Qin Zhang
  
  
• Unleashing the Power of Large Language Models in Zero-shot Relation Extraction via Self-Prompting,
  by Anonymous Submission
  
  
• A Unified Label-Aware Contrastive Learning Framework for Few-Shot Named Entity Recognition,
  by Haojie Zhang and Yimeng Zhuang
  
  
• LLM-DA: Data Augmentation via Large Language Models for Few-Shot Named Entity Recognition,
  by Junjie Ye, Nuo Xu, Yikun Wang, Jie Zhou, Qi Zhang, Tao Gui and Xuanjing Huang
  
  
• On-the-fly Definition Augmentation of LLMs for Biomedical NER,
  by Monica Munnangi, Sergey Feldman, Byron C Wallace, Silvio Amir, Tom Hope and Aakanksha Naik
  
  
• Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Models,
  by Loka Li, Guangyi Chen, Yusheng Su, Zhenhao Chen, Yixuan Zhang, Eric Xing and Kun Zhang
  
  
• Updating Language Models with Unstructured Facts: Towards Practical Knowledge Editing,
  by Xiaobao Wu, Liangming Pan, William Yang Wang and Anh Tuan Luu
  
  
• Event-level Knowledge Editing,
  by Hao Peng, Xiaozhi Wang, Chunyang Li, Kaisheng Zeng, Jiangshan Duo, Yixin Cao, Lei Hou and Juanzi Li
  
  
• Unlocking Instructive In-Context Learning with Tabular Prompting for Relational Triple Extraction,
  by Guozheng Li, Wenjun Ke, Peng Wang, Zijie Xu, Ke Ji, Jiajun Liu, Ziyu Shang and Qiqing Luo
  
  
• Visual Chain-of-Thought Prompting for Knowledge-Based Visual Reasoning,
  by Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Zhiqing Sun, Dan Gutfreund and Chuang Gan
  
  
• Can Large Language Models Understand Real-World Complex Instructions?,
  by Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Jiaqing Liang et al.
  
  
• Beyond Entities: A Large-Scale Multi-Modal Knowledge Graph with Triplet Fact Grounding,
  by Jingping Liu, Mingchuan Zhang, Weichen Li, Chao Wang, Shuang Li, Haiyun Jiang, Sihang Jiang, Yanghua Xiao et al.
  
  
• ConcEPT: Concept-Enhanced Pre-Training for Language Models,
  by Xintao Wang, Zhouhong Gu, Jiaqing Liang, Dakuan Lu, Yanghua Xiao and Wei Wang
  
  
• In-context Learning with Retrieved Demonstrations for Language Models: A Survey,
  by Man Luo, Xin Xu, Yue Liu, Panupong Pasupat and Mehran Kazemi
  
  
• 'One size doesn't fit all': Learning how many Examples to use for In-Context Learning for Improved Text Classification,
  by Manish Chandra, Debasis Ganguly, Yiwen Li and Iadh Ounis
  
  
• "In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval",
  by Andrew Parry, Debasis Ganguly and Manish Chandra
  
  
• Small Language Model Is a Good Guide for Large Language Model in Chinese Entity Relation Extraction,
  by Xuemei Tang, Jun Wang and Qi Su
  
  
• EcomGPT: Instruction-Tuning Large Language Models with Chain-of-Task Tasks for E-commerce,
  by Yangning Li, Shirong Ma, Xiaobin Wang, Shen Huang, Chengyue Jiang, Haitao Zheng, Pengjun Xie, Fei Huang et al.
  
  
• Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization,
  by Boshi Wang, Xiang Yue, Yu Su and Huan Sun
  
  
• Document-Level In-Context Few-Shot Relation Extraction via Pre-Trained Language Models,
  by Yilmazcan Ozyurt, Stefan Feuerriegel and Ce Zhang
  
  

2023

• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity,
  by Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji et al.
  本文提出了一个使用公开数据集定量评估交互式LLM（如ChatGPT）的框架。我们使用涵盖8个不同的常见NLP应用任务的21个数据集对ChatGPT进行了广泛的技术评估。我们基于这些数据集和一个新设计的多模态数据集评估了ChatGPT的多任务、多语言和多模态方面。
  
  
• Crawling the Internal Knowledge-Base of Language Models,
  by Roi Cohen, Mor Geva, Jonathan Berant and Amir Globerson
  本文提出一种从语言模型中提取结构化知识图谱的方法；使用专门设计的提示来控制提取过程中的精度和召回率；在GPT-3上进行了评估，显示了高精确度的结果。
  
  
• Multimodal Chain-of-Thought Reasoning in Language Models,
  by Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis and Alex Smola
  
  
• Rethinking with Retrieval: Faithful Large Language Model Inference,
  by Hangfeng He, Hongming Zhang and Dan Roth
  本文通过用GPT-3在三个复杂的推理任务：常识推理，时间推理和表格推理上进行大量实验来评估RR的有效性。结果表明，RR可以产生更忠实的解释，并提高LLM的性能。
  
  
• On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex,
  by Terry Yue Zhuo, Zhuang Li, Yujin Huang, Yuan-Fang Li, Weiqing Wang, Gholamreza Haffari and Fatemeh Shiri
  
  
• A Survey for In-context Learning,
  by Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu et al.
  This paper surveys and summarizes the progress and challenges of ICL, including ICL's formal definition, correlation to related studies, advanced techniques (training strategies, related analysis) and potential directions.
  
  
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot,
  by Elias Frantar and Dan Alistarh
  
  
• CodeT5Mix: A Pretrained Mixture of Encoder-decoder Transformers for Code Understanding and Generation,
  by Wang, Yue, Le, Hung, Gotmare, Akhilesh Deepak, Li, Junnan and Hoi, Steven
  
  
• CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code,
  by Zhou, Shuyan, Alon, Uri, Agarwal, Sumit and Neubig, Graham
  
  
• Explanation Selection Using Unlabeled Data for In-Context Learning,
  by Xi Ye and Greg Durrett
  
  
• Is ChatGPT a General-Purpose Natural Language Processing Task Solver?,
  by Qin, Chengwei, Zhang, Aston, Zhang, Zhuosheng, Chen, Jiaao, Yasunaga, Michihiro and Yang, Diyi
  
  
• ThoughtSource: A central hub for large language model reasoning data,
  by Simon Ott, Konstantin Hebenstreit, Valentin Li'evin, Christoffer Egeberg Hother, Milad Moradi, Maximilian Mayrhauser, Robert Praas, Ole Winther et al.
  
  
• In-Context Learning with Many Demonstration Examples,
  by Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu and Lingpeng Kong
  This paper proposes a LM named EvaLM to scale up the sequence length (trained with 8k tokens per batch line). Experiments based on EvaLM prove that in-context learning can achieve higher performance with more demonstrations under many-shot instruction tuning (8k) and further extending the length of instructions (16k) can further improve the upper bound of scaling in-context learning.
  
  
• Large Language Models Can Be Easily Distracted by Irrelevant Context,
  by Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Sch"arli and Denny Zhou
  
  
• Toward General Design Principles for Generative AI Applications,
  by Justin D. Weisz, Michael J. Muller, Jessica He and Stephanie Houde
  
  
• Knowledge-enhanced Neural Machine Reasoning: A Review,
  by Tanmoy Chowdhury, Chen Ling, Xuchao Zhang, Xujiang Zhao, Guangji Bai, Jian Pei, Haifeng Chen and Liang Zhao
  
  
• Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning,
  by Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang and Yongbin Li
  
  
• Specializing Smaller Language Models towards Multi-Step Reasoning,
  by Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal and Tushar Khot
  
  
• Scaling Vision Transformers to 22 Billion Parameters,
  by Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron et al.
  
  
• Chat2VIS: Generating Data Visualisations via Natural Language using ChatGPT, Codex and GPT-3 Large Language Models,
  by Paula Maddigan and Teo Susnjak
  
  
• The Capacity for Moral Self-Correction in Large Language Models,
  by Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamile Lukosiute, Anna Chen, Anna Goldie, Azalia Mirhoseini et al.
  
  
• Zero-Shot Information Extraction via Chatting with ChatGPT,
  by Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu et al.
  
  
• A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT,
  by Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji et al.
  
  
• ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots,
  by Reham Omar, Omij Mangukiya, Panos Kalnis and Essam Mansour
  
  
• Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models,
  by Kung, Tiffany H, Cheatham, Morgan, Medenilla, Arielle, Sillos, Czarina, De Leon, Lorie, Elepa~no, Camille, Madriaga, Maria, Aggabao, Rimel et al.
  
  
• ChatGPT passing USMLE shines a spotlight on the flaws of medical education,
  by Mbakwe, Amarachi B, Lourentzou, Ismini, Celi, Leo Anthony, Mechanic, Oren J and Dagan, Alon
  
  
• A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT,
  by Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith et al.
  
  
• Complex QA and language models hybrid architectures, Survey,
  by Xavier Daull, Patrice Bellot, Emmanuel Bruno, Vincent Martin and Elisabeth Murisasco
  
  
• Mathematical Capabilities of ChatGPT,
  by Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier and Julius Berner
  
  
• Active Prompting with Chain-of-Thought for Large Language Models,
  by Shizhe Diao, Pengcheng Wang, Yong Lin and Tong Zhang
  
  
• Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization,
  by Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen and Wei Cheng
  
  
• On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective,
  by Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang et al.
  
  
• ChatAug: Leveraging ChatGPT for Text Data Augmentation,
  by Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Zihao Wu, Lin Zhao, Wei Liu, Ninghao Liu et al.
  
  
• Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data,
  by Kashun Shum, Shizhe Diao and Tong Zhang
  
  
• ChatGPT is not all you need. A State of the Art Review of large Generative AI models,
  by Roberto Gozalo-Brizuela and Eduardo C. Garrido-Merch'an
  
  
• Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT,
  by Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du and Dacheng Tao
  
  
• PaLM-E: An Embodied Multimodal Language Model,
  by Driess, Danny, Xia, Fei, Sajjadi, Mehdi SM, Lynch, Corey, Chowdhery, Aakanksha, Ichter, Brian, Wahid, Ayzaan, Tompson, Jonathan et al.
  
  
• Conversation Regression Testing: A Design Technique for Prototyping Generalizable Prompt Strategies for Pre-trained Language Models,
  by J. D. Zamfirescu-Pereira, Bjoern Hartmann and Qian Yang
  
  
• Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Study,
  by Mingxu Tao, Yansong Feng and Dongyan Zhao
  
  
• In-Context Retrieval-Augmented Language Models,
  by Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown and Yoav Shoham
  
  
• Understanding Finetuning for Factual Knowledge Extraction from Language Models,
  by Mehran Kazemi, Sid Mittal and Deepak Ramachandran
  
  
• Learning Customized Visual Models with Retrieval-Augmented Knowledge,
  by Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee and Chunyuan Li
  
  
• REPLUG: Retrieval-Augmented Black-Box Language Models,
  by Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer and Wen-tau Yih
  
  
• Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning,
  by Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu et al.
  
  
• Robustness of edited neural networks,
  by Davis Brown, Charles Godfrey, Cody Nizinski, Jonathan Tu and Henry Kvinge
  
  
• Transformer-Patcher: One Mistake worth One Neuron,
  by Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong and Zhang Xiong
  
  
• Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models,
  by Wu, Chenfei, Yin, Shengming, Qi, Weizhen, Wang, Xiaodong, Tang, Zecheng and Duan, Nan
  
  
• Large Language Models Are Implicitly Topic Models: Explaining and Finding Good Demonstrations for In-Context Learning,
  by Xinyi Wang, Wanrong Zhu and William Yang Wang
  
  
• MathPrompter: Mathematical Reasoning using Large Language Models,
  by Imani, Shima, Du, Liang and Shrivastava, Harsh
  
  
• GPT-4 Technical Report,
  by OpenAI
  
  
• GPT-4 System Card,
  by OpenAI
  
  
• Augmented Language Models: a Survey,
  by Gr'egoire Mialon, Roberto Dess`\i, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozi`ere, Timo Schick et al.
  
  
• Aligning Text-to-Image Models using Human Feedback,
  by Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh et al.
  
  
• Evaluation of ChatGPT as a Question Answering System for Answering Complex Questions,
  by Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen and Guilin Qi
  
  
• Finding Supporting Examples for In-Context Learning,
  by Xiaonan Li and Xipeng Qiu
  
  
• Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning,
  by Gao, Jiahui, Pi, Renjie, Yong, LIN, Xu, Hang, Ye, Jiacheng, Wu, Zhiyong, ZHANG, WEIZHONG, Liang, Xiaodan et al.
  
  
• Mixture of Soft Prompts for Controllable Data Generation,
  by Derek Chen, Celine Lee, Yunan Lu, Domenic Rosati and Zhou Yu
  
  
• The Learnability of In-Context Learning,
  by Noam Wies, Yoav Levine and Amnon Shashua
  
  
• The Life Cycle of Knowledge in Big Language Models: A Survey,
  by Boxi Cao, Hongyu Lin, Xianpei Han and Le Sun
  
  
• ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models,
  by Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu and Ben He
  
  
• BloombergGPT: A Large Language Model for Finance,
  by Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg et al.
  
  
• Using Language Models For Knowledge Acquisition in Natural Language Reasoning Problems,
  by Fangzhen Lin, Ziyi Shou and Chengcai chen
  
  
• ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design,
  by Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith and Douglas C. Schmidt
  
  
• Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System,
  by Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang and Jiawei Zhang
  
  
• Exploring the Benefits of Training Expert Language Models over Instruction Tuning,
  by Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee and Minjoon Seo
  
  
• Learning to Program with Natural Language,
  by Yiduo Guo, Yaobo Liang, Chenfei Wu, Wenshan Wu, Dongyan Zhao and Nan Duan
  
  
• Tool Learning with Foundation Models,
  by Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang et al.
  
  
• Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models,
  by Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu and Jianfeng Gao
  
  
• Chinese Open Instruction Generalist: A Preliminary Release,
  by Ge Zhang, Yemin Shi, Ruibo Liu, Ruibin Yuan, Yizhi Li, Siwei Dong, Yu Shu, Zhaoqun Li et al.
  
  
• Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPT,
  by Jiawei Zhang
  
  
• How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?,
  by Xin Xu, Yuqi Zhu, Xiaohan Wang and Ningyu Zhang
  
  
• A Survey of Large Language Models,
  by Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang et al.
  
  
• Survey of Hallucination in Natural Language Generation,
  by Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang et al.
  
  
• In-Context Instruction Learning,
  by Seonghyeon Ye, Hyeonbin Hwang, Sohee Yang, Hyeongu Yun, Yireun Kim and Minjoon Seo
  
  
• How Does In-Context Learning Help Prompt Tuning?,
  by Simeng Sun, Yang Liu, Dan Iter, Chenguang Zhu and Mohit Iyyer
  
  
• Safety Assessment of Chinese Large Language Models,
  by Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng and Minlie Huang
  
  
• Aligning Large Language Models with Human: A Survey,
  by Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang et al.
  
  
• Secrets of RLHF in Large Language Models Part I: PPO,
  by Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin et al.
  
  
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback,
  by Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, J'er'emy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak et al.
  
  
• A Survey on Large Language Model based Autonomous Agents,
  by Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang et al.
  
  
• Instruction Tuning with GPT-4,
  by Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley and Jianfeng Gao
  
  
• ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope,
  by Ray, Partha Pratim
  
  
• Is GPT-4 a Good Data Analyst?,
  by Liying Cheng, Xingxuan Li and Lidong Bing
  
  
• Large Language Models for Software Engineering: A Systematic Literature Review,
  by Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo et al.
  
  
• Graph of Thoughts: Solving Elaborate Problems with Large Language Models,
  by Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski et al.
  
  
• Through the Lens of Core Competency: Survey on Evaluation of Large Language Models,
  by Ziyu Zhuang, Qiguang Chen, Longxuan Ma, Mingda Li, Yi Han, Yushan Qian, Haopeng Bai, Zixian Feng et al.
  
  
• Fairness-guided Few-shot Prompting for Large Language Models,
  by Huan Ma, Changqing Zhang, Yatao Bian, Lemao Liu, Zhirui Zhang, Peilin Zhao, Shu Zhang, Huazhu Fu et al.
  
  
• Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance,
  by Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng and Tushar Khot
  
  
• Let's Think Frame by Frame: Evaluating Video Chain of Thought with Video Infilling and Prediction,
  by Vaishnavi Himakunthala, Andy Ouyang, Daniel Rose, Ryan He, Alex Mei, Yujie Lu, Chinmay Sonar, Michael Saxon et al.
  
  
• Large Language Models Struggle to Learn Long-Tail Knowledge,
  by Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace and Colin Raffel
  
  
• Prompting Language Models for Linguistic Structure,
  by Terra Blevins, Hila Gonen and Luke Zettlemoyer
  
  
• Continual Pre-Training of Large Language Models: How to (re)warm your model?,
  by Kshitij Gupta, Benjamin Th'erien, Adam Ibrahim, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish and Timoth'ee Lesort
  
  
• Continual Pre-training of Language Models,
  by Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim and Bing Liu
  
  
• Crawling The Internal Knowledge-Base of Language Models,
  by Roi Cohen, Mor Geva, Jonathan Berant and Amir Globerson
  
  
• Measuring and Modifying Factual Knowledge in Large Language Models,
  by Pouya Pezeshkpour
  
  
• Unifying Large Language Models and Knowledge Graphs: A Roadmap,
  by Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang and Xindong Wu
  
  
• Multimodal Pretraining from Monolingual to Multilingual,
  by Liang Zhang, Ludan Ruan, Anwen Hu and Qin Jin
  
  
• Compositional Prompting Video-language Models to Understand Procedure in Instructional Videos,
  by Guyue Hu, Bin He and Hanwang Zhang
  
  
• EVA2.0: Investigating Open-domain Chinese Dialogue Systems with Large-scale Pre-training,
  by Yuxian Gu, Jiaxin Wen, Hao Sun, Yi Song, Pei Ke, Chujie Zheng, Zheng Zhang, Jianzhu Yao et al.
  
  
• The Web Can Be Your Oyster for Improving Language Models,
  by Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jingyuan Wang, Jian-Yun Nie and Ji-Rong Wen
  
  
• Small Pre-trained Language Models Can be Fine-tuned as Large Models via Over-Parameterization,
  by Ze-Feng Gao, Kun Zhou, Peiyu Liu, Wayne Xin Zhao and Ji-Rong Wen
  
  
• Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach,
  by Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin and Ji-Rong Wen
  
  
• Large Language Models for Information Retrieval: A Survey,
  by Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou and Ji-Rong Wen
  
  
• Dr.ICL: Demonstration-Retrieved In-context Learning,
  by Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Seyed Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite and Vincent Y. Zhao
  
  
• Unified Demonstration Retriever for In-Context Learning,
  by Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang et al.
  
  
• ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling,
  by Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding and Xindong Wu
  
  
• LLMs4OL: Large Language Models for Ontology Learning,
  by Hamed Babaei Giglou, Jennifer D'Souza and S"oren Auer
  
  
• Tab2KG: Semantic Table Interpretation with Lightweight Semantic Profiles,
  by Simon Gottschalk and Elena Demidova
  
  
• Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models,
  by Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee and Ee-Peng Lim
  
  
• Large Language Model Guided Tree-of-Thought,
  by Jieyi Long
  
  
• Causality-aware Concept Extraction based on Knowledge-guided Prompting,
  by Siyu Yuan, Deqing Yang, Jinxi Liu, Shuyu Tian, Jiaqing Liang, Yanghua Xiao and Rui Xie
  
  
• Revisiting Relation Extraction in the era of Large Language Models,
  by Somin Wadhwa, Silvio Amir and Byron C. Wallace
  
  
• Is Information Extraction Solved by ChatGPT? An Analysis of Performance, Evaluation Criteria, Robustness and Errors,
  by Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang, Lu Liu and Xiang Wan
  
  
• RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback,
  by Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune and Abhinav Rastogi
  
  
• Speciality vs Generality: An Empirical Study on Catastrophic Forgetting in Fine-tuning Foundation Models,
  by Lin, Yong, Tan, Lu, Lin, Hangyu, Zheng, Zeming, Pi, Renjie, Zhang, Jipeng, Diao, Shizhe, Wang, Haoxiang et al.
  
  
• Learning In-context Learning for Named Entity Recognition,
  by Jiawei Chen, Yaojie Lu, Hongyu Lin, Jie Lou, Wei Jia, Dai Dai, Hua Wu, Boxi Cao et al.
  
  
• Knowledge Solver: Teaching LLMs to Search for Domain Knowledge from Knowledge Graphs,
  by Chao Feng, Xinyu Zhang and Zichu Fei
  
  
• Exploring In-Context Learning Capabilities of Foundation Models for Generating Knowledge Graphs from Text,
  by Hanieh Khorashadizadeh, Nandana Mihindukulasooriya, Sanju Tiwari, Jinghua Groppe and Sven Groppe
  
  
• WebIE: Faithful and Robust Information Extraction on the Web,
  by Chenxi Whitehouse, Clara Vania, Alham Fikri Aji, Christos Christodoulopoulos and Andrea Pierleoni
  
  
• A Survey on Evaluation of Large Language Models,
  by Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi et al.
  
  
• Exploring the In-context Learning Ability of Large Language Model for Biomedical Concept Linking,
  by Qinyong Wang, Zhenxiang Gao and Rong Xu
  
  
• Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs,
  by Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin et al.
  
  
• Natural Language is All a Graph Needs,
  by Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu and Yongfeng Zhang
  
  
• Large Graph Models: A Perspective,
  by Ziwei Zhang, Haoyang Li, Zeyang Zhang, Yijian Qin, Xin Wang and Wenwu Zhu
  
  
• Unleashing the Power of Graph Learning through LLM-based Autonomous Agents,
  by Lanning Wei, Zhiqiang He, Huan Zhao and Quanming Yao
  
  
• Editing Language Model-based Knowledge Graph Embeddings,
  by Siyuan Cheng, Ningyu Zhang, Bozhong Tian, Zelin Dai, Feiyu Xiong, Wei Guo and Huajun Chen
  
  
• Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons,
  by Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu and Jun Zhao
  
  
• PMET: Precise Model Editing in a Transformer,
  by Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma and Jie Yu
  
  
• Can Neural Network Memorization Be Localized?,
  by Pratyush Maini, Michael Curtis Mozer, Hanie Sedghi, Zachary Chase Lipton, J. Zico Kolter and Chiyuan Zhang
  
  
• MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions,
  by Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts and Danqi Chen
  
  
• AIGC for Various Data Modalities: A Survey,
  by Lin Geng Foo, Hossein Rahmani and Jun Liu
  
  
• Domain specialization as the key to make large language models disruptive: A comprehensive survey,
  by Ling, Chen, Zhao, Xujiang, Lu, Jiaying, Deng, Chengyuan, Zheng, Can, Wang, Junxiang, Chowdhury, Tanmoy, Li, Yun et al.
  
  
• Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark,
  by Jason Hoelscher-Obermaier, Julia Persson, Esben Kran, Ioannis Konstas and Fazl Barez
  
  
• Methods for Measuring, Updating, and Visualizing Factual Beliefs in Language Models,
  by Peter Hase, Mona T. Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal and Srinivasan Iyer
  
  
• Language Model Analysis for Ontology Subsumption Inference,
  by Yuan He, Jiaoyan Chen, Ernesto Jim'enez-Ruiz, Hang Dong and Ian Horrocks
  
  
• BertNet: Harvesting Knowledge Graphs with Arbitrary Relations from Pretrained Language Models,
  by Shibo Hao, Bowen Tan, Kaiwen Tang, Bin Ni, Xiyan Shao, Hengzhe Zhang, Eric P. Xing and Zhiting Hu
  
  
• Text Augmented Open Knowledge Graph Completion via Pre-Trained Language Models,
  by Pengcheng Jiang, Shivam Agarwal, Bowen Jin, Xuan Wang, Jimeng Sun and Jiawei Han
  
  
• Chain-of-Verification Reduces Hallucination in Large Language Models,
  by Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz and Jason Weston
  
  
• Language models are multilingual chain-of-thought reasoners,
  by Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay et al.
  
  
• Recommender Systems in the Era of Large Language Models (LLMs),
  by Wenqi Fan, Zihuai Zhao, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Jiliang Tang and Qing Li
  
  
• A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT,
  by Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip S. Yu and Lichao Sun
  
  
• MedEdit: Model Editing for Medical Question Answering with External Knowledge Bases,
  by Yucheng Shi, Shaochen Xu, Zhengliang Liu, Tianming Liu, Xiang Li and Ninghao Liu
  
  
• Beyond Chain-of-Thought, Effective Graph-of-Thought Reasoning in Large Language Models,
  by Yao Yao, Zuchao Li and Hai Zhao
  
  
• Thinking Like an Expert: Multimodal Hypergraph-of-Thought (HoT) Reasoning to boost Foundation Modals,
  by Fanglong Yao, Changyuan Tian, Jintao Liu, Zequn Zhang, Qing Liu, Li Jin, Shuchao Li, Xiaoyu Li et al.
  
  
• Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models,
  by Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji and Ajay Divakaran
  
  
• Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thought,
  by Bin Lei, Pei-Hung Lin, Chunhua Liao and Caiwen Ding
  
  
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models,
  by Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao and Karthik Narasimhan
  
  
• Tree-of-Mixed-Thought: Combining Fast and Slow Thinking for Multi-hop Visual Reasoning,
  by Pengbo Hu, Ji Qi, Xingyu Li, Hong Li, Xinqi Wang, Bing Quan, Ruiyu Wang and Yi Zhou
  
  
• Instruction Tuning for Large Language Models: A Survey,
  by Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu et al.
  
  
• In-Context Demonstration Selection with Cross Entropy Difference,
  by Dan Iter, Reid Pryzant, Ruochen Xu, Shuohang Wang, Yang Liu, Yichong Xu and Chenguang Zhu
  
  
• <img src=https://img.shields.io/badge/the_61st_Annual_Meeting_of_the_Association_for_Computational Linguistics_(Volume_2:Short_Papers),{ACL}_2023,_Toronto,_Canada, July_9--14,_2023-2023-blue alt="img" style="zoom:100%; vertical-align: middle" /> MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models,
  by Masoud Monajatipoor, Liunian Harold Li, Mozhdeh Rouhsedaghat, Lin Yang and Kai-Wei Chang
  
  
• SINC: Self-Supervised In-Context Learning for Vision-Language Tasks,
  by Yi-Syuan Chen, Yun-Zhu Song, Cheng Yu Yeo, Bei Liu, Jianlong Fu and Hong-Han Shuai
  
  
• LawBench: Benchmarking Legal Knowledge of Large Language Models,
  by Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen et al.
  
  
• LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models,
  by Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher R'e, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters et al.
  
  
• Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity,
  by Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Jiayang Cheng, Yunzhi Yao, Wenyang Gao et al.
  
  
• Rethinking Language Models as Symbolic Knowledge Graphs,
  by Vishwas Mruthyunjaya, Pouya Pezeshkpour, Estevam Hruschka and Nikita Bhutani
  
  
• Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors,
  by Kai Zhang, Bernal Jimenez Gutierrez and Yu Su
  
  
• How Many Demonstrations Do You Need for In-context Learning?,
  by Jiuhai Chen, Lichang Chen, Chen Zhu and Tianyi Zhou
  
  
• Explaining Emergent In-Context Learning as Kernel Regression,
  by Chi Han, Ziqi Wang, Han Zhao and Heng Ji
  
  
• Dataless Knowledge Fusion by Merging Weights of Language Models,
  by Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro and Pengxiang Cheng
  
  
• Deep Model Fusion: A Survey,
  by Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu and Li Shen
  
  
• AdaMerging: Adaptive Model Merging for Multi-Task Learning,
  by Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang and Dacheng Tao
  
  
• Resolving Interference When Merging Models,
  by Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel and Mohit Bansal
  
  
• Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy,
  by Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal and Tianlong Chen
  
  
• Model Merging by Uncertainty-Based Gradient Matching,
  by Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti, Iryna Gurevych and Mohammad Emtiyaz Khan
  
  
• Making Large Language Models Better Reasoners with Alignment,
  by Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu and Zhifang Sui
  
  
• Making Large Language Models Better Reasoners with Step-Aware Verifier,
  by Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou and Weizhu Chen
  
  
• Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes,
  by Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee et al.
  
  
• Fusing Models with Complementary Expertise,
  by Hongyi Wang, Felipe Maia Polo, Yuekai Sun, Souvik Kundu, Eric P. Xing and Mikhail Yurochkin
  
  
• Complexity-Based Prompting for Multi-step Reasoning,
  by Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark and Tushar Khot
  
  
• CITING: Large Language Models Create Curriculum for Instruction Tuning,
  by Tao Feng, Zifeng Wang and Jimeng Sun
  
  
• A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future,
  by Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu et al.
  
  
• Penguins Don't Fly: Reasoning about Generics through Instantiations and Exceptions,
  by Emily Allaway, Jena D. Hwang, Chandra Bhagavatula, Kathleen R. McKeown, Doug Downey and Yejin Choi
  
  
• A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets,
  by Md. Tahmid Rahman Laskar, M. Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty and Jimmy X. Huang
  
  
• GPTEval: A Survey on Assessments of ChatGPT and GPT-4,
  by Rui Mao, Guanyi Chen, Xulang Zhang, Frank Guerin and Erik Cambria
  
  
• Revisiting Large Language Models as Zero-shot Relation Extractors,
  by Guozheng Li, Peng Wang and Wenjun Ke
  
  
• Meta-CoT: Generalizable Chain-of-Thought Prompting in Mixed-task Scenarios with Large Language Models,
  by Anni Zou, Zhuosheng Zhang, Hai Zhao and Xiangru Tang
  
  
• NeuroComparatives: Neuro-Symbolic Distillation of Comparative Knowledge,
  by Phillip Howard, Junlin Wang, Vasudev Lal, Gadi Singer, Yejin Choi and Swabha Swayamdipta
  
  
• PIVOINE: Instruction Tuning for Open-world Information Extraction,
  by Keming Lu, Xiaoman Pan, Kaiqiang Song, Hongming Zhang, Dong Yu and Jianshu Chen
  
  
• Product Attribute Value Extraction using Large Language Models,
  by Alexander Brinkmann, Roee Shraga and Christian Bizer
  
  
• GPT-RE: In-context Learning for Relation Extraction using Large Language Models,
  by Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li and Sadao Kurohashi
  
  
• Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge,
  by Jiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng, Lei Li and Yanghua Xiao
  
  
• Chain of Thought Prompting Elicits Knowledge Augmentation,
  by Dingjun Wu, Jing Zhang and Xinmei Huang
  
  
• Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency,
  by Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu and Zhaoran Wang
  
  
• Joint Prompt Optimization of Stacked LLMs using Variational Inference,
  by Alessandro Sordoni, Xingdi Yuan, Marc-Alexandre Cote, Matheus Pereira, Adam Trischler, Ziang Xiao, Arian Hosseini, Friederike Niedtner et al.
  
  
• All in One: Multi-Task Prompting for Graph Neural Networks,
  by Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu and Jihong Guan
  
  
• Graph Neural Prompting with Large Language Models,
  by Yijun Tian, Huan Song, Zichen Wang, Haozhu Wang, Ziqing Hu, Fang Wang, Nitesh V. Chawla and Panpan Xu
  
  
• Graph Prompt Learning: A Comprehensive Survey and Beyond,
  by Xiangguo Sun, Jiawen Zhang, Xixi Wu, Hong Cheng, Yun Xiong and Jia Li
  
  
• Large Language Models on Graphs: A Comprehensive Survey,
  by Bowen Jin, Gang Liu, Chi Han, Meng Jiang, Heng Ji and Jiawei Han
  
  
• Bring Your Own KG: Self-Supervised Program Synthesis for Zero-Shot KGQA,
  by Dhruv Agarwal, Rajarshi Das, Sopan Khosla and Rashmi Gangadharaiah
  
  
• Explainability for Large Language Models: A Survey,
  by Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin et al.
  
  
• Large Language Models for Generative Information Extraction: A Survey,
  by Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng et al.
  
  
• Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text,
  by Nandana Mihindukulasooriya, Sanju Tiwari, Carlos F. Enguix and Kusum Lata
  
  
• Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation,
  by Jinglong Gao, Xiao Ding, Bing Qin and Ting Liu
  
  
• A Survey of Graph Meets Large Language Model: Progress and Future Directions,
  by Yuhan Li, Zhixun Li, Peisong Wang, Jia Li, Xiangguo Sun, Hong Cheng and Jeffrey Xu Yu
  
  
• <img src=https://img.shields.io/badge/Findings_of_the_Association_for_Computational_Linguistics:_{EMNLP} 2023,_Singapore,_December_6--10,_2023-2023-blue alt="img" style="zoom:100%; vertical-align: middle" /> PIVOINE: Instruction Tuning for Open-world Entity Profiling,
  by Keming Lu, Xiaoman Pan, Kaiqiang Song, Hongming Zhang, Dong Yu and Jianshu Chen
  
  
• Zero- and Few-Shots Knowledge Graph Triplet Extraction with Large Language Models,
  by Andrea Papaluca, Daniel Krefl, Sergio Mendez Rodriguez, Artem Lensky and Hanna Suominen
  
  
• <img src=https://img.shields.io/badge/{IEEE/CVF}_Conference_on_Computer_Vision_and_Pattern_Recognition, {CVPR}_2023,_Vancouver,_BC,_Canada,_June_17--24,_2023-2023-blue alt="img" style="zoom:100%; vertical-align: middle" /> Visual Atoms: Pre-Training Vision Transformers with Sinusoidal Waves,
  by Sora Takashima, Ryo Hayamizu, Nakamasa Inoue, Hirokatsu Kataoka and Rio Yokota
  
  
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models,
  by Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li and Mohamed Elhoseiny
  
  
• VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks,
  by Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu et al.
  
  
• Graph Meets LLM: A Novel Approach to Collaborative Filtering for Robust Conversational Understanding,
  by Zheng Chen, Ziyan Jiang, Fan Yang, Eunah Cho, Xing Fan, Xiaojiang Huang, Yanbin Lu and Aram Galstyan
  
  
• Prophet: Prompting Large Language Models with Complementary Answer Heuristics for Knowledge-based Visual Question Answering,
  by Zhou Yu, Xuecheng Ouyang, Zhenwei Shao, Meng Wang and Jun Yu
  
  
• keqing: knowledge-based question answering is a nature chain-of-thought mentor of LLM,
  by Chaojie Wang, Yishi Xu, Zhong Peng, Chenxi Zhang, Bo Chen, Xinrun Wang, Lei Feng and Bo An
  
  
• Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles,
  by Ye, Shuquan, Xie, Yujia, Chen, Dongdong, Xu, Yichong, Yuan, Lu, Zhu, Chenguang and Liao, Jing
  
  
• Fusing Knowledge Graphs and Large Language Models,
  by Rudy Agovic
  
  
• RAG with a Neo4j Knowledge Graph: How it Works and How to Set It Up,
  by Neo4j
  
  
• Making Large Language Models Perform Better in Knowledge Graph Completion,
  by Yichi Zhang, Zhuo Chen, Wen Zhang and Huajun Chen
  
  
• Visual Instruction Tuning,
  by Haotian Liu, Chunyuan Li, Qingyang Wu and Yong Jae Lee
  
  
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning,
  by Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung et al.
  
  
• Instruct and Extract: Instruction Tuning for On-Demand Information Extraction,
  by Yizhu Jiao, Ming Zhong, Sha Li, Ruining Zhao, Siru Ouyang, Heng Ji and Jiawei Han
  
  
• GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction,
  by Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li and Ying Shan
  
  
• LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark,
  by Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang et al.
  
  
• Zero- and Few-Shots Knowledge Graph Triplet Extraction with Large Language Models,
  by Andrea Papaluca, Daniel Krefl, Sergio Mendez Rodriguez, Artem Lenskiy and Hanna Suominen
  
  
• Empirical Study of Zero-Shot NER with ChatGPT,
  by Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang, Zuozhu Liu and Hongwei Wang
  
  
• Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond,
  by Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu and Erik Cambria
  
  
• Towards Foundation Models for Knowledge Graph Reasoning,
  by Mikhail Galkin, Xinyu Yuan, Hesham Mostafa, Jian Tang and Zhaocheng Zhu
  
  
• Large Language Models and Knowledge Graphs: Opportunities and Challenges,
  by Jeff Z. Pan, Simon Razniewski, Jan-Christoph Kalo, Sneha Singhania, Jiaoyan Chen, Stefan Dietze, Hajira Jabeen, Janna Omeliyanenko et al.
  
  
• Extracting Multi-valued Relations from Language Models,
  by Sneha Singhania, Simon Razniewski and Gerhard Weikum
  
  
• Evaluating the Knowledge Base Completion Potential of GPT,
  by Blerta Veseli, Simon Razniewski, Jan-Christoph Kalo and Gerhard Weikum
  
  
• Make a Choice! Knowledge Base Question Answering with In-Context Learning,
  by Chuanyuan Tan, Yuehe Chen, Wenbiao Shao and Wenliang Chen
  
  
• Few-shot In-context Learning for Knowledge Base Question Answering,
  by Tianle Li, Xueguang Ma, Alex Zhuang, Yu Gu, Yu Su and Wenhu Chen
  
  
• In-Context Learning for Knowledge Base Question Answering for Unmanned Systems based on Large Language Models,
  by Yunlong Chen, Yaming Zhang, Jianfei Yu, Li Yang and Rui Xia
  
  
• Learning To Teach Large Language Models Logical Reasoning,
  by Meiqi Chen, Yubo Ma, Kaitao Song, Yixin Cao, Yan Zhang and Dongsheng Li
  
  
• Robot Learning in the Era of Foundation Models: A Survey,
  by Xuan Xiao, Jiahang Liu, Zhipeng Wang, Yanmin Zhou, Yong Qi, Qian Cheng, Bin He and Shuo Jiang
  
  
• Zero-shot Triplet Extraction by Template Infilling,
  by Bosung Kim, Hayate Iso, Nikita Bhutani, Estevam Hruschka, Ndapa Nakashole and Tom M. Mitchell
  
  
• LLM Instruction-Example Adaptive Prompting (LEAP) Framework for Clinical Relation Extraction,
  by Anonymous Submission
  
  
• Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!,
  by Yubo Ma, Yixin Cao, Yong Hong and Aixin Sun
  
  
• Chain of Thought with Explicit Evidence Reasoning for Few-shot Relation Extraction,
  by Xilai Ma, Jing Li and Min Zhang
  
  
• Self-Verification Improves Few-Shot Clinical Information Extraction,
  by Zelalem Gero, Chandan Singh, Hao Cheng, Tristan Naumann, Michel Galley, Jianfeng Gao and Hoifung Poon
  
  
• Guideline Learning for In-Context Information Extraction,
  by Chaoxu Pang, Yixuan Cao, Qiang Ding and Ping Luo
  
  
• Meta-in-context learning in large language models,
  by Julian Coda-Forno, Marcel Binz, Zeynep Akata, Matt M. Botvinick, Jane X. Wang and Eric Schulz
  
  
• DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models,
  by Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou and Sibei Yang
  
  
• Chain of Thought Prompt Tuning in Vision Language Models,
  by Jiaxin Ge, Hongyin Luo, Siyuan Qian, Yulu Gan, Jie Fu and Shanghang Zhang
  
  
• Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models,
  by Paul Youssef, Osman Alperen Koras, Meijie Li, J"org Schl"otterer and Christin Seifert
  
  
• Dissecting Chain-of-Thought: Compositionality through In-Context Filtering and Learning,
  by Yingcong Li, Kartik Sreenivasan, Angeliki Giannou, Dimitris Papailiopoulos and Samet Oymak
  
  
• Schema-learning and rebinding as mechanisms of in-context learning and emergence,
  by Sivaramakrishnan Swaminathan, Antoine Dedieu, Rajkumar Vasudeva Raju, Murray Shanahan, Miguel L'azaro-Gredilla and Dileep George
  
  

2022

• Meta-learning via Language Model In-context Tuning,
  by Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis and He He
  This paper proposes in-context tuning, which recasts task adaptation and prediction as a simple sequence prediction problem: to form the input sequence, concatenate the task instruction, labeled in-context examples, and the target input to predict; to meta train the model to learn from in-context examples, finetune a PLM to predict the target label given the input sequence on a collection of tasks (very similar to MetaICL). On LAMA and BinaryClfs, the proposed method outperforms MAML.
  
  
• Code4Struct: Code Generation for Few-Shot Structured Prediction from Natural Language,
  by Xingyao Wang, Sha Li and Heng Ji
  
  
• Learning to Prompt for Continual Learning,
  by Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot et al.
  
  
• <img src=https://img.shields.io/badge/Findings_of_the_Association_for_Computational_Linguistics:_{EMNLP} 2022,_Abu_Dhabi,_United_Arab_Emirates,_December_7--11,_2022-2022-blue alt="img" style="zoom:100%; vertical-align: middle" /> In-Context Learning for Few-Shot Dialogue State Tracking,
  by Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith and Mari Ostendorf
  
  
• <img src=https://img.shields.io/badge/the_2022_Conference_on_Empirical_Methods_in_Natural Language_Processing,_{EMNLP}_2022,_Abu_Dhabi,_United_Arab_Emirates, December_7--11,_2022-2022-blue alt="img" style="zoom:100%; vertical-align: middle" /> InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning,
  by Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, Maxine Esk'enazi and Jeffrey P. Bigham
  
  
• Language Models of Code are Few-Shot Commonsense Learners,
  by Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang and Graham Neubig
  
  
• Do Prompt-Based Models Really Understand the Meaning of Their Prompts?,
  by Albert Webson and Ellie Pavlick
  
  
• MetaICL: Learning to Learn In Context,
  by Sewon Min, Mike Lewis, Luke Zettlemoyer and Hannaneh Hajishirzi
  MetaICL proposes a supervised meta-training framework to enable LMs to more effectively learn a new task in context. In MetaICL, each meta-training example includes several training examples from one task that will be presented together as a single sequence to the LM, and the prediction of the final example is used to calculate the loss.
  
  
• News Summarization and Evaluation in the Era of GPT-3,
  by Tanya Goyal, Junyi Jessy Li and Greg Durrett
  
  
• Deep Bidirectional Language-Knowledge Graph Pretraining,
  by Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D. Manning, Percy Liang and Jure Leskovec
  
  
• Selective Annotation Makes Language Models Better Few-Shot Learners,
  by Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf et al.
  This paper proposes a graph-based selective annotation method named vote-k to
(1) select a pool of examples to annotate from unlabeled data,
(2) retrieve prompts (contexts) from the annotated data pool for in-context learning.
Specifically, the selection method first selects a small set of unlabeled examples iteratively and then labels them to serve as contexts for LLMs to predict the labels of the rest unlabeled data. The method selects the predictions with highest confidence (log probability of generation output) to fill up the selective annotation pool.
  
  
• Selective Data Acquisition in the Wild for Model Charging,
  by Chengliang Chai, Jiabin Liu, Nan Tang, Guoliang Li and Yuyu Luo
  
  
• When Neural Model Meets NL2Code: A Survey,
  by Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang and Jian-Guang Lou
  
  
• A Survey on Knowledge-Enhanced Pre-trained Language Models,
  by Chaoqi Zhen, Yanlei Shang, Xiangyu Liu, Yifei Li, Yong Chen and Dell Zhang
  
  
• Review of Knowledge-Enhanced Pre-trained Language Models,
  by Yi, HAN, Linbo, QIAO, Dongsheng, LI and Xiangke, LIAO
  
  
• Holistic Evaluation of Language Models,
  by Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan et al.
  
  
• Improving In-Context Few-Shot Learning via Self-Supervised Training,
  by Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov and Zornitsa Kozareva
  This paper proposes to use self-supervision (MLM, NSP, CL, etc.) between pre-training and downstream usage to teach the LM to perform in-context learning. Analysis reveals that:
(1) benefits of self-supervised depends on the amount of training data,
(2) semantic similarity between training and evaluation tasks matters,
(3) adding training objectives without diversity does not help,
(4) model performance improves when choosing similar templates for both self-supervised and downstream tasks,
(5) self-supervised tasks and human-annotated datasets are complementary,
(6) self-supervised-trained models are better at following task instructions.
  
  
• Finetuned Language Models are Zero-Shot Learners,
  by Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai et al.
  
  
• LaMDA: Language Models for Dialog Applications,
  by Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos et al.
  
  
• Scaling Instruction-Finetuned Language Models,
  by Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang et al.
  
  
• Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks,
  by Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran et al.
  
  
• Instruction Induction: From Few Examples to Natural Language Task Descriptions,
  by Or Honovich, Uri Shaham, Samuel R. Bowman and Omer Levy
  (1) 探索了利用LLM在几个样本的情况下归纳出任务指令的能力；
(2) 测量两个指标：1. 模型归纳指令与人类归纳的指令对比，2. 利用模型归纳的指令作为prompt进行预测的执行准确率；
(3) 相比于GPT-3，InstructGPT效果更好，理所当然。
  
  
• Large Language Models Are Human-Level Prompt Engineers,
  by Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan and Jimmy Ba
  
  
• Self-Instruct: Aligning Language Model with Self Generated Instructions,
  by Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi and Hannaneh Hajishirzi
  
  
• Iteratively Prompt Pre-trained Language Models for Chain of Thought,
  by Boshi Wang, Xiang Deng and Huan Sun
  (1) 提出了一种迭代式的prompt-tuning方法，他们认为soft prompt应该带有语境，即在自回归解码时不同时刻应该有不同的prompt向量；
(2) 利用BERT为encoder-decoder架构的PLM生成prompt，在每个解码时刻BERT都会根据先前时刻的上下文生成一组新的prompt向量，提供给PLM生成新的上下文，迭代往复。
  
  
• Complexity-Based Prompting for Multi-Step Reasoning,
  by Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark and Tushar Khot
  
  
• Measuring and Narrowing the Compositionality Gap in Language Models,
  by Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith and Mike Lewis
  
  
• Automatic Chain of Thought Prompting in Large Language Models,
  by Zhuosheng Zhang, Aston Zhang, Mu Li and Alex Smola
  
  
• Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity,
  by Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel and Pontus Stenetorp
  (1) This work demonstrates that few-shot prompts suffer from order sensitivity, in that for the same prompt the order in which samples are provided can make a difference to model performance.
(2) This work introduces a probing method which constructs an artificial development set by language models themselves to alleviate the order sensitivity problem.
  
  
• Learning To Retrieve Prompts for In-Context Learning,
  by Ohad Rubin, Jonathan Herzig and Jonathan Berant
  This paper proposes a method to retrieve good contexts for in-context learning. Specifically, the method
(1) uses an unsupervised retriever (BM25/SBERT) to obtain a set of context candidates,
(2) passes the candidates to a scoring model (GPT-Neo/GPT-J/GPT-3/Codex) and select the top/bottom k as positive/negative examples,
(3) uses the examples to train a dense retriever (BERT-based).
  
  
• Active Example Selection for In-Context Learning,
  by Yiming Zhang, Shi Feng and Chenhao Tan
  (1) This paper revisits the effect of example selection (re-ordering & calibration) for ICL, observing that a large variance across set of demonstration examples still exists.
(2) This paper applies reinforcement learning (Q-Learning) to optimize example selection by formulating this task as sequential decision-making problem, which is appropriate for example selection from unlabeled datasets.
  
  
• An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels,
  by Taylor Sorensen, Joshua Robinson, Christopher Michael Rytting, Alexander Glenn Shaw, Kyle Jeffrey Rogers, Alexia Pauline Delorey, Mahmoud Khalil, Nancy Fulda et al.
  
  
• Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator,
  by Hyuhng Joon Kim, Hyunsoo Cho, Junyeob Kim, Taeuk Kim, Kang Min Yoo and Sang-goo Lee
  
  
• Demystifying Prompts in Language Models via Perplexity Estimation,
  by Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith and Luke Zettlemoyer
  
  
• Measuring Convergence Inertia: Online Learning in Self-adaptive Systems with Context Shifts,
  by Elvin Alberts and Ilias Gerostathopoulos
  
  
• Chain of Thought Prompting Elicits Reasoning in Large Language Models,
  by Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le and Denny Zhou
  
  
• Self-Consistency Improves Chain of Thought Reasoning in Language Models,
  by Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi and Denny Zhou
  
  
• Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models,
  by Robert L. Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh and Sebastian Riedel
  
  
• Adversarial Soft Prompt Tuning for Cross-Domain Sentiment Analysis,
  by Hui Wu and Xiaodong Shi
  
  
• Evaluating the Text-to-SQL Capabilities of Large Language Models,
  by Nitarshan Rajkumar, Raymond Li and Dzmitry Bahdanau
  
  
• Fine-Grained Controllable Text Generation Using Non-Residual Prompting,
  by Fredrik Carlsson, Joey "Ohman, Fangyu Liu, Severine Verlinden, Joakim Nivre and Magnus Sahlgren
  
  
• MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators,
  by Zhixing Tan, Xiangwen Zhang, Shuo Wang and Yang Liu
  
  
• Noisy Channel Language Model Prompting for Few-Shot Text Classification,
  by Sewon Min, Mike Lewis, Hannaneh Hajishirzi and Luke Zettlemoyer
  
  
• SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer,
  by Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou' and Daniel Cer
  
  
• A Continual Learning Survey: Defying Forgetting in Classification Tasks,
  by Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory G. Slabaugh and Tinne Tuytelaars
  
  
• ELLE: Efficient Lifelong Pre-training for Emerging Data,
  by Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun and Jie Zhou
  
  
• An Explanation of In-context Learning as Implicit Bayesian Inference,
  by Sang Michael Xie, Aditi Raghunathan, Percy Liang and Tengyu Ma
  
  
• Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models,
  by Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen et al.
  
  
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?,
  by Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi and Luke Zettlemoyer
  
  
• Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora,
  by Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew O. Arnold and Xiang Ren
  
  
• Meta-Adapters: Parameter Efficient Few-shot Fine-tuning through Meta-Learning,
  by Trapit Bansal, Salaheddin Alzubi, Tong Wang, Jay-Yoon Lee and Andrew McCallum
  
  
• Sparse Structure Search for Delta Tuning,
  by Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang, Yasheng Wang, Zhiyuan Liu and Maosong Sun
  
  
• Ontology-enhanced Prompt-tuning for Few-shot Learning,
  by Hongbin Ye, Ningyu Zhang, Shumin Deng, Xiang Chen, Hui Chen, Feiyu Xiong, Xi Chen and Huajun Chen
  
  
• Pre-trained Language Models can be Fully Zero-Shot Learners,
  by Xuandong Zhao, Siqi Ouyang, Zhiguo Yu, Ming Wu and Lei Li
  
  
• Towards Continual Reinforcement Learning: A Review and Perspectives,
  by Khimya Khetarpal, Matthew Riemer, Irina Rish and Doina Precup
  
  
• Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Models,
  by Zijian Zhang, Zhou Zhao and Zhijie Lin
  
  
• The survey: Text generation models in deep learning,
  by Touseef Iqbal and Shaima Qureshi
  
  
• Fairness and accuracy in horizontal federated learning,
  by Wei Huang, Tianrui Li, Dexian Wang, Shengdong Du, Junbo Zhang and Tianqiang Huang
  
  
• Federated Learning Meets Multi-Objective Optimization,
  by Zeou Hu, Kiarash Shaloudegi, Guojun Zhang and Yaoliang Yu
  
  
• From distributed machine learning to federated learning: a survey,
  by Ji Liu, Jizhou Huang, Yang Zhou, Xuhong Li, Shilei Ji, Haoyi Xiong and Dejing Dou
  
  
• Meta-Learning Based Knowledge Extrapolation for Knowledge Graphs in the Federated Setting,
  by Mingyang Chen, Wen Zhang, Zhen Yao, Xiangnan Chen, Mengxiao Ding, Fei Huang and Huajun Chen
  
  
• Mitigating Biases in Student Performance Prediction via Attention-Based Personalized Federated Learning,
  by Yun-Wei Chu, Seyyedali Hosseinalipour, Elizabeth Tenorio, Laura M. Cruz Castro, Kerrie A. Douglas, Andrew Lan and Christopher G. Brinton
  
  
• Pretrained Models for Multilingual Federated Learning,
  by Orion Weller, Marc Marone, Vladimir Braverman, Dawn J. Lawrie and Benjamin Van Durme
  
  
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning,
  by Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, Ehsan Adeli, Li Fei-Fei and Daniel L. Rubin
  
  
• FedBERT: When Federated Learning Meets Pre-training,
  by Yuanyishu Tian, Yao Wan, Lingjuan Lyu, Dezhong Yao, Hai Jin and Lichao Sun
  
  
• Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning,
  by John Nguyen, Jianyu Wang, Kshitiz Malik, Maziar Sanjabi and Michael Rabbat
  
  
• Distributed Training of Knowledge Graph Embedding Models using Ray,
  by Nasrullah Sheikh, Xiao Qin, Yaniv Gur and Berthold Reinwald
  
  
• Distributed Learning With Sparsified Gradient Differences,
  by Yicheng Chen, Rick S. Blum, Martin Tak'ac and Brian M. Sadler
  
  
• Graph Attention Neural Network Distributed Model Training,
  by Esmaeilzadeh, Armin, Zadeh Nojoo Kambar, Mina Esmail and Heidari, Maryam
  
  
• An extensive study on pre-trained models for program understanding and generation,
  by Zhengran Zeng, Hanzhuo Tan, Haotian Zhang, Jing Li, Yuqun Zhang and Lingming Zhang
  
  
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning,
  by Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese and Steven C. H. Hoi
  
  
• CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training,
  by Xin Wang, Yasheng Wang, Yao Wan, Jiawei Wang, Pingyi Zhou, Li Li, Hao Wu and Jin Liu
  
  
• UniXcoder: Unified Cross-Modal Pre-training for Code Representation,
  by Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou and Jian Yin
  
  
• CIRCLE: continual repair across programming languages,
  by Wei Yuan, Quanjun Zhang, Tieke He, Chunrong Fang, Nguyen Quoc Viet Hung, Xiaodong Hao and Hongzhi Yin
  
  
• Detect-Localize-Repair: A Unified Framework for Learning to Debug with CodeT5,
  by Nghi Bui, Yue Wang and Steven C. H. Hoi
  
  
• Multi-view Pre-trained Model for Code Vulnerability Identification,
  by Xuxiang Jiang, Yinhao Xiao, Jun Wang and Wei Zhang
  
  
• Towards JavaScript program repair with Generative Pre-trained Transformer (GPT-2),
  by M'ark Lajk'o, Viktor Csuvik and L'aszl'o Vid'acs
  
  
• Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code,
  by Changan Niu, Chuanyi Li, Bin Luo and Vincent Ng
  
  
• Do Pre-trained Language Models Indeed Understand Software Engineering Tasks?,
  by Yao Li, Tao Zhang, Xiapu Luo, Haipeng Cai, Sen Fang and Dawei Yuan
  
  
• AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language models,
  by Jos'e Antonio Hern'andez L'opez, Martin Weyssow, Jes'us S'anchez Cuadrado and Houari A. Sahraoui
  
  
• CoditT5: Pretraining for Source Code and Natural Language Editing,
  by Jiyang Zhang, Sheena Panthaplackel, Pengyu Nie, Junyi Jessy Li and Milos Gligoric
  
  
• Compressing Pre-trained Models of Code into 3 MB,
  by Jieke Shi, Zhou Yang, Bowen Xu, Hong Jin Kang and David Lo
  
  
• AUGER: automatically generating review comments with pre-training models,
  by Lingwei Li, Li Yang, Huaxi Jiang, Jun Yan, Tiejian Luo, Zihan Hua, Geng Liang and Chun Zuo
  
  
• Automating code review activities by large-scale pre-training,
  by Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green et al.
  
  
• Diet code is healthy: simplifying programs for pre-trained models of code,
  by Zhaowei Zhang, Hongyu Zhang, Beijun Shen and Xiaodong Gu
  
  
• NatGen: generative pre-training by "naturalizing" source code,
  by Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar T. Devanbu and Baishakhi Ray
  
  
• Bridging Pre-trained Models and Downstream Tasks for Source Code Understanding,
  by Deze Wang, Zhouyang Jia, Shanshan Li, Yue Yu, Yun Xiong, Wei Dong and Xiangke Liao
  
  
• Jigsaw: Large Language Models meet Program Synthesis,
  by Naman Jain, Skanda Vaidyanath, Arun Shankar Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram K. Rajamani and Rahul Sharma
  
  
• Natural Attack for Pre-trained Models of Code,
  by Zhou Yang, Jieke Shi, Junda He and David Lo
  
  
• Using Pre-Trained Models to Boost Code Review Automation,
  by Rosalia Tufano, Simone Masiero, Antonio Mastropaolo, Luca Pascarella, Denys Poshyvanyk and Gabriele Bavota
  
  
• What Do They Capture? - A Structural Analysis of Pre-Trained Language Models for Source Code,
  by Yao Wan, Wei Zhao, Hongyu Zhang, Yulei Sui, Guandong Xu and Hai Jin
  
  
• A Survey on Knowledge Graph-Based Recommender Systems,
  by Qingyu Guo, Fuzhen Zhuang, Chuan Qin, Hengshu Zhu, Xing Xie, Hui Xiong and Qing He
  
  
• Are Graph Augmentations Necessary?: Simple Graph Contrastive Learning for Recommendation,
  by Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Lizhen Cui and Quoc Viet Hung Nguyen
  
  
• Disentangled Representations Learning for Multi-Target Cross-Domain Recommendation,
  by Guo, Xiaobo, Li, Shaoshuai, Guo, Naicheng, Cao, Jiangxia, Liu, Xiaolei, Ma, Qiongxu, Gan, Runsheng and Zhao, Yunan
  
  
• DiSTRICT: Dialogue State Tracking with Retriever Driven In-Context Tuning,
  by Praveen Venkateswaran, Evelyn Duesterwald and Vatche Isahagian
  
  
• Does GPT-3 Generate Empathetic Dialogues? A Novel In-Context Example Selection Method and Automatic Evaluation Metric for Empathetic Dialogue Generation,
  by Young-Jun Lee, Chae-Gyun Lim and Ho-Jin Choi
  
  
• Fusing Task-Oriented and Open-Domain Dialogues in Conversational Agents,
  by Tom Young, Frank Xing, Vlad Pandelea, Jinjie Ni and Erik Cambria
  
  
• GODEL: Large-Scale Pre-Training for Goal-Directed Dialog,
  by Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan et al.
  
  
• Mind the Knowledge Gap: A Survey of Knowledge-enhanced Dialogue Systems,
  by Sagi Shaier, Lawrence Hunter and Katharina Kann
  
  
• Rethinking Reinforcement Learning for Recommendation: A Prompt Perspective,
  by Xin Xin, Tiago Pimentel, Alexandros Karatzoglou, Pengjie Ren, Konstantina Christakopoulou and Zhaochun Ren
  
  
• Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models,
  by Sami Sarsa, Paul Denny, Arto Hellas and Juho Leinonen
  
  
• Continual Pre-training of Language Models for Math Problem Understanding with Syntax-Aware Memory Network,
  by Zheng Gong, Kun Zhou, Xin Zhao, Jing Sha, Shijin Wang and Ji-Rong Wen
  
  
• Generative Knowledge Graph Construction: A Review,
  by Hongbin Ye, Ningyu Zhang, Hui Chen and Huajun Chen
  
  
• Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango,
  by Aman Madaan and Amir Yazdanbakhsh
  
  
• Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters,
  by Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer and Huan Sun
  
  
• Prompting GPT-3 To Be Reliable,
  by Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan L. Boyd-Graber and Lijuan Wang
  
  
• PaLM: Scaling Language Modeling with Pathways,
  by Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung et al.
  
  
• Word-Label Alignment for Event Detection: A New Perspective via Optimal Transport,
  by Amir Pouran Ben Veyseh and Thien Huu Nguyen
  
  
• Learning Cross-Task Dependencies for Joint Extraction of Entities, Events, Event Arguments, and Relations,
  by Minh Van Nguyen, Bonan Min, Franck Dernoncourt and Thien Nguyen
  
  
• CLIP-Event: Connecting Text and Images with Event Structures,
  by Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji et al.
  
  
• Selecting Optimal Context Sentences for Event-Event Relation Extraction,
  by Hieu Man, Nghia Trung Ngo, Linh Ngo Van and Thien Huu Nguyen
  
  
• Multilingual SubEvent Relation Extraction: A Novel Dataset and Structure Induction Method,
  by Viet Dac Lai, Hieu Man, Linh Ngo Van, Franck Dernoncourt and Thien Nguyen
  
  
• Event Causality Identification via Derivative Prompt Joint Learning,
  by Shirong Shen, Heng Zhou, Tongtong Wu and Guilin Qi
  
  
• Improved logical reasoning of language models via differentiable symbolic programming,
  by Zhang, Hanlin, Li, Ziyang, Huang, Jiani, Naik, Mayur and Xing, Eric
  
  
• LILA: A Unified Benchmark for Mathematical Reasoning,
  by Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord et al.
  
  
• Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations,
  by Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras and Yejin Choi
  
  
• LAMBADA: Backward Chaining for Automated Reasoning in Natural Language,
  by Seyed Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu and Deepak Ramachandran
  
  
• MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generation,
  by Swarnadeep Saha, Xinyan Velocity Yu, Mohit Bansal, Ramakanth Pasunuru and Asli Celikyilmaz
  
  
• Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks,
  by Wenhu Chen, Xueguang Ma, Xinyi Wang and William W. Cohen
  
  
• The Impact of Symbolic Representations on In-context Learning for Few-shot Reasoning,
  by Hanlin Zhang, Yi-Fan Zhang, Li Erran Li and Eric P. Xing
  
  
• Towards Reasoning in Large Language Models: A Survey,
  by Jie Huang and Kevin Chen-Chuan Chang
  
  
• UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression,
  by Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen and Xiaodan Liang
  
  
• Least-to-Most Prompting Enables Complex Reasoning in Large Language Models,
  by Denny Zhou, Nathanael Sch"arli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet et al.
  (1) 两阶段的prompt，第一阶段问题分解（通过in-context learning实现，context中包含了其他问题的分解示例），对于每个问题，分解出回答该问题需要先回答什么子问题；
(2) 在第二阶段中，从后往前依次解决子问题，同样通过in-context learing得到，每次LLM的回答会参与组成下一个问题的prompt。
  
  
• Rationale-Augmented Ensembles in Language Models,
  by Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi and Denny Zhou
  
  
• Are Visual-Linguistic Models Commonsense Knowledge Bases?,
  by Hsiu-Yu Yang and Carina Silberer
  
  
• What Makes Good In-Context Examples for GPT-3?,
  by Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin and Weizhu Chen
  (1) 探索了在in-context learning中什么样的demonstration example可以对GPT-3的效果取得帮助；
(2) 利用roberta对样本进行编码，并计算demonstration与test example的向量距离（欧氏距离），最终发现与test example越相近的demonstration越能取得较好的效果。
  
  
• Fast Changeset-based Bug Localization with BERT,
  by Agnieszka Ciborowska and Kostadin Damevski
  
  
• Star: Self-taught reasoner bootstrapping reasoning with reasoning,
  by Zelikman, Eric, Mu, Jesse, Goodman, Noah D and Wu, Yuhuai Tony
  
  
• A Domain Knowledge Enhanced Pre-Trained Language Model for Vertical Search: Case Study on Medicinal Products,
  by Kesong Liu, Jianhui Jiang and Feifei Lyu
  
  
• Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding,
  by Jianing Wang, Wenkang Huang, Minghui Qiu, Qiuhui Shi, Hongbin Wang, Xiang Li and Ming Gao
  
  
• Large language models are few-shot clinical information extractors,
  by Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim and David A. Sontag
  
  
• Snapshot-Guided Domain Adaptation for ELECTRA,
  by Daixuan Cheng, Shaohan Huang, Jianfeng Liu, Yuefeng Zhan, Hao Sun, Furu Wei, Denvy Deng and Qi Zhang
  
  
• Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again,
  by Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun and Yu Su
  
  
• VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding,
  by Dou Hu, Xiaolong Hou, Xiyang Du, Mengyuan Zhou, Lianxin Jiang, Yang Mo and Xiaofeng Shi
  
  
• Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought,
  by Abulhair Saparov and He He
  
  
• Towards the Generation of Musical Explanations with GPT-3,
  by Stephen James Krol, Maria Teresa Llano and Jon McCormack
  
  
• Plex: Towards Reliability using Pretrained Large Model Extensions,
  by Dustin Tran, Jeremiah Z. Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang et al.
  
  
• The unreliability of explanations in few-shot prompting for textual reasoning,
  by Ye, Xi and Durrett, Greg
  
  
• Is GPT-3 a Good Data Annotator?,
  by Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq R. Joty and Boyang Li
  
  
• Is GPT-3 a Psychopath? Evaluating Large Language Models from a Psychological Perspective,
  by Xingxuan Li, Yutong Li, Linlin Liu, Lidong Bing and Shafiq R. Joty
  
  
• Improving Supervised Drug-Protein Relation Extraction with Distantly Supervised Models,
  by Naoki Iinuma, Makoto Miwa and Yutaka Sasaki
  
  
• Comparing Encoder-Only and Encoder-Decoder Transformers for Relation Extraction from Biomedical Texts: An Empirical Study on Ten Benchmark Datasets,
  by Mourad Sarrouti, Carson Tao and Yoann Mamy Randriamihaja
  
  
• Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers,
  by Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui and Furu Wei
  (1) 与The Dual Form of Neural Networks Revisited结合一起看，可以进一步理解in-context learning，通过与NN线性层对偶形式的类比，可以将ICL流程描述为：1. 基于Transformer的预训练语言模型作为元优化器；2. 通过正向计算，根据示范例子产生元梯度；3. 通过关注，将元梯度应用于原始语言模型，建立一个ICL模型；
(2)与Fine-tune类似，ICL也是在zero-shot learning参数的基础上，提供了一个更新量。
  
  
• LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5,
  by Chengwei Qin and Shafiq Joty
  We define a challenging yet practical problem as Lifelong Few-shot Language Learning and propose a unified framework for it based on prompt tuning of T5.
  
  
• Towards Continual Knowledge Learning of Language Models,
  by Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun KIM, Stanley Jungkyu Choi and Minjoon Seo
  We propose a novel continual learning formulation named Continual Knowledge Learning which allows large language models to constantly obtain new and updated knowledge while mitigating forgetting of previous learned time-invariant knowledge.
  
  
• Pretrained Language Model in Continual Learning: A Comparative Study,
  by Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi and Gholamreza Haffari
  To explore the layer-wise property of pretrained languge models in continual learning, we thoroughly compare the continual learning performance over the combination of 5 PLMs and 4 veins of CL methods on 3 benchmarks in 2 typical incremental settings.
  
  
• Efficient Large Scale Language Modeling with Mixtures of Experts,
  by Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du et al.
  
  
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback,
  by Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort et al.
  
  
• Ask Me Anything: A simple strategy for prompting language models,
  by Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel J. Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala et al.
  
  
• Large Language Models are Zero-Shot Reasoners,
  by Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo and Yusuke Iwasawa
  
  
• Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning,
  by Antonia Creswell, Murray Shanahan and Irina Higgins
  
  
• Emergent Abilities of Large Language Models,
  by Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma et al.
  
  
• JiuZhang: A Chinese Pre-trained Language Model for Mathematical Problem Understanding,
  by Wayne Xin Zhao, Kun Zhou, Zheng Gong, Beichen Zhang, Yuanhang Zhou, Jing Sha, Zhigang Chen, Shijin Wang et al.
  
  
• Large Language Models Are Reasoning Teachers,
  by Namgyu Ho, Laura Schmid and Se-Young Yun
  
  
• Large Language Models are reasoners with Self-Verification,
  by Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu and Jun Zhao
  
  
• Reasoning with Language Model Prompting: A Survey,
  by Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang et al.
  
  
• PAL: Program-aided Language Models,
  by Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan and Graham Neubig
  
  
• Large Language Models are few(1)-shot Table Reasoners,
  by Wenhu Chen
  
  
• Large Language Models Can Self-Improve,
  by Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu and Jiawei Han
  
  
• MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation,
  by Simiao Zuo, Qingru Zhang, Chen Liang, Pengcheng He, Tuo Zhao and Weizhu Chen
  
  
• Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints,
  by Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani et al.
  
  
• Less is More: Task-aware Layer-wise Distillation for Language Model Compression,
  by Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen and Tuo Zhao
  
  
• Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts,
  by Tao Zhong, Zhixiang Chi, Li Gu, Yang Wang, Yuanhao Yu and Jin Tang
  
  
• The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention,
  by Kazuki Irie, R'obert Csord'as and J"urgen Schmidhuber
  (1) 很有意思的一篇，回顾神经网络（NN）线性层Y=WX（省略偏置b）的原始形式与对偶形式，两种形式完全等价；
(2) 从对偶形式中可以发现，通过反向传播训练的NN线性层的输出主要是该层在训练期间的训练误差信号et的线性组合，其中权重是通过比较测试查询x和每个训练输入计算出来的；进一步可以得出，如果测试时输入的x和训练时的输入是正交的，那么梯度下降所得到的参数更新对于该样本x完全没有影响。
  
  
• OPT: Open Pre-trained Transformer Language Models,
  by Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab et al.
  
  
• Fine-tuning Pre-trained Language Models with Noise Stability Regularization,
  by Hang Hua, Xingjian Li, Dejing Dou, Cheng-Zhong Xu and Jiebo Luo
  
  
• ThinkSum: Probabilistic reasoning over sets using large language models,
  by Batu Ozturkler, Nikolay Malkin, Zhen Wang and Nebojsa Jojic
  
  
• TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models,
  by Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim and Minjoon Seo
  
  
• StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models,
  by Adam Liska, Tom'as Kocisk'y, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d'Autume, Tim Scholtes et al.
  
  
• CN-AutoMIC: Distilling Chinese Commonsense Knowledge from Pretrained Language Models,
  by Chenhao Wang, Jiachun Li, Yubo Chen, Kang Liu and Jun Zhao
  
  
• Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal View,
  by Boxi Cao, Hongyu Lin, Xianpei Han, Fangchao Liu and Le Sun
  
  
• Fast Model Editing at Scale,
  by Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn and Christopher D. Manning
  
  
• Memory-Based Model Editing at Scale,
  by Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning and Chelsea Finn
  
  
• Locating and editing factual associations in gpt,
  by Meng, Kevin, Bau, David, Andonian, Alex J and Belinkov, Yonatan
  
  
• A Survey on Retrieval-Augmented Text Generation,
  by Huayang Li, Yixuan Su, Deng Cai, Yan Wang and Lemao Liu
  
  
• Retrieval-Augmented Multimodal Language Modeling,
  by Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer et al.
  
  
• Atlas: Few-shot learning with retrieval augmented language models,
  by Izacard, Gautier, Lewis, Patrick, Lomeli, Maria, Hosseini, Lucas, Petroni, Fabio, Schick, Timo, Dwivedi-Yu, Jane, Joulin, Armand et al.
  
  
• Training Language Models with Memory Augmentation,
  by Zexuan Zhong, Tao Lei and Danqi Chen
  
  
• Improving Language Models by Retrieving from Trillions of Tokens,
  by Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau et al.
  
  
• Factuality Enhanced Language Models for Open-Ended Text Generation,
  by Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Mohammad Shoeybi and Bryan Catanzaro
  
  
• Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors,
  by Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim and Marzyeh Ghassemi
  
  
• Calibrating Factual Knowledge in Pretrained Language Models,
  by Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui and Lei Li
  
  
• Can Language Models Serve as Temporal Knowledge Bases?,
  by Ruilin Zhao, Feng Zhao, Guandong Xu, Sixiao Zhang and Hai Jin
  
  
• Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries,
  by Xiao Liu, Shiyu Zhao, Kai Su, Yukuo Cen, Jiezhong Qiu, Mengdi Zhang, Wei Wu, Yuxiao Dong et al.
  
  
• Sequence-to-Sequence Knowledge Graph Completion and Question Answering,
  by Apoorv Saxena, Adrian Kochsiek and Rainer Gemulla
  
  
• Dict-BERT: Enhancing Language Model Pre-training with Dictionary,
  by Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu, Michael Zeng and Meng Jiang
  
  
• Contrastive Language-Image Pre-Training with Knowledge Graphs,
  by Xuran Pan, Tianzhu Ye, Dongchen Han, Shiji Song and Gao Huang
  
  
• P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Prompts,
  by Benjamin Newman, Prafulla Kumar Choubey and Nazneen Rajani
  
  
• Finding Structural Knowledge in Multimodal-BERT,
  by Victor Milewski, Miryam de Lhoneux and Marie-Francine Moens
  
  
• GreaseLM: Graph REASoning Enhanced Language Models,
  by Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning and Jure Leskovec
  
  
• Reframing Instructional Prompts to GPTk's Language,
  by Daniel Khashabi, Chitta Baral, Yejin Choi and Hannaneh Hajishirzi
  
  
• Generated Knowledge Prompting for Commonsense Reasoning,
  by Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi and Hannaneh Hajishirzi
  
  
• Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering,
  by Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh Hajishirzi and Yejin Choi
  
  
• Symbolic Knowledge Distillation: from General Language Models to Commonsense Models,
  by Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck et al.
  
  
• I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation,
  by Chandra Bhagavatula, Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West et al.
  
  
• RealTime QA: What's the Answer Right Now?,
  by Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir R. Radev, Noah A. Smith et al.
  
  
• Generating Sequences by Learning to Self-Correct,
  by Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi and Yejin Choi
  
  
• Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization,
  by Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kiant'e Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi and Yejin Choi
  
  
• Quark: Controllable Text Generation with Reinforced Unlearning,
  by Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Ammanabrolu and Yejin Choi
  
  
• GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models,
  by Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li and Kai-Wei Chang
  
  
• RobustLR: A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners,
  by Soumya Sanyal, Zeyi Liao and Xiang Ren
  
  
• Domain Knowledge Transferring for Pre-trained Language Model via Calibrated Activation Boundary Distillation,
  by Dongha Choi, Hongseok Choi and Hyunju Lee
  
  
• Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey,
  by Prajjwal Bhargava and Vincent Ng
  
  
• WeLM: A Well-Read Pre-trained Language Model for Chinese,
  by Hui Su, Xiao Zhou, Houjin Yu, Yuwen Chen, Zilin Zhu, Yang Yu and Jie Zhou
  
  
• Teaching language models to support answers with verified quotes,
  by Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, H. Francis Song, Martin Chadwick, Mia Glaese, Susannah Young et al.
  
  
• Improving alignment of dialogue agents via targeted human judgements,
  by Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger et al.
  
  
• Scaling Laws for Reward Model Overoptimization,
  by Gao, Leo, Schulman, John and Hilton, Jacob
  
  
• Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned,
  by Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez et al.
  
  
• Dynamic Planning in Open-Ended Dialogue using Reinforcement Learning,
  by Deborah Cohen, Moonkyung Ryu, Yinlam Chow, Orgad Keller, Ido Greenberg, Avinatan Hassidim, Michael Fink, Yossi Matias et al.
  
  
• Training language models to follow instructions with human feedback,
  by Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal et al.
  
  
• Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences,
  by Erdem Biyik, Dylan P. Losey, Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk and Dorsa Sadigh
  
  
• Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?,
  by Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov and Luke Zettlemoyer
  
  
• Self-adaptive In-context Learning,
  by Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye and Lingpeng Kong
  
  
• How Many Data Samples is an Additional Instruction Worth?,
  by Ravsehaj Singh Puri, Swaroop Mishra, Mihir Parmar and Chitta Baral
  
  
• Careful Data Curation Stabilizes In-context Learning,
  by Ting-Yun Chang and Robin Jia
  
  
• Towards Unified Prompt Tuning for Few-shot Text Classification,
  by Jianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan, Minghui Qiu, Fei Yang, Qiuhui Shi, Songfang Huang et al.
  
  
• Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning,
  by Xiangyu Peng, Chen Xing, Prafulla Kumar Choubey, Chien-Sheng Wu and Caiming Xiong
  
  
• Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale,
  by Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff and Dan Roth
  
  
• Pretrained Transformers Do not Always Improve Robustness,
  by Swaroop Mishra, Bhavdeep Singh Sachdeva and Chitta Baral
  
  
• Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations,
  by Swarnadeep Saha, Peter Hase, Nazneen Rajani and Mohit Bansal
  
  
• Solving Quantitative Reasoning Problems with Language Models,
  by Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil et al.
  
  
• ZeroGen: Efficient Zero-shot Learning via Dataset Generation,
  by Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu and Lingpeng Kong
  
  
• Generating Training Data with Language Models: Towards Zero-Shot Language Understanding,
  by Yu Meng, Jiaxin Huang, Yu Zhang and Jiawei Han
  
  
• SPOT: Knowledge-Enhanced Language Representations for Information Extraction,
  by Jiacheng Li, Yannis Katsis, Tyler Baldwin, Ho-Cheol Kim, Andrew Bartko, Julian J. McAuley and Chun-Nan Hsu
  
  
• Relation-aware Language-Graph Transformer for Question Answering,
  by Jinyoung Park, Hyeong Kyu Choi, Juyeon Ko, Hyeon-Jin Park, Ji-Hoon Kim, Jisu Jeong, Kyung-Min Kim and Hyunwoo J. Kim
  
  
• A Review on Language Models as Knowledge Bases,
  by Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona T. Diab and Marjan Ghazvininejad
  
  
• A Systematic Evaluation of Large Language Models of Code,
  by Frank F. Xu, Uri Alon, Graham Neubig and Vincent J. Hellendoorn
  
  
• Time-Aware Language Models as Temporal Knowledge Bases,
  by Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein and William W. Cohen
  
  
• Prompting as Probing: Using Language Models for Knowledge Base Construction,
  by Dimitrios Alivanistos, Selene Baez Santamar'\ia, Michael Cochez, Jan-Christoph Kalo, Emile van Krieken and Thiviyan Thanapalasingam
  
  
• Towards Robust NLG Bias Evaluation with Syntactically-diverse Prompts,
  by Arshiya Aggarwal, Jiao Sun and Nanyun Peng
  
  
• GPPT: Graph Pre-training and Prompt Tuning to Generalize Graph Neural Networks,
  by Mingchen Sun, Kaixiong Zhou, Xin He, Ying Wang and Xin Wang
  
  
• LogicNMR: Probing the Non-monotonic Reasoning Ability of Pre-trained Language Models,
  by Yeliang Xiu, Zhanhao Xiao and Yongmei Liu
  
  

2021

• Evaluating Large Language Models Trained on Code,
  by Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond'e de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda et al.
  
  
• FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models,
  by Rakesh Chada and Pradeep Natarajan
  
  
• The Power of Scale for Parameter-Efficient Prompt Tuning,
  by Brian Lester, Rami Al-Rfou and Noah Constant
  
  
• Controllable Open-ended Question Generation with A New Question Type Ontology,
  by Shuyang Cao and Lu Wang
  
  
• All NLP Tasks Are Generation Tasks: A General Pretraining Framework,
  by Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang and Jie Tang
  
  
• Smelting Gold and Silver for Improved Multilingual AMR-to-Text Generation,
  by Leonardo F. R. Ribeiro, Jonas Pfeiffer, Yue Zhang and Iryna Gurevych
  
  
• PRAL: A Tailored Pre-Training Model for Task-Oriented Dialog Generation,
  by Jing Gu, Qingyang Wu, Chongruo Wu, Weiyan Shi and Zhou Yu
  
  
• DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances,
  by Xiaodong Gu, Kang Min Yoo and Jung-Woo Ha
  
  
• DYPLOC: Dynamic Planning of Content Using Mixed Language Models for Text Generation,
  by Xinyu Hua, Ashwin Sreevatsa and Lu Wang
  
  
• Latent Reasoning for Low-Resource Question Generation,
  by Xinting Huang, Jianzhong Qi, Yu Sun and Rui Zhang
  
  
• JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs,
  by Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Linfeng Song, Xiaoyan Zhu and Minlie Huang
  
  
• A Distributional Approach to Controlled Text Generation,
  by Muhammad Khalifa, Hady Elsahar and Marc Dymetman
  
  
• TextBox: A Unified, Modularized, and Extensible Framework for Text Generation,
  by Junyi Li, Tianyi Tang, Gaole He, Jinhao Jiang, Xiaoxuan Hu, Puzhao Xie, Zhipeng Chen, Zhuohao Yu et al.
  
  
• Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models,
  by Junyi Li, Tianyi Tang, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan and Ji-Rong Wen
  
  
• Knowledge-based Review Generation by Coherence Enhanced Text Planning,
  by Junyi Li, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan and Ji-Rong Wen
  
  
• Prefix-Tuning: Optimizing Continuous Prompts for Generation,
  by Xiang Lisa Li and Percy Liang
  
  
• GLGE: A New General Language Generation Evaluation Benchmark,
  by Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu et al.
  
  
• A Three-Stage Learning Framework for Low-Resource Knowledge-Grounded Dialogue Generation,
  by Shilei Liu, Xiaofeng Zhao, Bochao Li, Feiliang Ren, Longhui Zhang and Shujuan Yin
  
  
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation,
  by Fuli Luo, Wei Wang, Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang and Luo Si
  
  
• Ask what's missing and what's useful: Improving Clarification Question Generation using Global Knowledge,
  by Bodhisattwa Prasad Majumder, Sudha Rao, Michel Galley and Julian J. McAuley
  
  
• ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation,
  by Kaushal Kumar Maurya, Maunendra Sankar Desarkar, Yoshinobu Kano and Kumari Deepshikha
  
  
• A Plug-and-Play Method for Controlled Text Generation,
  by Damian Pascual, Beni Egressy, Clara Meister, Ryan Cotterell and Roger Wattenhofer
  
  
• Structural Adapters in Pretrained Language Models for AMR-to-Text Generation,
  by Leonardo F. R. Ribeiro, Yue Zhang and Iryna Gurevych
  
  
• Towards Table-to-Text Generation with Numerical Reasoning,
  by Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura and Hiroya Takamura
  
  
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation,
  by Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen et al.
  
  
• Progressive Generation of Long Text with Pretrained Language Models,
  by Bowen Tan, Zichao Yang, Maruan Al-Shedivat, Eric P. Xing and Zhiting Hu
  
  
• Consistency and Coherency Enhanced Story Generation,
  by Wei Wang, Piji Li and Hai-Tao Zheng
  
  
• Structure-Aware Pre-Training for Table-to-Text Generation,
  by Xinyu Xing and Xiaojun Wan
  
  
• AugNLG: Few-shot Natural Language Generation using Self-trained Data Augmentation,
  by Xinnuo Xu, Guoyin Wang, Young-Bum Kim and Sungjin Lee
  
  
• DeepRapper: Neural Rap Generation with Rhyme and Rhythm Modeling,
  by Lanqing Xue, Kaitao Song, Duocai Wu, Xu Tan, Nevin L. Zhang, Tao Qin, Wei-Qiang Zhang and Tie-Yan Liu
  
  
• FastSeq: Make Sequence Generation Faster,
  by Yu Yan, Fei Hu, Jiusheng Chen, Nikhil Bhendawade, Ting Ye, Yeyun Gong, Nan Duan, Desheng Cui et al.
  
  
• A Simple and Efficient Multi-Task Learning Approach for Conditioned Dialogue Generation,
  by Yan Zeng and Jian-Yun Nie
  
  
• DSGPT: Domain-Specific Generative Pre-Training of Transformers for Text Generation in E-commerce Title and Review Summarization,
  by Xueying Zhang, Yunjiang Jiang, Yue Shang, Zhaomeng Cheng, Chi Zhang, Xiaochuan Fan, Yun Xiao and Bo Long
  
  
• Preparing lessons: Improve knowledge distillation with better supervision,
  by Tiancheng Wen, Shenqi Lai and Xueming Qian
  
  
• Ditto: Fair and Robust Federated Learning Through Personalization,
  by Tian Li, Shengyuan Hu, Ahmad Beirami and Virginia Smith
  
  
• Fine-tuning is Fine in Federated Learning,
  by Gary Cheng, Karan N. Chadha and John C. Duchi
  
  
• Federated Learning with Fair Averaging,
  by Zheng Wang, Xiaoliang Fan, Jianzhong Qi, Chenglu Wen, Cheng Wang and Rongshan Yu
  
  
• Elastic Deep Learning Using Knowledge Distillation with Heterogeneous Computing Resources,
  by Daxiang Dong, Ji Liu, Xi Wang, Weibao Gong, An Qin, Xingjian Li, Dianhai Yu, Patrick Valduriez et al.
  
  
• GRACE: A Compressed Communication Framework for Distributed Machine Learning,
  by Hang Xu, Chen-Yu Ho, Ahmed M. Abdelmoniem, Aritra Dutta, El Houcine Bergou, Konstantinos Karatsenidis, Marco Canini and Panos Kalnis
  
  
• Load Balancing Optimization for Transformer in Distributed Environment,
  by Delu Ma, Zhou Lei, Shengbo Chen and Peng Wang
  
  
• CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation,
  by Yue Wang, Weishi Wang, Shafiq R. Joty and Steven C. H. Hoi
  
  
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation,
  by Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain et al.
  
  
• Unified Pre-training for Program Understanding and Generation,
  by Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray and Kai-Wei Chang
  
  
• GraphCodeBERT: Pre-training Code Representations with Data Flow,
  by Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan et al.
  
  
• CLSEBERT: Contrastive Learning for Syntax Enhanced Code Pre-Trained Model,
  by Xin Wang, Yasheng Wang, Pingyi Zhou, Fei Mi, Meng Xiao, Yadao Wang, Li Li, Xiao Liu et al.
  
  
• Applying CodeBERT for Automated Program Repair of Java Simple Bugs,
  by Ehsan Mashhadi and Hadi Hemmati
  
  
• A model with iterative trials for correcting logic errors in source code,
  by Matsumoto, Taku, Watanobe, Yutaka and Nakamura, Keita
  
  
• DeepDebug: Fixing Python Bugs Using Stack Traces, Backtranslation, and Code Skeletons,
  by Dawn Drain, Colin B. Clement, Guillermo Serrato and Neel Sundaresan
  
  
• Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviews,
  by Mohammad Abdul Hadi and Fatemeh H. Fard
  
  
• What do pre-trained code models know about code?,
  by Anjan Karmakar and Romain Robbes
  
  
• Traceability Transformed: Generating more Accurate Links with Pre-Trained BERT Models,
  by Jinfeng Lin, Yalin Liu, Qingkai Zeng, Meng Jiang and Jane Cleland-Huang
  
  
• Less Is More: ClipBERT for Video-and-Language Learning via Sparse Sampling,
  by Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal and Jingjing Liu
  
  
• Transformer is All You Need: Multimodal Multitask Learning with a Unified Transformer,
  by Ronghang Hu and Amanpreet Singh
  
  
• Advances and Challenges in Conversational Recommender Systems: A Survey,
  by Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke and Tat-Seng Chua
  
  
• Dialogue State Tracking with a Language Model using Schema-Driven Prompting,
  by Chia-Hsuan Lee, Hao Cheng and Mari Ostendorf
  
  
• Few-Shot Bot: Prompt-Based Learning for Dialogue Systems,
  by Andrea Madotto, Zhaojiang Lin, Genta Indra Winata and Pascale Fung
  
  
• Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems,
  by Derek Chen, Howard Chen, Yi Yang, Alexander Lin and Zhou Yu
  
  
• Fine-grained Post-training for Improving Retrieval-based Dialogue Systems,
  by Janghoon Han, Taesuk Hong, Byoungjae Kim, Youngjoong Ko and Jungyun Seo
  
  
• Pre-training Graph Transformer with Multimodal Side Information for Recommendation,
  by Yong Liu, Susen Yang, Chenyi Lei, Guoxin Wang, Haihong Tang, Juyong Zhang, Aixin Sun and Chunyan Miao
  
  
• Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey,
  by Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, Vinay Adiga and Erik Cambria
  
  
• Slot Self-Attentive Dialogue State Tracking,
  by Fanghua Ye, Jarana Manotumruksa, Qiang Zhang, Shenghui Li and Emine Yilmaz
  
  
• Pretraining the Noisy Channel Model for Task-Oriented Dialogue,
  by Qi Liu, Lei Yu, Laura Rimell and Phil Blunsom
  
  
• UBAR: Towards Fully End-to-End Task-Oriented Dialog System with GPT-2,
  by Yunyi Yang, Yunhao Li and Xiaojun Quan
  
  
• Augmenting Open-Domain Event Detection with Synthetic Data from GPT-2,
  by Amir Pouran Ben Veyseh, Minh Van Nguyen, Bonan Min and Thien Huu Nguyen
  
  
• Salience-Aware Event Chain Modeling for Narrative Understanding,
  by Xiyang Zhang, Muhao Chen and Jonathan May
  
  
• Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains,
  by Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong and Furu Wei
  
  
• Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation,
  by Shizhe Diao, Ruijia Xu, Hongjin Su, Yilei Jiang, Yan Song and Tong Zhang
  
  
• Improving Biomedical Pretrained Language Models with Knowledge,
  by Zheng Yuan, Yijia Liu, Chuanqi Tan, Songfang Huang and Fei Huang
  
  
• Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm,
  by Laria Reynolds and Kyle McDonell
  
  
• Combining Ensembles and Data Augmentation Can Harm Your Calibration,
  by Yeming Wen, Ghassen Jerfel, Rafael Muller, Michael W. Dusenberry, Jasper Snoek, Balaji Lakshminarayanan and Dustin Tran
  
  
• Revisiting the Calibration of Modern Neural Networks,
  by Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran and Mario Lucic
  
  
• Soft Calibration Objectives for Neural Networks,
  by Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C. Mozer and Becca Roelofs
  
  
• Want To Reduce Labeling Cost? GPT-3 Can Help,
  by Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu and Michael Zeng
  
  
• Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning,
  by Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu and Lei Shu
  NeurIPS 2021, The key component of CTR is the CL-plugin inserted in BERT. A CL-plugin is a capsule network with a new transfer routing mechanism to encourage knowledge transfer among tasks and also to isolate task-specific knowledge to avoid forgetting.
  
  
• Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning,
  by Jin, Xisen , Lin, Bill Yuchen , Rostami, Mohammad and Ren, Xiang
  We present a new learning setup, Continual Learning of Few-Shot Learners, to address challenges of both learning settings in a unified setup, with a hyper-network for task-specific adapter generation.
  
  
• Analyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models,
  by Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James R. Glass and Fuchun Peng
  Our major finding is that after standard finetuning, the model forgets some of the important language generation skills acquired during large-scale pretraining. We propose an intuitive finetuning strategy named “mix-review”: : For each finetuning epoch, we mix the target dialogue data with a random subset of the pretraining data, mix_ratio is 4, decay is 0.9.
  
  
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters,
  by Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang et al.
  We propose KADAPTER, a framework that retains the original parameters of the pre-trained model fixed
and supports the development of versatile
knowledge-infused model.
  
  
• Domain-Lifelong Learning for Dialogue State Tracking via Knowledge Preservation Networks,
  by Liu, Qingbin , Cao, Pengfei , Liu, Cao , Chen, Jiansong , Cai, Xunliang , Yang, Fan , He, Shizhu , Liu, Kang et al.
  This paper explores Domain-Lifelong Learning for Dialogue State Tracking, we propose Knowledge Preservation Network, which consists of multi-prototype enhanced retrospection and multi-strategy knowledge distillation, to solve the problems of expression diversity and combinatorial explosion in the DLL-DST task
  
  
• CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks,
  by Ke, Zixuan , Liu, Bing , Xu, Hu and Shu, Lei
  The key novelty is a contrastive continual learning method that enables both knowledge transfer across tasks and knowledge distillation from old tasks to the new task, which eliminates the need for task ids in testing.
  
  
• Lifelong Explainer for Lifelong Learners,
  by Situ, Xuelin , Maruf, Sameen , Zukerman, Ingrid , Paris, Cecile and Haffari, Gholamreza
  We propose a novel Lifelong Explanation approach that continuously trains a student explainer under the supervision of a teacher – an arbitrary explanation algorithm – on different tasks undertaken in LL. We also leverage the Experience Replay mechanism to prevent catastrophic forgetting in the student explainer.
  
  
• A Unified Speaker Adaptation Approach for ASR,
  by Yingzhu Zhao, Chongjia Ni, Cheung-Chi Leung, Shafiq R. Joty, Eng Siong Chng and Bin Ma
  Prefix-based user identifier, Continual ASR / Architecture Search / Network Pruning.
  
  
• Dynamic Language Models for Continuously Evolving Content,
  by Amba Hombaiah, Spurthi, Chen, Tao, Zhang, Mingyang, Bendersky, Michael and Najork, Marc
  
  
• Parameter-Efficient Transfer Learning with Diff Pruning,
  by Guo, Demi , Rush, Alexander and Kim, Yoon
  The approach learns a task-specific “diff” vector that extends the original pretrained parameters. As the number of tasks increases, diff pruning remains parameter-efficient, as it requires storing only a small diff vector for each task.
  
  
• Refining Sample Embeddings with Relation Prototypes to Enhance Continual Relation Extraction,
  by Cui, Li , Yang, Deqing , Yu, Jiaxin , Hu, Chengwei , Cheng, Jiayang , Yi, Jingjie and Xiao, Yanghua
  To fully utilize memorized samples, in this paper, we employ relation prototype to extract useful information of each relation.
  
  
• On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation,
  by He, Ruidan , Liu, Linlin , Ye, Hai , Tan, Qingyu , Ding, Bosheng , Cheng, Liying , Low, Jiawei , Bing, Lidong et al.
  we first show that adapter-based tuning better mitigates forgetting issues than fine-tuning since it yields representations with less deviation from those generated by the initial PrLM. Effectiveness: it tendsto outperform fine-tuning on both low-resource and cross-lingual tasks; 2 it demonstrates higher stability under different learning rates compared to fine-tuning.
  
  
• Rational LAMOL: A Rationale-based Lifelong Learning Framework,
  by Kanwatchara, Kasidis , Horsuwan, Thanapapas , Lertvittayakumjorn, Piyawat , Kijsirikul, Boonserm and Vateekul, Peerapon
  Rational LAMOL enhances LAMOL, a recent LL model, by applying critical freezing guided by human rationales. When the human rationales are not available, we propose exploiting unsupervised generated rationales as substitutions.
  
  
• Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution,
  by Garcia, Xavier , Constant, Noah , Parikh, Ankur and Firat, Orhan
  Introducing the catastrophic forgetting problem in incremental multi-language translation, and utilizing a vocabulary substitution manner to alleviate the above problem.
  
  
• Continual Learning for Text Classification with Information Disentanglement Based Regularization,
  by Huang, Yufan , Zhang, Yanzhe , Chen, Jiaao , Wang, Xuezhi and Yang, Diyi
  Proposing a regularization-based method for continual text classification, introducing the next sentence prediction and task id prediction as auxiliary tasks.
  
  
• Incremental Few-shot Text Classification with Multi-round New Classes: Formulation, Dataset and System,
  by Xia, Congying , Yin, Wenpeng , Feng, Yihao and Yu, Philip
  Proposing a new setting and respective benchmark for few-shot incremental text classification, modeling continual text classification with text entailment.
  
  
• Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding,
  by Hua, Ting , Shen, Yilin , Zhao, Changsheng , Hsu, Yen-Chang and Jin, Hongxia
  Inspired by EWC and proposing a hyperparameter-free (Fisher information-based) sampling method for memory replay.
  
  
• Lifelong Knowledge-Enriched Social Event Representation Learning,
  by Vijayaraghavan, Prashanth and Roy, Deb
  Proposing a rehearsal-based method, i.e.,Domain-Representative Episodic Memory Replay (DR-EMR), for lifelong event representation with embedding alignment and external social commonsense knowledge.
  
  
• Lifelong Intent Detection via Multi-Strategy Rebalancing,
  by Qingbin Liu, Xiaoyan Yu, Shizhu He, Kang Liu and Jun Zhao
  We propose the lifelong intent detection task to handle continually emerging user intents. And, we propose multistrategy rebalancing to address multiple adverse effects caused by the data imbalance problem.
  
  
• GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation,
  by Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee and Woo-Myoung Park
  
  
• Do Language Models Perform Generalizable Commonsense Inference?,
  by Peifeng Wang, Filip Ilievski, Muhao Chen and Xiang Ren
  
  
• Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries,
  by Benjamin Heinzerling and Kentaro Inui
  
  
• Editing Factual Knowledge in Language Models,
  by Nicola De Cao, Wilker Aziz and Ivan Titov
  
  
• Relational World Knowledge Representation in Contextual Language Models: A Review,
  by Tara Safavi and Danai Koutra
  
  
• RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms,
  by Pei Zhou, Rahul Khanna, Seyeon Lee, Bill Yuchen Lin, Daniel Ho, Jay Pujara and Xiang Ren
  
  
• Pre-training Text-to-Text Transformers for Concept-centric Common Sense,
  by Wangchunshu Zhou, Dong-Ho Lee, Ravi Kiran Selvam, Seyeon Lee and Xiang Ren
  
  
• Mention Flags (MF): Constraining Transformer-based Text Generators,
  by Yufei Wang, Ian D. Wood, Stephen Wan, Mark Dras and Mark Johnson
  
  
• Prompting Contrastive Explanations for Commonsense Reasoning Tasks,
  by Bhargavi Paranjape, Julian Michael, Marjan Ghazvininejad, Hannaneh Hajishirzi and Luke Zettlemoyer
  
  
• WebGPT: Browser-assisted question-answering with human feedback,
  by Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain et al.
  
  
• Recursively Summarizing Books with Human Feedback,
  by Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike and Paul F. Christiano
  
  
• Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases,
  by Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue and Jin Xu
  
  
• Can Language Models be Biomedical Knowledge Bases?,
  by Mujeen Sung, Jinhyuk Lee, Sean S. Yi, Minji Jeon, Sungdong Kim and Jaewoo Kang
  
  
• Transformer Feed-Forward Layers Are Key-Value Memories,
  by Mor Geva, Roei Schuster, Jonathan Berant and Omer Levy
  
  
• Measuring and Improving Consistency in Pretrained Language Models,
  by Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard H. Hovy, Hinrich Sch"utze and Yoav Goldberg
  
  

2020

• Parsing as Pretraining,
  by David Vilares, Michalina Strzyz, Anders S\ogaard and Carlos G'omez-Rodr'\iguez
  
  
• Language Models are Few-Shot Learners,
  by Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam et al.
  
  
• Unsupervised Deep Learning via Affinity Diffusion,
  by Jiabo Huang, Qi Dong, Shaogang Gong and Xiatian Zhu
  
  
• PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable,
  by Siqi Bao, Huang He, Fan Wang, Hua Wu and Haifeng Wang
  
  
• Evaluation of Text Generation: A Survey,
  by Asli Celikyilmaz, Elizabeth Clark and Jianfeng Gao
  
  
• KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation,
  by Wenhu Chen, Yu Su, Xifeng Yan and William Yang Wang
  
  
• Distilling Knowledge Learned in BERT for Text Generation,
  by Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu and Jingjing Liu
  
  
• Logic2Text: High-Fidelity Natural Language Generation from Logical Forms,
  by Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan and William Yang Wang
  
  
• Cross-Lingual Natural Language Generation via Pre-Training,
  by Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, Xian-Ling Mao and Heyan Huang
  
  
• Plug and Play Language Models: A Simple Approach to Controlled Text Generation,
  by Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski and Rosanne Liu
  
  
• Neural Language Generation: Formulation, Methods, and Evaluation,
  by Cristina Garbacea and Qiaozhu Mei
  
  
• TableGPT: Few-shot Table-to-Text Generation with Table Structure Reconstruction and Content Matching,
  by Heng Gong, Yawei Sun, Xiaocheng Feng, Bing Qin, Wei Bi, Xiaojiang Liu and Ting Liu
  
  
• A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation,
  by Jian Guan, Fei Huang, Minlie Huang, Zhihao Zhao and Xiaoyan Zhu
  
  
• Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity,
  by Hamza Harkous, Isabel Groves and Amir Saffari
  
  
• Reformulating Unsupervised Style Transfer as Paraphrase Generation,
  by Kalpesh Krishna, John Wieting and Mohit Iyyer
  
  
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension,
  by Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov and Luke Zettlemoyer
  
  
• Knowledge-Enhanced Personalized Review Generation with Capsule Graph Neural Network,
  by Junyi Li, Siqing Li, Wayne Xin Zhao, Gaole He, Zhicheng Wei, Nicholas Jing Yuan and Ji-Rong Wen
  
  
• Rigid Formats Controlled Text Generation,
  by Piji Li, Haisong Zhang, Xiaojiang Liu and Shuming Shi
  
  
• UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation,
  by Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Xilin Chen and Ming Zhou
  
  
• GPT-too: A Language-Model-First Approach for AMR-to-Text Generation,
  by Manuel Mager, Ram'on Fernandez Astudillo, Tahira Naseem, Md. Arafat Sultan, Young-Suk Lee, Radu Florian and Salim Roukos
  
  
• Few-shot Natural Language Generation for Task-Oriented Dialog,
  by Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng and Jianfeng Gao
  
  
• PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking,
  by Hannah Rashkin, Asli Celikyilmaz, Yejin Choi and Jianfeng Gao
  
  
• Investigating Pretrained Language Models for Graph-to-Text Generation,
  by Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Sch"utze and Iryna Gurevych
  
  
• Leveraging Pre-trained Checkpoints for Sequence Generation Tasks,
  by Sascha Rothe, Shashi Narayan and Aliaksei Severyn
  
  
• T3: Tree-Autoencoder Constrained Adversarial Text Generation for Targeted Attack,
  by Boxin Wang, Hengzhi Pei, Boyuan Pan, Qian Chen, Shuohang Wang and Bo Li
  
  
• MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models,
  by Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar and Bryan Catanzaro
  
  
• StyleDGPT: Stylized Response Generation with Pre-trained Language Models,
  by Ze Yang, Wei Wu, Can Xu, Xinnian Liang, Jiaqi Bai, Liran Wang, Wei Wang and Zhoujun Li
  
  
• Generalized Conditioned Dialogue Generation Based on Pre-trained Language Model,
  by Yan Zeng and Jian-Yun Nie
  
  
• BERTScore: Evaluating Text Generation with BERT,
  by Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger and Yoav Artzi
  
  
• DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation,
  by Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu et al.
  
  
• Improved Knowledge Distillation via Teacher Assistant,
  by Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa and Hassan Ghasemzadeh
  
  
• Regularizing Class-Wise Predictions via Self-Knowledge Distillation,
  by Sukmin Yun, Jongjin Park, Kimin Lee and Jinwoo Shin
  
  
• Collaborative Fairness in Federated Learning,
  by Lingjuan Lyu, Xinyi Xu, Qian Wang and Han Yu
  
  
• Federated Visual Classification with Real-World Data Distribution,
  by Tzu-Ming Harry Hsu, Hang Qi and Matthew Brown
  
  
• DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs,
  by Da Zheng, Chao Ma, Minjie Wang, Jinjing Zhou, Qidong Su, Xiang Song, Quan Gan, Zheng Zhang et al.
  
  
• PyTorch Distributed: Experiences on Accelerating Data Parallel Training,
  by Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith et al.
  
  
• IntelliCode compose: Code Generation using transformer,
  by Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu and Neel Sundaresan
  
  
• Sentiment analysis for software engineering: How far can pre-trained transformer models go?,
  by Zhang, Ting, Xu, Bowen, Thung, Ferdian, Haryono, Stefanus Agus, Lo, David and Jiang, Lingxiao
  
  
• Multi-task Learning based Pre-trained Language Model for Code Completion,
  by Fang Liu, Ge Li, Yunfei Zhao and Zhi Jin
  
  
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning,
  by Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng and Jingjing Liu
  
  
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision,
  by Hao Tan and Mohit Bansal
  
  
• Integrating Multimodal Information in Large Pretrained Transformers,
  by Wasifur Rahman, Md. Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency and Mohammed E. Hoque
  
  
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations,
  by Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei and Jifeng Dai
  
  
• End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2,
  by DongHoon Ham, Jeong-Gwan Lee, Youngsoo Jang and Kee-Eung Kim
  
  
• A Simple Language Model for Task-Oriented Dialogue,
  by Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz and Richard Socher
  
  
• Towards Hands-Free Visual Dialog Interactive Recommendation,
  by Tong Yu, Yilin Shen and Hongxia Jin
  
  
• SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup,
  by Rongzhi Zhang, Yue Yu and Chao Zhang
  
  
• Joint Constrained Learning for Event-Event Relation Extraction,
  by Haoyu Wang, Muhao Chen, Hongming Zhang and Dan Roth
  
  
• ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators,
  by Kevin Clark, Minh-Thang Luong, Quoc V. Le and Christopher D. Manning
  
  
• Revisiting Pre-Trained Models for Chinese Natural Language Processing,
  by Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang and Guoping Hu
  
  
• Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting,
  by Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu and Xiangzhan Yu
  We propose a recall and learn mechanism, which adopts the idea of multi-task learning and jointly learns pretraining tasks and downstream tasks. Specifically, we introduce a Pretraining Simulation mechanism to recall the knowledge from pretraining tasks without data, and an Objective Shifting mechanism to focus the learning on downstream tasks gradually.
  
  
• Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning,
  by Zhaojiang Lin, Andrea Madotto and Pascale Fung
  Proposing an adapter-based method for continual learning in text generation. One of the insights is a frozen PLM can be well-applied in continual learning.
  
  
• An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training,
  by Arumae, Kristjan , Sun, Qing and Bhatia, Parminder
  We find that elastic weight consolidation provides best overall scores yielding only a 0.33% drop in performance across seven generic tasks while remaining competitive in bio-medical tasks.
  
  
• Visually Grounded Continual Learning of Compositional Phrases,
  by Jin, Xisen , Du, Junyi , Sadhu, Arka , Nevatia, Ram and Ren, Xiang
  A novel continual learning setting and a new benchmark for continual caption generation, evaluated with exiting rehearsal-based methods
  
  
• Incremental Event Detection via Knowledge Consolidation Networks,
  by Cao, Pengfei , Chen, Yubo , Zhao, Jun and Wang, Taifeng
  Proposing a hybrid continual learning method for event detection, combining experience replay and Knowledge Distillation, focusing on (1) semantic ambiguity in NLP and (2) data imbalance between memory and current task.
  
  
• A Multi-Task Incremental Learning Framework with Category Name Embedding for Aspect-Category Sentiment Analysis,
  by Dai, Zehui , Peng, Cheng , Chen, Huajie and Ding, Yadong
  Utilizing BERT for sentence and category encoding, preserving category encoding to prevent catastrophic forgetting.
  
  
• Efficient Meta Lifelong-Learning with Limited Memory,
  by Wang, Zirui , Mehta, Sanket Vaibhav , Poczos, Barnabas and Carbonell, Jaime
  A meta learning-enhanced version of MbPA (NeurIPS19), sharing the continual setting as well. Figure 1 is interesting.
  
  
• Lifelong Language Knowledge Distillation,
  by Chuang, Yung-Sung , Su, Shang-Yu and Chen, Yun-Nung
  Proposing a Knowledge Distillation-enhanced Method LLL based on LAMOL (ICLR 2020) model for continual learning, evaluated on text generation and text classification.
  
  
• Distill and Replay for Continual Language Learning,
  by Sun, Jingyuan , Wang, Shaonan , Zhang, Jiajun and Zong, Chengqing
  Proposing a distill and replay method (DnR) which follows the setting of LAMOL. As a distillation-based method, DnR also shows the ability in incrementally compressing the model size while still outperforming most of the baselines.
  
  
• ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding,
  by Sun, Yu, Wang, Shuohuan, Li, Yukun, Feng, Shikun, Tian, Hao, Wu, Hua and Wang, Haifeng
  In order to extract the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which incrementally builds pre-training tasks and then learn pre-trained models on these constructed tasks via continual multi-task learning.
  
  
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention,
  by Pengcheng He, Xiaodong Liu, Jianfeng Gao and Weizhu Chen
  
  
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,
  by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li et al.
  
  
• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts,
  by Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace and Sameer Singh
  
  
• A Primer in BERTology: What We Know About How BERT Works,
  by Anna Rogers, Olga Kovaleva and Anna Rumshisky
  
  
• REALM: Retrieval-Augmented Language Model Pre-Training,
  by Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat and Ming-Wei Chang
  
  
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,
  by Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K"uttler, Mike Lewis et al.
  
  
• CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning,
  by Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi and Xiang Ren
  
  
• Learning to summarize with human feedback,
  by Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei et al.
  
  
• Dialogue Response Ranking Training with Large-Scale Human Feedback Data,
  by Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett and Bill Dolan
  
  
• Pretrained Transformers Improve Out-of-Distribution Robustness,
  by Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan and Dawn Song
  
  
• Thinking Like a Skeptic: Defeasible Inference in Natural Language,
  by Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith and Yejin Choi
  
  

2019

• Language Models are Unsupervised Multitask Learners,
  by Radford, Alec, Wu, Jeffrey, Child, Rewon, Luan, David, Amodei, Dario and Sutskever, Ilya
  
  
• Unified Language Model Pre-training for Natural Language Understanding and Generation,
  by Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou et al.
  
  
• Large-Scale Transfer Learning for Natural Language Generation,
  by Sergey Golovanov, Rauf Kurbanov, Sergey I. Nikolenko, Kyryl Truskovskyi, Alexander Tselousov and Thomas Wolf
  
  
• CTRL: A Conditional Transformer Language Model for Controllable Generation,
  by Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong and Richard Socher
  
  
• Improving Neural Story Generation by Targeted Common Sense Grounding,
  by Huanru Henry Mao, Bodhisattwa Prasad Majumder, Julian J. McAuley and Garrison W. Cottrell
  
  
• MASS: Masked Sequence to Sequence Pre-training for Language Generation,
  by Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu and Tie-Yan Liu
  
  
• Relational Knowledge Distillation,
  by Wonpyo Park, Dongju Kim, Yan Lu and Minsu Cho
  
  
• Revisit Knowledge Distillation: a Teacher-free Framework,
  by Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang and Jiashi Feng
  
  
• Knowledge Distillation via Route Constrained Optimization,
  by Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan and Xiaolin Hu
  
  
• Improving Generalization and Robustness with Noisy Collaboration in Knowledge Distillation,
  by Elahe Arani, Fahad Sarfraz and Bahram Zonooz
  
  
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks,
  by Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova and Jimmy Lin
  
  
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks,
  by Jonathan Frankle and Michael Carbin
  
  
• VisualBERT: A Simple and Performant Baseline for Vision and Language,
  by Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh and Kai-Wei Chang
  
  
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks,
  by Jiasen Lu, Dhruv Batra, Devi Parikh and Stefan Lee
  
  
• VideoBERT: A Joint Model for Video and Language Representation Learning,
  by Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy and Cordelia Schmid
  
  
• Exploring Pre-trained Language Models for Event Extraction and Generation,
  by Sen Yang, Dawei Feng, Linbo Qiao, Zhigang Kan and Dongsheng Li
  
  
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,
  by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova
  
  
• RoBERTa: A Robustly Optimized BERT Pretraining Approach,
  by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis et al.
  
  
• Episodic Memory in Lifelong Language Learning,
  by Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong and Dani Yogatama
  MbPA++. This paper proposes the use of memory (a fixed memory network) in life-long learning to prevent catastrophic forgetting by means of experience replay and local adaptation.
  
  
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,
  by Nils Reimers and Iryna Gurevych
  
  
• Language Models as Knowledge Bases?,
  by Fabio Petroni, Tim Rockt"aschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu and Alexander H. Miller
  
  
• HellaSwag: Can a Machine Really Finish Your Sentence?,
  by Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi and Yejin Choi
  
  
• Fine-Tuning Language Models from Human Preferences,
  by Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano and Geoffrey Irving
  
  
• Barack's Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling,
  by Robert L. Logan IV, Nelson F. Liu, Matthew E. Peters, Matt Gardner and Sameer Singh
  
  

2018

• Generating Wikipedia by Summarizing Long Sequences,
  by Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser and Noam Shazeer
  
  
• Improving language understanding by generative pre-training,
  by Radford, Alec, Narasimhan, Karthik, Salimans, Tim, Sutskever, Ilya and others
  
  
• Born-Again Neural Networks,
  by Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti and Anima Anandkumar
  
  
• Ray: A Distributed Framework for Emerging AI Applications,
  by Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang et al.
  
  

2017

• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer,
  by Sergey Zagoruyko and Nikos Komodakis
  
  
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,
  by Antti Tarvainen and Harri Valpola
  
  
• Deep Mutual Learning,
  by Ying Zhang, Tao Xiang, Timothy M. Hospedales and Huchuan Lu
  
  
• Deep Reinforcement Learning from Human Preferences,
  by Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg and Dario Amodei
  
  

2016

• Deep Model Compression: Distilling Knowledge from Noisy Teachers,
  by Bharat Bhusan Sau and Vineeth N. Balasubramanian
  
  

2015

• Distilling the Knowledge in a Neural Network,
  by Geoffrey E. Hinton, Oriol Vinyals and Jeffrey Dean
  
  

2009

• Zero-shot Learning with Semantic Output Codes,
  by Mark Palatucci, Dean Pomerleau, Geoffrey E. Hinton and Tom M. Mitchell
  
  
• Learning to detect unseen object classes by between-class attribute transfer,
  by Christoph H. Lampert, Hannes Nickisch and Stefan Harmeling