Knowledge Injection in the era of LLMs

knowledge injection in the pre-training or fine-tuning process

The approach in this part mainly involves fine-tuning LLMs using domain-specific data, thereby resulting in numerous vertical domain LLMs. The generation of their dataset can be sourced from domain-specific knowledge graphs or online data, and so on. Here, we mainly list some open-source large models in the medical field.

1. BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining, Briefings in Bioinformatics, 2022
2. DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task, Arxiv, 2023
3. ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge, Arxiv, 2023
4. PMC-LLaMA: Towards Building Open-source Language Models for Medicine, Arxiv, 2023
5. HuatuoGPT, Towards Taming Language Models To Be a Doctor, Arxiv, 2023

knowledge injection with KB

The approach in this part is mainly to integrate the domain knowledge base into LLMs, usually involving the graph related algorithms and the retrieval way from the knowledge base.

1. DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases, ICLR2023
2. Knowledge Solver: Teaching LLMs to Search for Domain Knowledge from Knowledge Graphs, Arxiv
3. KnowledGPT: Enhancing Large Language Models with Retrieval and Storage Access on Knowledge Bases, Arxiv
4. MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Models, Arxiv
5. Augmenting Black-box LLMs with Medical Textbooks for Clinical Question Answering, Arxiv
6. Chain of Knowledge: A Framework for Grounding Large Language Models with Structured Knowledge Bases, Arxiv
7. KITLM: Domain-Specific Knowledge InTegration into Language Models for Question Answering, Arxiv
8. RET-LLM: Towards a General Read-Write Memory for Large Language Models, Arxiv
9. ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings, Arxiv, 2023
10. Leveraging A Medical Knowledge Graph into Large Language Models for Diagnosis Prediction, Arxiv
11. Knowledge-tuning Large Language Models with Structured Medical Knowledge Bases for Reliable Response Generation in Chinese, Arxiv
12. Two is Better Than One: Answering Complex Questions by Multiple Knowledge Sources with Generalized Links, Arxiv

knowlege injection with external knowledge (document corpus or other types of knowledge that is different from KB)

The method in this part mainly integrates externally available domain knowledge into LLMs. Note that this external knowledge is different from the knowledge graph. It is generally presented in the form of natural text (not a graph), so there is no need to use graphs to retrieve or query.

1. Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework, ACL2023
2. Unified Demonstration Retriever for In-Context Learning, ACL2023
3. LLaMAIndex, Github, 2022
4. LangChain, Github, 2022
5. ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory, Arxiv, 2023
6. The CALLA Dataset: Probing LLMs' Interactive Knowledge Acquisition from Chinese Medical Literature, Arxiv, 2023

knowledge injection with model self-driving

This part organizes the methods of model self-driving to obtain domain knowledge. Specifically, this type of method designs prompts to allow the LLM to generate the required domain-related text by itself, and then uses the generated text to perform in-domain tasks.

1. Generate rather than Retrieve: Large Language Models are Strong Context Generators, ICLR2023
2. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, ICLR2023
3. Large Language Models Are Reasoning Teachers, ACL2023
4. Symbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-Step, ACL2023
5. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Arxiv

knowledge probing benchmarks

1. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories, ACL2023
2. Do Large Language Models Know What They Don’t Know?, findings of ACL2023
3. Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation, ACL2023