Alignment and Reinforcement Learning with Large Language Models (LLMs)

This repository contains Jupyter notebooks for the courses "Aligning Large Language Models" and "Reinforcement Learning with Large Language Models" by Sinan Ozdemir. Published by Pearson, the course covers effective best practices and industry case studies in using Large Language Models (LLMs).

Aligning Large Language Models

• In-depth exploration of various alignment techniques with hands-on case studies, such as Constitutional AI
• Comprehensive coverage of evaluating alignment, offering specific tools and metrics for continuous assessment and adaptation of LLM alignment strategies
• A focus on ethical considerations and future directions, ensuring participants not only understand the current landscape but are also prepared for emerging trends and challenges in LLM alignment

This class is an intensive exploration into the alignment of Large Language Models (LLMs), a vital topic in modern AI development. Through a combination of theoretical insights and hands-on practice, participants will be exposed to various alignment techniques, including a focus on Constitutional AI, constructing reward mechanisms from human feedback, and instructional alignment. The course will also provide detailed guidance on evaluating alignment, with specific tools and metrics to ensure that models align with desired goals, ethical standards, and real-world applications.

Course Set-Up

• Jupyter notebooks can be run alongside the instructor, but you can also follow along without coding by viewing pre-run notebooks here.

Notebooks

• rlaif.ipynb: Investigating Principles of RLAIF and Constituional AI
• fine_tuning_llama_2: A workshop in fine-tuning Llama 2 with instructional data and incorporating further pre-training to update it's knowledge base

Reinforcement Learning with Large Language Models

• An immersive deep dive into advanced concepts of reinforcement learning in the context of LLMs.
• A practical, hands-on approach to fine-tuning LLMs, with a focus on real-world applications such as generating neutral summaries using T5.
• A unique opportunity to understand and apply innovative concepts like RLHF, RLAIF, and Constitutional AI in reinforcement learning.

This training offers an intensive exploration into the frontier of reinforcement learning techniques with large language models (LLMs). We will explore advanced topics such as Reinforcement Learning with Human Feedback (RLHF), Reinforcement Learning from AI Feedback (RLAIF), and Constitutional AI, and demonstrate practical applications such as fine-tuning open source LLMs like FLAN-T5 and GPT-2. This course is critical for those keen on deepening their understanding of reinforcement learning, its latest trends, and its application to LLMs.

Course Set-Up

• Jupyter notebooks can be run alongside the instructor, but you can also follow along without coding by viewing pre-run notebooks here.

Notebooks

• rl_flan_t5_summaries.ipynb: Working with FLAN-T5 models using Reinforcement Learning

  • Colab Version:

• Fine-tuning the instruction model for the SAWYER bot

  • 

• Training a reward model for the SAWYER bot from human preferences

  • 

• Using Reinforcement Learning from Feedback (RLF) to further align SAWYER

  • 

• Using and Evalauting SAWYER

  • 

For a detailed schedule, refer to the Course Description.

Further Resources

• Other Useful Links