📔 Database develop RoadMap

Author is a database amateur, so the content is not authoritative, welcome to point out my error.

Since the database course is recommended, only good articles, papers or blogs are listed starting from part 3.

List of Knowledge

• List of Knowledge
• 1. Description
• 2. Recommended Basic Course and Books
  • Pre-Skill
  • 2.1 Computer Organization and Design
    • 2.1.1 Courses
    • 2.1.2 Books
  • 2.2 Computer Operating System
    • 2.2.1 Courses
    • 2.2.2 Books
  • 2.3 Computer Network
    • 2.3.1 Courses
    • 2.3.2 Books
  • 2.4 Data Structure and Algorithmn
    • 2.4.1 Courses
    • 2.4.2 Books
  • 2.5 Algorithms Design and Analysis
    • 2.5.1 Courses
    • 2.5.2 Books
  • 2.6 Database
    • 2.6.1 Course
    • 2.6.2 Books
  • 2.6.3 Papers
  • 2.7 Distributed System
• 3. SQL & Relational Algebra
• 4. DDL & DML
• 5. Relational Model
• 6. Storage management
  • 6.1 Buffer management
  • 6.2 Index structure
  • 6.3 B+ Tree
  • 6.4 B- Tree
  • 6.5 Hash table
  • 6.6 LSM Tree
• 8. Query Evaluation（Processing）
• SQL Query Parser
• Query Executor
• 9. Query Optimization
• Lock manager
• 10. Transaction management
• 11. Network
• 12. Serialization
• 13. Concurrency Control
• 14. Crash Recovery management
• 15. NoSQL
• 16. NewSQL
• 17. Distributed & Paralleled
• 18. AI4DB and DB4AI (frontier tech)
• 15. OLAP、OLTP、HTAP
  • 15.1 OLAP
  • 15.2 OLTP
  • 15.3 HTAP
• 16. Project Source Code Analysis
• 17. Mini-Project Labs

1. Description

This repo will not include knowledge about basic programming language. it mainly focus on direction of database kernal development.

The tutorial will include content:

• Courses
• Blogs
• Papers
• Activities
• Talks

Ok, The repo include two language content：

• English Content
• Chinese Content.

If you have any good materials, welcome to submit a pull request for this repo.

Finally, I would like to thank the professor who provided the Course resource, papper resource Etc. Also thanks the blogger who provided the study notes. Thanks to all content resource providers.

2. Recommended Basic Course and Books

I will not update course notes and labs.

Pre-Skill

• MIT The Missing Semester of Your CS Education
• C/C++
• Go
• Rust

You only need to know a programming language.

2.1 Computer Organization and Design

2.1.1 Courses

University	ID	Name	Time	Comment
CMU	15-213/14-513/15-513	Intro to Computer Systems(ICS)	Fall 2015	CS:APP3e
UC Berkeley	CS61C	Great Ideas in Computer Architecture (Machine Structures)	Spring 2017	N/A
UC Berkeley	CS 152/252A	Computer Architecture and Engineering	Spring 2022	N/A
N/A	N/A	Crash Course Computer Science		bilibili地址

2.1.2 Books

• Reference Course Resources

2.2 Computer Operating System

2.2.1 Courses

University	ID	Name	Time	Comment
Stanford	CS110	Principles of Computer Systems	Winter 2022
MIT	6.828	Operating System Engineering	Fall 2018
UC Berkeley	CS162	Operating Systems and System Programming	Fall 2020
MIT	6.033	http://web.mit.edu/6.033/www/index.shtml	N/A	covers four units of technical content: operating systems, networking, distributed systems, and security

2.2.2 Books

• Reference Course Resources

2.3 Computer Network

2.3.1 Courses

University	ID	Name	Time	Comment
Stanford	CS144	Introduction to Computer Networking	Fall 2021
CMU	CS 15-744	Computer Networks	Spring 2018
中科大USTC	N/A	计算机网络	N/A	undergraduate stage
中科大USTC	N/A	高级计算机网络	N/A	postgraduate stage

2.3.2 Books

• Reference Course Resources

2.4 Data Structure and Algorithmn

2.4.1 Courses

University	ID	Name	Time	Comment
UC Berkeley	CS 61B	Data Structures	Fall 2020
CMU	CS 15-122	Principles of Imperative Computation	Spring 2023	Past Courses Page 404
CMU	CS 15-121	Introduction Data Structures	Spring 2018

2.4.2 Books

• Reference Course Resources

2.5 Algorithms Design and Analysis

2.5.1 Courses

University	ID	Name	Time	Comment
UC Berkeley	UC Berkeley	Efficient Algorithms and Intractable Problems
MIT	6.006	Introduction to Algorithms	Spring 2020	Introduction to Algorithmns
CMU	CS 15-451/651	Algorithm Design and Analysis	Fall 2021
CMU	CS 15-850	Advanced Algorithms	Fall 2020

2.5.2 Books

• Reference Course Resources

2.6 Database

2.6.1 Course

University	ID	Name	Time	Comment
CMU	CS 15-445/645	Database Systems	Fall 2022	Instructor: Andy Pavlo
UC Berkeley	CS 186	Introduction to Database Systems	Spring 2022
Pennsylvania	CMPSC 431W	Database Management Systems	Fall 2015
Stanford	CS346	Database System Implementation	Spring 2015
uwaterloo	CS 448/648	Database Systems Implementation	Winter 2009
CMU	CS 15-721	Advanced Database Systems	Spring 2020	Instructor: Andy Pavlo
CMU	CS 15-799	Special Topics: Self-Driving Database Management Systems	Spring 2022	Instructor: Andy Pavlo
uwaterloo	CS 856	Distributed data management fundamentals (architectures, data placement, query optimization) Distributed transaction processing, concurrency control, recovery, interoperability	Fall 2002	Only need these two slides, nothing else
N/A	N/A	Let's Build a Simple Database: Writing a sqlite clone from scratch in C		Thanks to cstack

2.6.2 Books

• fundamentals of database systems 7th edition solutions, by Ramez Elmasri, Shamkant B. Navathe

• Database System Concepts Seventh Edition, by Silberschatz, Korth and Sudarshan 中文版：《数据库系统概念》

• Database Management Systems, by Ramakrishnan and Gehrke 中文版：《数据库管理系统原理与设计》

• Database Internals A Deep-Dive into How Distributed Data System Work, by Alex Petrov 中文版：《数据库系统内幕》

• Designing Data-Intensive Applications, by Martin Kleppmann

• Database Design and Implementation, by Edward Sciore

• Database System Implementation， by Hector Garcia-Molina, Jeff Ullman, and Jennifer Widom 中文版：《数据库系统实现》

  • This book has been replaced by Database Systems: The Complete Book

• Transaction Processing Concepts and Techniques by Jim Gray

• Principles of Distributed Database Systems, by M. Tamer Özsu and Patrick Valduriez 中文版：《分布式数据库系统原理》

• nosql distilled a brief guide to the emerging world of polyglot persistence, by Pramod J. Sadalage and Martin Fowler 中文版：《NoSQL精髓》

• High Performance MySQL Third Edition, by Baron Schwartz, Peter Zaitsev, Vadim Tkachenko 中文版：《高性能MySQL》

• MySQL High Availability: Tools for Building Robust Data Centers, by CharlesBell,MatsKindahl,LarsThalmann 中文版：《高可用MySQL:构建健壮的数据中心》

2.6.3 Papers

CS 15-721 Topics Papers

2.7 Distributed System

This Section is for distributed database.

University	ID	Name	Time	Comment
MIT	6.824	MIT Distributed Systems	Spring 2021	Golang
N/A	N/A	Distributed Systems
Columbia University	COMS 4113	Distributed Systems Fundamentals
CMU	CS 15-440	CMU Distributed Systems
Princeton	COS 418	Princeton Distributed Systems	Fall 2019
Columbia University		Advanced Distributed Systems	Research Papers

3. SQL & Relational Algebra

• SQL 知识点：Book《Database System Conceptsm》Chapter3、Chapter4、Chapter5

• Relational Algebra 知识点：Book《Database System Conceptsm》Chapter6

• w3cSchools SQL: https://www.w3schools.com/sql/

• Introduction of Relational Algebra in DBMS

• Relational Algebra in DBMS: Operations with Examples

• slide : Relational Algebra and SQL

• SQL深入理解|关系代数、简单查询、连接

• SQL 与关系代数

4. DDL & DML

• DDL : https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language

• DML : https://cloud.google.com/bigquery/docs/reference/standard-sql/dml-syntax

• Difference Between DDL and DML in DBMS

5. Relational Model

• WikiPedia : Relational model

• Relational Data Model in DBMS | Database Concepts & Example

• What is ER Modeling? Learn with Example

• ER (Entity Relationship) Diagram in DBMS

• Relational Model in DBMS

• 数据库系统原理学习笔记（二）-数据库设计和ER图

• Understanding Relational Databases

• Understanding the Enhanced ER Model Simplified 101

• Data Model Relationships 101: Simplified Guide for Beginners

• 9 ER Model Tools to use in 2023: A Comprehensive List

6. Storage management

CS 15-445 课程 Lecture03、Lecture04

• Managing Database Storage Structures

• Some study on database storage internals

• Storage 101 Series

  • Storage 101: Understanding the Hard-Disk Drive

  • Storage 101: Understanding the NAND Flash Solid State Drive

  • Storage 101: RAID

• RAID

• RAID and Its Impact on your SQL Performance

• Importance of RAID in Databases

6.1 Buffer management

CS 15-445 课程 Lecture05

6.2 Index structure

• wikiPedia : Database index

• Database Indexes Explained

• An in-depth look at Database Indexing

• Database Index: An Introduction for Beginners

• Indexing in DBMS: What is, Types of Indexes with EXAMPLES

6.3 B+ Tree

• wikiPedia : B+ tree

• CS 186 Spring notes pdf : B+ tree

• University of Utah sliede :Database Systems Index: B+ Tree

• implement a B+ Tree index file

• Paper 2004 : Query and Update Efficient B-Tree Based Indexing of Moving Object

6.4 B- Tree

• 课程：《Let's Build a Simple Database》

  • Part 7 - Introduction to the B-Tree
  • Part 8 - B-Tree Leaf Node Format

• B- Tree Datastructure

• B- Trees : Software Design Using C++

• Paper 2010, Efficient B-tree Based Indexing for Cloud Data Processing, VLDB, National University of Singapore & IBM Watson Research Center

6.5 Hash table

• wikipedia : Hash table

• CMU CS15-445 slide : Hash table slide

• GAtech.edu slide : Hash table & Extendible Hash & Linear Hash

• BlogAn Introduction to B-Tree and Hash Indexes in PostgreSQL

• Extendible Hashing (Dynamic approach to DBMS)

• Extendible hashing for COSC 311

6.6 LSM Tree

• What is a LSM Tree?

• Log Structured Merge Trees

• slide : Log Structured Merge Tree, thanks to Pinglei Guo

• Paper : The Log-Structured Merge-Tree (LSM-Tree)

• AlibabaCloud Community : Starting from Zero: Build an LSM Database with 500 Lines of Code

• B-Tree vs Log-Structured Merge-Tree

8. Query Evaluation（Processing）

SQL Query Parser

Query Executor

9. Query Optimization

• Top 10 SQL Query Optimization Tips to Improve Database Performance, by Avishek Singh

Lock manager

10. Transaction management

• 2006, Cost-based query transformation in Oracle, VLDB

11. Network

12. Serialization

13. Concurrency Control

14. Crash Recovery management

15. NoSQL

16. NewSQL

17. Distributed & Paralleled

• Oracle7 Server Concepts Manual : Distributed Databases
• Spanner: Google's Globally-Distributed Database

18. AI4DB and DB4AI (frontier tech)

• University of Magdeburg slides:
  • slide 01 : https://www.dbse.ovgu.de/en/-p-578-EGOTEC-35lsmf8qh52t9v3rhrjdojrd46/_/5_ai-1.pdf
  • slide 02 : https://www.dbse.ovgu.de/-p-578/_/5_ai-2.pdf
• Tsinghua University databaseGroup Github : https://github.com/TsinghuaDatabaseGroup/AIDB
• Tsinghua & MIT Paper : https://dbgroup.cs.tsinghua.edu.cn/ligl/papers/sigmod21-tutorial-paper.pdf
• openGauss Blog : openGauss AI4DB and DB4AI
• MIT databaseGroup : http://dsg.csail.mit.edu/mlforsystems/papers/
• Blogger Github : https://github.com/LumingSun/ML4DB-paper-list

15. OLAP、OLTP、HTAP

15.1 OLAP

OnLine Analytical Processing

• An Overview of Data Warehousing and OLAP Technology

15.2 OLTP

Online Transaction Processing

15.3 HTAP

Hybrid Transactional/Analytical Processing

• A Common Database Approach for OLTP and OLAP Using an In-Memory Column DataBase

• PingCAP : How We Build an HTAP Database That Simplifies Your Data Platform

• StoneDB 文章

  • 什么是真正的HTAP？（一）背景篇
  • 什么是真正的 HTAP ？（二）挑战篇

16. Project Source Code Analysis

ID	Database	DataBase Type	Blog	Github
1	SQLite		SQLite源码分析	https://github.com/sqlite/sqlite
2	LevelDB	fast key-value storage library	LevelDB 源码剖析	https://github.com/google/leveldb
3	PolarDB-X	cloud native distributed SQL Database	PolarDB-X 源码解读	https://github.com/polardb/polardbx-sql
4	OceanBase	distributed relational database		https://github.com/oceanbase/oceanbase
5	TiDB	cloud-native, distributed, MySQL-Compatible database	TiDB源码阅读分析	https://github.com/pingcap/tidb
6	openGauss	open source relational database management system	openGauss数据库源码解析	https://github.com/opengauss-mirror/openGauss-server

taobao MySQL 数据库内核月报：http://mysql.taobao.org/monthly/

17. Mini-Project Labs

• DeepDB : https://github.com/deepbodra97/Database-System-Implementation

• Mini-OB : https://open.oceanbase.com/activities/4921877

• CS 15-445 Labs