GSoC 2024: WAL Infrastructure

[sh-soltanpour]

Intro

Hi everyone,

My name is Shahryar (pronounced as shah-ree-yar), and I'm selected as a contributor for GSoC 2024 in the PostgreSQL organization and Pgmoneta - WAL Infrastructure project.

In this project, we want to implement an infrastructure in Pgmoneta that can read and parse the WAL files into WAL records. When our infrastructure has a deep understanding of the WAL files and records, we can apply the appropriate WAL records on a base backup that we have, and we can reach the desired point in the restored database, which is a very helpful feature to achieve point-in-time-recovery (PiTR). As a bonus goal, we can even extend this understanding from a physical to a logical understanding.

By the end of GSoC 2024, I’m aiming to achieve these items:
Milestone 1: Implement an in-house infrastructure for parsing WAL segment files to WAL records.
Milestone 2: Being able to restore the database using a base backup and applying a series of WAL records to get to the required point.
(Bonus) Milestone 3: Extract logical information from WAL records and implement a policy definition framework so developers can define some custom logic for their backup database and filter some tables, rows, etc.

For more details, feel free to check out my initial proposal for this project here.

I would be more than happy to receive any suggestions or ideas here or by sending them to my email: shahryar.soltanpour@gmail.com

I will update the sections below during my progress in this project.

List of the work done

My main work branch for the whole WAL parsing functionality is: parse-wal-records.

• Implemented Implement Extraction of WAL Segment General Portion Headers #299 and merged it to the parse-wal-records branch.

Progress Log

23/June/2024

• This week I was implementing the rm_desc function for different resource managers, until now, I have implemented it for the following list:

1. XLOG
2. Transaction
3. Heap
4. Heap2
5. Btree
6. Gin
7. Standby

• There are 22 resource managers in total, implementing it for the rest of them doesn't need a lot of creativity and it's mostly about trying to do it in a clean way to make the next step (refactoring) easier.
• Did some minor refactors where needed for the newly added functions.

16/June/2024

1. Implemented the rm_desc for Btree records following Standby.
2. Implement retrieving the data portion of records that are split between two pages and load them into the buffer.
3. After implementing the first item, I noticed a bug in the parsing of the data portion of some Heap and Btree records, I spent most of my time this week figuring out the issue and fixing it, and was successful at the end :). The problem is that some of the flags such as BKPIMAGE_IS_COMPRESSED are used in Postgres 14 but are changed in the newer versions, I was parsing a record generated by Postgres 14 with the flags from the latest version in here.
4. Implemented the description for the BkpBlock functionality and compared it to the actual pg_waldump and verified that it's working correctly.
5. Pushed an up-to-date, squashed, and rebased version of the code into parse-wal-records-squashed branch here.

09/June/2024

1. This week was a good one in terms of my progress. I was able to fix the issues I had last week. For the offset calculation problem, I found that I needed to use the MAXALIGN function, and this function aligned the mentioned index from 90 to 96. Afterward, the next records were being parsed correctly.
2. After this, I was able to iterate the WAL segment record by record and print each record's header, I was able to also skip the parts that a record was separated between two pages and handled these cases correctly.
3. Then, I started decoding each WAL record, I mostly re-implemented the logic used in the DecodeXLogRecord function in a simplified and lighter version in Pgmoneta.
4. I verified the correctness of the result by comparing the detected resource manager with the expected one by comparing my output against pg_waldump, I also compared and verified the record total length.
5. Implemented an initial version of the rm_desc function for the Standby resource manager, I want to use this to make sure that the data is also parsed correctly, This was successful for the Standby resource manager, and for the next week, I'm going to implement it for other resource managers as well. Note that the code still needs a lot of cleaning, I'm planning to refactor it at some point before merging and after where we have the basic expected functionality.

02/June/2024

1. Continuing the work from the previous week, I was able to parse the first XLogRecord. but I can't use the xl_tot_len field to jump over the first record and get the second XLogRecord. For instance, I'm at offset 40 of the first page and I read the first XLogRecord, and the xl_tot_len is 50, so I expect the next record to start at offset 90 = 40 + 50, but the parsed value in that offset is not correct, I also tried various ways such as 40 + 50 - 24 or 40 + 50 + 24 (24 is the header size), or even and removing 2 padding bytes, but none of them worked.
2. So I started reading the code of XLogFindNextRecord, ReadInternalPage, and other dependent functions in the pg_waldump source code. The difficult part of this process is figuring out what is useful in our case and what should be skipped, as well as navigating through a lot of nested functions.
3. If someone knows about the first item and what's wrong with my calculations, or has useful resources on the second item that make it easier to understand the code, I would be more than happy if you could share them with me.

26/May/2024

1. Follow parse-wal-records for my most up-to-date progress in implementing the WAL parser.
2. Implemented the basic mechanism to go page by page and parse the header of each page. (XLogLongPageHeaderData, and XLogPageHeaderData). Verified the result by comparing the xlp_magic value against the expected value.
3. Started the implementation of the XLogRecord fixed header of each record, until now, I was able to parse the fixed header of the first record and figured out how to get the resource manager data from the parsed header.
4. I haven't committed the code of step 3 yet as I'm struggling with how to skip the rest of the first record and get the fixed header of the second one.

20/May/2024
Find the longer version here, in summary:

1. Learn more about how WAL segments and records are conducted and named.
2. Read through the pg_waldump source code.
3. Try to implement a WAL segment parser from scratch, it was not really successful :)
4. Read through the source code of pg_rewind and watch some videos on how it can be used in production.
5. I talked to @jesperpedersen in the thread mentioned and decided to go with record extraction from WAL segments. Created a parent issue for the whole milestone Implement the ability to extract WAL records from WAL files #296 and broke it into three child issues Implement Extraction of WAL Segment General Portion Headers #299, Implement extraction of the header part of the data portion in WAL segments #300, and Implement extraction of the data part of data portion in WAL segments #301.

17/May/2024

1- I have been able to run Pgmoneta on my Mac (Kudos to @GuChad369 for this PR, it made my life easier a lot so I don't need to run a Fedora OS virtually).
2- As @jesperpedersen suggested, I have enabled compression and encryption to make sure these scenarios are covered.
3- Have gone through the wal.c file in the project, to get some idea about how the WAL file is streamed.

For the next step, I want to parse the content of a WAL file streamed into Pgmoneta and load it to WAL records (issue: #296).

In general, I think I have a roadmap in my mind about what issues need to be created for the project to be done, but I need to organize my thoughts and ideas until May 20th, then I can create the list of issues here in GitHub so the mentors can comment on it and we may also be able to use the community help to do some of them.

[jesperpedersen]

Great !

Try and do a setup where both compression and encryption are enabled such that you can make sure that both scenarios are covered when reading the WAL files.

Welcome to GSoC 2024 !

[sh-soltanpour]

Hi @jesperpedersen,
During the last two days, I've been working on how to read WAL files and parse them. I started with pg_waldump. At first, I felt the code was too complicated for our use case and decided to write my own parser from scratch. But after understanding the cases such as variable headers and data types and sizes, and continuous data over multiple pages, I feel that the code is getting more and more similar to what already exists in pg_waldump here.

I know that pg_waldump is using many shared resources from PostgreSQL and we need to re-implement them in our codebase if we aim to keep Pgmoneta independent.

On the other hand, I started reading the pg_rewind code as well. As far as I figured out, it tries to find a common ancestor between two databases using the information in the pg_control file and then tries to apply the source WAL files from the forked point to the target. If I understood correctly, pg_rewind does not need to parse all of the WAL segments to records and does its job on a segment level.

Based on this, I see two areas that I should choose to put my focus on first:
1- If we want PiTR, we can proceed with something similar to what pg_rewind does for now. Then, we can implement segment parsing later when we want to extract logical information.
2- Adding the complete parsing ability to Pgmoneta so it will be able to parse WAL records from segments, and then go for the implementation of PiTR.

In general, I guess the first approach is more like a low-hanging fruit compared to the second one.

Would be happy to know your opinion and corrections where I'm wrong, so we can set the goals and create the issues needed based on the chosen path.

Thanks!

[jesperpedersen]

We need to understand the full WAL format without having dependencies on PostgreSQL code, and its data structures.

But, we need to do it in small steps and improve over time - maybe that means opening the file parse the header, skip the body and go from there.

As we discussed, this is a hard project - so we need to figure out how to best tackle it.

The end-goal is the per WAL record apply criteria based on the given input, so we need to know each WAL record

[sh-soltanpour]

Sounds good thank you!
So, I think I can break this issue into these issues:
1- Implement extraction of WAL segment general portion headers
2- Implement extraction of the header part of the data portion
3- Implement extraction of the data part of data portion (This may need to be separated into two issues for block data and main data as well)

Please let me know if these issues make sense to you and if you have any suggestions, then I will add them.
This should cover the first milestone of the project which is parsing and understanding WAL.

I'm thinking about creating the issues for the next milestone (which is iterating over the WAL records and selectively applying them) when we get there and have more info on the implementation. What do you think?

source

[jesperpedersen]

Sounds like a good plan.

We can start to merge patches once there is something functional complete that we need for a foundation for a later patch.

I think it is very important for this work to have an active work branch registered in the discussion, or as a WIP pull request if close

[sh-soltanpour]

Hi @jesperpedersen,
I have implemented the codebase that is able to parse the page headers and verified the correctness using the xlp_magic value.
I've pushed it to my forked repo in the parse-wal-page-headers branch.

I wanted to create a main branch for the whole WAL parsing functionality, so finally it will be merged to main, but I don't have access to create a new branch on the original repo. So I think you can create the main working branch named parse-wal-records and I will make small patches and PRs to it?

[jesperpedersen]

The important is to have documented work branches for your work.

They will be in your repository, but documented in this discussion.

Keep your work off main since that will be updated frequently, and you want to rebase against it as often as possible.

Breaking down the work into smaller pieces is always good - we can merge functionality as soon as it is complete and make sense to be included in the official release.

I'm expecting that the initial patch will be fairly large, and that is why a documented work branch is important to have. If you go too much down the wrong path, then it will take a lot of time to change...

You can update your work log when you even pushed something like an important bug fix

[jesperpedersen]

I think you are making good progress !

Remember that you have to support the version range so,

• PostgreSQL 13: 0xD106
• PostgreSQL 14: 0xD10D
• PostgreSQL 15: 0xD110
• PostgreSQL 16: 0xD113
• PostgreSQL 17: 0xD115

Also, it would be good to have a work branch where your commits are squashed, and rebased against main - this will make it easier to test. See https://github.com/pgmoneta/pgmoneta/blob/main/doc/DEVELOPERS.md

Use license headers, and uncrustify to get your coding style closer to the official one.

[sh-soltanpour]

I have pushed a squashed version of the code here, will keep it rebased with the main branch and merge and squash changes into it weekly. Also applied uncrustify and license headers.

For this week, I'm going to focus on implementing rm_desc for different resource managers and debug them if needed, to make sure the data is parsed correctly,

Then I start refactoring the code, testing it with different versions, and updating the incompatible parts to work based on the Postgres version.

[jesperpedersen]

Sounds like a good plan !

[jesperpedersen]

@GuChad369 is working a test suite setup that will help us - follow @ pgmoneta/pgmoneta_ext#11

[jesperpedersen]

@sh-soltanpour How is it going ?

[sh-soltanpour]

Hey Jesper!
just updated the progress report for this week.

In summary, worked on adding the rm_desc function for different resource managers. There are 22 in total, so I’ve still got some to go, the process is kinda straightforward but time-consuming as there are lots of resource managers, and just need to make sure to keep things clean to make the upcoming refactoring easier.