Merge the smgr branch back to main #227
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Because the current smgr patchset is based on the PG17 branch, smgr development needs to be based on that. But our current copied PG16 code doesn't compile with PG17. This commit doesn't try to make the pg_tde AM work with PG17, only "comments out" with conditional commands parts that do not compile. PG16 builds work as before, without any changes. PG17 at least compiles with missing code, probably doesn't really work.
* Encrypts XLog pages while writing them to the segment file. * Add WAL ecryption GUC. * Both streaming and logical replication are working. * TODO: needs key management.
…es. (Percona-Lab#196) This commit implements ddl-start and ddl-end event triggers to identify index creation operations on encrypted tables. Upon creating an index on an encrypted table, the trigger function updates the global state, which can be accessed by the storage manager (mgr) to decide if smgr_create needs to do encryption or not. The start-ddl function analyzes the CREATE TABLE and CREATE INDEX statements and identifies if the table uses the pg_tde access method. When the table is created or the one on which the index is being created utilizes the pg_tde access method, the start-ddl trigger function populates relevant information about the encrypted table into a global structure. This structure can be accessed using the GetCurrentTdeCreateEvent() function. After the execution of the current DDL command finishes, the end-ddl function clears out this structure.
…on (Percona-Lab#199) * Introduces `pg_tde2` access method * New access method uses the event trigger changes from Percona-Lab#196 * Keys are now loaded from the keyring * This required changes to the map file / master key infrastructure * This commit only modifies/fixes those as little as required for simplicity. More refactoring/changes coming in separate commits / PRs * Removes reliance from MyDatabaseId, as things now have to work with multiple databases in the checkpointer * Removes some error reports, where functions should work even without a configured keyring * Fixes some bugs in the map file functions * Map file functions now work with multiple databases in a single process, but this is a hackish solution, global state needs a proper refactoring * Contains anti-recursion hack in the new SMGR code, which is needed until we store the metadata in the catalog
With this commit, old pg_tde code should work as before in the same branch, and CI tests also should pass.
* Make the *.map *.dat processing code aware of custom databases and table spaces * Add XLog GUC and init the keyring based on that. Only FS for now * Make the internal/external key infrastructure work with custom (not stored in the database) keyrings.
* Check and create an internal key for XLog during the server start. If the key is created (not the first start with the EncryptWAL), then upload it into the cache. We can't read the key from files while writing the XLog to the disk as it happens in the critical section and no palloc is allowed. * Create a custom cache for the global catalog external key as we can't use PG's hashmap during the (again, no pallocs in critical section).
During the server start, when pg_tde module is loading and it needs to read *.map, *.dat file, InitFileAccess is yet to be called, hence Vfd isn't ready to use. The same gonna happen during recovery. So use raw pread/pwrite calls istead.
Add key management for WAL Make the *.map *.dat processing code aware of custom databases and table spaces Add XLog GUC and init the keyring based on that. Only FS for now Make the internal/external key infrastructure work with custom (not stored in the database) keyrings. Check and create an internal key for XLog during the server start. If the key is created (not the first start with the EncryptWAL), then upload it into the cache. We can't read the key from files while writing the XLog to the disk as it happens in the critical section and no palloc is allowed. Create a custom cache for the global catalog external key as we can't use PG's hashmap during the (again, no pallocs in critical section). During the server start, when pg_tde module is loading and it needs to read *.map, *.dat file, InitFileAccess is yet to be called, hence Vfd isn't ready to use. The same gonna happen during recovery. So use raw pread/pwrite calls istead.
dAdAbird
approved these changes
Jun 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.