Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge the smgr branch back to main #227

Merged
merged 16 commits into from
Jun 26, 2024
Merged

Merge the smgr branch back to main #227

merged 16 commits into from
Jun 26, 2024

Commits on Apr 16, 2024

  1. Temporary build fix with PG17 development branch

    Because the current smgr patchset is based on the PG17 branch, smgr
    development needs to be based on that. But our current copied PG16
    code doesn't compile with PG17.
    
    This commit doesn't try to make the pg_tde AM work with PG17, only
    "comments out" with conditional commands parts that do not compile.
    
    PG16 builds work as before, without any changes.
    
    PG17 at least compiles with missing code, probably doesn't really work.
    dutow committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    cbec8ce View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b447640 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2024

  1. XLog encryption prototype (Percona-Lab#183)

    * Encrypts XLog pages while writing them to the segment file.
    * Add WAL ecryption GUC.
    * Both streaming and logical replication are working.
    * TODO: needs key management.
    dAdAbird committed May 30, 2024
    Configuration menu
    Copy the full SHA
    8973619 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. Implement event triggers to identify index creation on encrypted tabl…

    …es. (Percona-Lab#196)
    
    This commit implements ddl-start and ddl-end event triggers to identify index
    creation operations on encrypted tables. Upon creating an index on an encrypted
    table, the trigger function updates the global state, which can be accessed by
    the storage manager (mgr) to decide if smgr_create needs to do encryption or not.
    
    The start-ddl function analyzes the CREATE TABLE and CREATE INDEX statements
    and identifies if the table uses the pg_tde access method. When the table is
    created or the one on which the index is being created utilizes the
    pg_tde access method, the start-ddl trigger function populates relevant
    information about the encrypted table into a global structure.
    This structure can be accessed using the GetCurrentTdeCreateEvent() function.
    After the execution of the current DDL command finishes, the end-ddl
    function clears out this structure.
    codeforall committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    48d2119 View commit details
    Browse the repository at this point in the history
  2. Minimal working prototype for SMGR + Event trigger + keyring encrypti…

    …on (Percona-Lab#199)
    
    * Introduces `pg_tde2` access method
    * New access method uses the event trigger changes from Percona-Lab#196
    * Keys are now loaded from the keyring
    * This required changes to the map file / master key infrastructure
    * This commit only modifies/fixes those as little as required for simplicity.
      More refactoring/changes coming in separate commits / PRs
    * Removes reliance from MyDatabaseId, as things now have to work with multiple databases in the checkpointer
    * Removes some error reports, where functions should work even without a configured keyring
    * Fixes some bugs in the map file functions
    * Map file functions now work with multiple databases in a single process, but this is a hackish solution, global state needs a proper refactoring
    * Contains anti-recursion hack in the new SMGR code, which is needed until we store the metadata in the catalog
    dutow committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    8935125 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. Test and other fixes for the smgr branch (Percona-Lab#216)

    With this commit, old pg_tde code should work as before in the
    same branch, and CI tests also should pass.
    dutow committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    dfb9b8d View commit details
    Browse the repository at this point in the history
  2. Prepare for XLog keyring

    * Make the *.map *.dat processing code aware of custom databases and
      table spaces
    * Add XLog GUC and init the keyring based on that. Only FS for now
    * Make the internal/external key infrastructure work with custom
      (not stored in the database) keyrings.
    dAdAbird committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    2f52b21 View commit details
    Browse the repository at this point in the history
  3. Create and use global catalog key

    * Check and create an internal key for XLog during the server start.
    If the key is created (not the first start with the EncryptWAL), then
    upload it into the cache. We can't read the key from files while
    writing the XLog to the disk as it happens in the critical section and
    no palloc is allowed.
    
    * Create a custom cache for the global catalog external key as we can't
    use PG's hashmap during the (again, no pallocs in critical section).
    dAdAbird committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    594a582 View commit details
    Browse the repository at this point in the history
  4. Use Linux fd syscalls instead of PG's File Vfd

    During the server start, when pg_tde module is loading and it needs to
    read *.map, *.dat file, InitFileAccess is yet to be called, hence Vfd
    isn't ready to use. The same gonna happen during recovery. So use raw
    pread/pwrite calls istead.
    dAdAbird committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    5b4672a View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    b90bb3a View commit details
    Browse the repository at this point in the history
  6. Add todo for key management

    dAdAbird committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    33e76da View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    3e007b2 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. Fix key rotation

    dAdAbird committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    9302137 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. Configuration menu
    Copy the full SHA
    d4181b6 View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. Merge pull request Percona-Lab#214 from dAdAbird/xlog_key_mgmt

    Add key management for WAL
    
    Make the *.map *.dat processing code aware of custom databases and
    table spaces
    
    Add XLog GUC and init the keyring based on that. Only FS for now
    
    Make the internal/external key infrastructure work with custom
    (not stored in the database) keyrings.
    
    Check and create an internal key for XLog during the server start.
    If the key is created (not the first start with the EncryptWAL), then
    upload it into the cache. We can't read the key from files while
    writing the XLog to the disk as it happens in the critical section and
    no palloc is allowed.
    
    Create a custom cache for the global catalog external key as we can't
    use PG's hashmap during the (again, no pallocs in critical section).
    
    During the server start, when pg_tde module is loading and it needs to
    read *.map, *.dat file, InitFileAccess is yet to be called, hence Vfd
    isn't ready to use. The same gonna happen during recovery. So use raw
    pread/pwrite calls istead.
    dAdAbird committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    a670e46 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4312c48 View commit details
    Browse the repository at this point in the history