tresor: improved module framework and clean-up #5062

m-stein · 2023-11-24T04:56:26Z

Until now the Tresor library kept on suffering from design decisions made years ago with its predecessor, the CBE library. I have identified these main problems:

The code has much unnecessary state and indirections in execution flow because a Tresor module can generate a request to another module only after having waited for the Module framework to ask the remote module whether it is ready for receiving.
All Module request classes must be copyable, which implies several negative effects.
Some modules, like the Free Tree module have such complex state machines and execution flow patterns, that debugging them almost always becomes a forensic, tiresome task. A dedicated internal restructuring of these modules is overdue.
There is a lot of redundancy because much bureaucracy around the communication between modules is not yet part of the Module framework but implemented module-specific.
Further redundancy comes from the fact that some implementations, like securing a superblock are required in many places but are not well extracted because of their asynchronous nature. Such implementations would benefit from a concept for sub-state-machines that is easy to understand when looking at the surrounding state machine.
The back end modules (block_io, crypto, trust_anchor) all work on files but each has its own implementation of this access with only subtle but dangerous differences. These modules would benefit from a generic file abstraction in form of a sub-state machine.
All users of the library (Tresor Tester, VFS Tresor, Tresor Init, Tresor Check) implement kind-of modules and individually glue them to the Tresor modules. Tresor users should use only generic Module utilities for Module-related tasks.

* The command pool used to be kind of a module but it was driven via custom tresor-tester specific code. With this commit it becomes a proper module that is driven by the module framework instead. * Move the code for creating and handling the module-execution progress flag into Module_composition::execute_modules as the function is always used with this code surrounding it. Ref genodelabs#5062

Ref genodelabs#5062

* A new class Module_channel is introduced in the module framework and all channel classes inherit from it. With that class in place, the formerly module-specific implementations of the following methods are replaced by new generic implementations in the Module framework: * ready_to_submit_request * submit_request * _peek_completed_request * _drop_completed_request * _peek_generated_request * _drop_generated_request * generated_request_complete * Module requests are now held for the duration of their lifetime at the module they originate from and not, like before, at their target module. As a result, modules can generate new requests inline (without having to wait for the target module), making code much simpler to read, reducing the amount of channel state, and allowing for non-copyable request types. * Introduce a sub-state-machine for securing a superblock in the superblock_control module in order to reduce redundancy. * Some modules, like free_tree, were completely re-designed in order to make them more readable. * Replace all conditional exceptions by using the macros in tresor/assertion.h . * Move methods that are used in multiple modules but that were implemented redundantly in each module to tresor/types.h. * Remove verbosity node and all that was related to it from tresor tester config as the targeted verbosity can be achieved with the VERBOSE_MODULE_COMMUNICATION flag in tresor/verbosity.h . * Extract the aspect of translating the byte-granular I/O-requests to tresor-block requests from the tresor VFS-plugin and move it to a new module called splitter. * Rename the files and interface of the hashing back-end to not reflect the used hashing algorithm/config anymore, while at the same time making the hashing interface strict regarding the used types. * Introduce the NONCOPYABLE macro that makes marking a class noncopyable short and clear. * Replace the former tresor/vfs_utilities.h/.cc with a new tresor/file.h that contains the classes Read_write_file and Write_only_file. These classes significantly simplify the modules crypto, block_io, and trust_anchor by moving the details of file access to a sub-state machine. * The former, rather trivial block allocator module is replaced by a normal object of type Pba_allocator that must be provided by the client of the Sb_initializer (reference in the Sb_initializer_request). Ref genodelabs#5062

Virtual addresses in a Tresor container that were not yet written by the user should always return a data block that is all-zeroes. This was the concept right from the beginning of the project. However, somehow this aspect either never got implement or got lost along the way. Some context for understanding the commit: The Tresor doesn't initialize the payload data blocks of a container when creating a new container as this would be rather expensive. Instead, it marks the leaf metadata nodes of the virtual-block-device tree (those that reference the payload data blocks in physical address space) with generation 0. Now, this commit ensures that, whenever the virtual-block-device module reads such a generation-0 leaf, instead of asking the block_io and crypto to deliver data from disc, it directly provides the user with 4K of zeroes. Ref genodelabs#5062

The tresor_check tool became outdated back when the Tresor project was created by re-writing its predecessor, the CBE, in C++. At this time, the check tool was merely renamed but not updated. As there was also no autopilot test for the tool, the tool remained outdated. This commit rewrites the tool for the most recent Tresor version and adds an autopilot test. Ref genodelabs#5062

* The command pool used to be kind of a module but it was driven via custom tresor-tester specific code. With this commit it becomes a proper module that is driven by the module framework instead. * Move the code for creating and handling the module-execution progress flag into Module_composition::execute_modules as the function is always used with this code surrounding it. Ref genodelabs#5062

Ref genodelabs#5062

* A new class Module_channel is introduced in the module framework and all channel classes inherit from it. With that class in place, the formerly module-specific implementations of the following methods are replaced by new generic implementations in the Module framework: * ready_to_submit_request * submit_request * _peek_completed_request * _drop_completed_request * _peek_generated_request * _drop_generated_request * generated_request_complete * Module requests are now held for the duration of their lifetime at the module they originate from and not, like before, at their target module. As a result, modules can generate new requests inline (without having to wait for the target module), making code much simpler to read, reducing the amount of channel state, and allowing for non-copyable request types. * Introduce a sub-state-machine for securing a superblock in the superblock_control module in order to reduce redundancy. * Some modules, like free_tree, were completely re-designed in order to make them more readable. * Replace all conditional exceptions by using the macros in tresor/assertion.h . * Move methods that are used in multiple modules but that were implemented redundantly in each module to tresor/types.h. * Remove verbosity node and all that was related to it from tresor tester config as the targeted verbosity can be achieved with the VERBOSE_MODULE_COMMUNICATION flag in tresor/verbosity.h . * Extract the aspect of translating the byte-granular I/O-requests to tresor-block requests from the tresor VFS-plugin and move it to a new module called splitter. * Rename the files and interface of the hashing back-end to not reflect the used hashing algorithm/config anymore, while at the same time making the hashing interface strict regarding the used types. * Introduce the NONCOPYABLE macro that makes marking a class noncopyable short and clear. * Replace the former tresor/vfs_utilities.h/.cc with a new tresor/file.h that contains the classes Read_write_file and Write_only_file. These classes significantly simplify the modules crypto, block_io, and trust_anchor by moving the details of file access to a sub-state machine. * The former, rather trivial block allocator module is replaced by a normal object of type Pba_allocator that must be provided by the client of the Sb_initializer (reference in the Sb_initializer_request). Ref genodelabs#5062

Virtual addresses in a Tresor container that were not yet written by the user should always return a data block that is all-zeroes. This was the concept right from the beginning of the project. However, somehow this aspect either never got implement or got lost along the way. Some context for understanding the commit: The Tresor doesn't initialize the payload data blocks of a container when creating a new container as this would be rather expensive. Instead, it marks the leaf metadata nodes of the virtual-block-device tree (those that reference the payload data blocks in physical address space) with generation 0. Now, this commit ensures that, whenever the virtual-block-device module reads such a generation-0 leaf, instead of asking the block_io and crypto to deliver data from disc, it directly provides the user with 4K of zeroes. Ref genodelabs#5062

The tresor_check tool became outdated back when the Tresor project was created by re-writing its predecessor, the CBE, in C++. At this time, the check tool was merely renamed but not updated. As there was also no autopilot test for the tool, the tool remained outdated. This commit rewrites the tool for the most recent Tresor version and adds an autopilot test. Ref genodelabs#5062

The request classes Block_io::Read_client_data and Block_io::Write_client_data used to receive a block reference for no reason. This commit removes these args. Ref genodelabs#5062

During one of the many re-factorization steps that were applied to the Tresor library and its predecessor, the CBE library, one of the main features of the project, the integrity check, accidentally received a grave regression. The most recent version of the Tresor still used to check all hashes of meta-data blocks but ignored the hashes of the actual data blocks. With this commit, the hashes of all data-block get checked. Note, I have included also the hashes of yet uninitialized data blocks although they were, up until now, always ignored on purpose. The reason for ignoring uninitialized blocks was that they are not actually read from disc but simply generated as an all-zeros block in the driver in order to prevent having to initialize them all to zero in Tresor-Init. Therefore, the integrity of these blocks cannot be compomised and the stored hashes of these blocks are guarded by the above hash tree. I decided to check these hashes anyway for two reasons. First, it simplifies the code and makes it easier to verify that integrity is indeed preserved. And second, not checking the hashes would allow for Tresor containers that are broken in a certain way to be still accepted by the driver, thereby potentially covering up regressions in the Tresor tooling. This commit also adapts the Tresor initializer and check modules to the new behavior regarding hashes of uninitialized data blocks. Ref genodelabs#5062

During one of the many re-factorization steps that were applied to the Tresor library and its predecessor, the CBE library, one of the main features of the project, the integrity check, accidentally received a grave regression. The most recent version of the Tresor still used to check all hashes of meta-data blocks but ignored the hashes of the actual data blocks. With this commit, the hashes of all but yet uninitialized data blocks get checked. The reason for ignoring uninitialized blocks is that they are not actually read from disc but simply generated as an all-zeros block in the driver in order to prevent having to initialize them all to zero in Tresor-Init. That said, the integrity of these blocks cannot be compomised. The according hashes in the meta data remain unset until the data block gets written for the first time. Ref genodelabs#5062

* The command pool used to be kind of a module but it was driven via custom tresor-tester specific code. With this commit it becomes a proper module that is driven by the module framework instead. * Move the code for creating and handling the module-execution progress flag into Module_composition::execute_modules as the function is always used with this code surrounding it. Ref genodelabs#5062

Ref genodelabs#5062

* A new class Module_channel is introduced in the module framework and all channel classes inherit from it. With that class in place, the formerly module-specific implementations of the following methods are replaced by new generic implementations in the Module framework: * ready_to_submit_request * submit_request * _peek_completed_request * _drop_completed_request * _peek_generated_request * _drop_generated_request * generated_request_complete * Module requests are now held for the duration of their lifetime at the module they originate from and not, like before, at their target module. As a result, modules can generate new requests inline (without having to wait for the target module), making code much simpler to read, reducing the amount of channel state, and allowing for non-copyable request types. * Introduce a sub-state-machine for securing a superblock in the superblock_control module in order to reduce redundancy. * Some modules, like free_tree, were completely re-designed in order to make them more readable. * Replace all conditional exceptions by using the macros in tresor/assertion.h . * Move methods that are used in multiple modules but that were implemented redundantly in each module to tresor/types.h. * Remove verbosity node and all that was related to it from tresor tester config as the targeted verbosity can be achieved with the VERBOSE_MODULE_COMMUNICATION flag in tresor/verbosity.h . * Extract the aspect of translating the byte-granular I/O-requests to tresor-block requests from the tresor VFS-plugin and move it to a new module called splitter. * Rename the files and interface of the hashing back-end to not reflect the used hashing algorithm/config anymore, while at the same time making the hashing interface strict regarding the used types. * Introduce the NONCOPYABLE macro that makes marking a class noncopyable short and clear. * Replace the former tresor/vfs_utilities.h/.cc with a new tresor/file.h that contains the classes Read_write_file and Write_only_file. These classes significantly simplify the modules crypto, block_io, and trust_anchor by moving the details of file access to a sub-state machine. * The former, rather trivial block allocator module is replaced by a normal object of type Pba_allocator that must be provided by the client of the Sb_initializer (reference in the Sb_initializer_request). Ref genodelabs#5062

Virtual addresses in a Tresor container that were not yet written by the user should always return a data block that is all-zeroes. This was the concept right from the beginning of the project. However, somehow this aspect either never got implement or got lost along the way. Some context for understanding the commit: The Tresor doesn't initialize the payload data blocks of a container when creating a new container as this would be rather expensive. Instead, it marks the leaf metadata nodes of the virtual-block-device tree (those that reference the payload data blocks in physical address space) with generation 0. Now, this commit ensures that, whenever the virtual-block-device module reads such a generation-0 leaf, instead of asking the block_io and crypto to deliver data from disc, it directly provides the user with 4K of zeroes. Ref genodelabs#5062

The tresor_check tool became outdated back when the Tresor project was created by re-writing its predecessor, the CBE, in C++. At this time, the check tool was merely renamed but not updated. As there was also no autopilot test for the tool, the tool remained outdated. This commit rewrites the tool for the most recent Tresor version and adds an autopilot test. Ref genodelabs#5062

…LOCK_IO) Ref genodelabs#5062

…t type) Ref genodelabs#5062

…selection) Ref genodelabs#5062

The request classes Block_io::Read_client_data and Block_io::Write_client_data used to receive a block reference for no reason. This commit removes these args. Ref genodelabs#5062

During one of the many re-factorization steps that were applied to the Tresor library and its predecessor, the CBE library, one of the main features of the project, the integrity check, accidentally received a grave regression. The most recent version of the Tresor still used to check all hashes of meta-data blocks but ignored the hashes of the actual data blocks. With this commit, the hashes of all but yet uninitialized data blocks get checked. The reason for ignoring uninitialized blocks is that they are not actually read from disc but simply generated as an all-zeros block in the driver in order to prevent having to initialize them all to zero in Tresor-Init. That said, the integrity of these blocks cannot be compomised. The according hashes in the meta data remain unset until the data block gets written for the first time. Ref genodelabs#5062

Ref genodelabs#5062

The script tests the use of an encrypted file system that is created and provided via the File Vault. Furthermore the script can be used for test-driving existing File-Vault containers (created with potentially older File-Vault versions) under the current File-Vault version. This is done via the "LX_FS_DIR_TEMPLATE" env variable. Ref genodelabs#5062

Virtual addresses in a Tresor container that were not yet written by the user should always return a data block that is all-zeroes. This was the concept right from the beginning of the project. However, somehow this aspect either never got implement or got lost along the way. Some context for understanding the commit: The Tresor doesn't initialize the payload data blocks of a container when creating a new container as this would be rather expensive. Instead, it marks the leaf metadata nodes of the virtual-block-device tree (those that reference the payload data blocks in physical address space) with generation 0. Now, this commit ensures that, whenever the virtual-block-device module reads such a generation-0 leaf, instead of asking the block_io and crypto to deliver data from disc, it directly provides the user with 4K of zeroes. Ref genodelabs#5062

The tresor_check tool became outdated back when the Tresor project was created by re-writing its predecessor, the CBE, in C++. At this time, the check tool was merely renamed but not updated. As there was also no autopilot test for the tool, the tool remained outdated. This commit rewrites the tool for the most recent Tresor version and adds an autopilot test. Ref genodelabs#5062

During one of the many re-factorization steps that were applied to the Tresor library and its predecessor, the CBE library, one of the main features of the project, the integrity check, accidentally received a grave regression. The most recent version of the Tresor still used to check all hashes of meta-data blocks but ignored the hashes of the actual data blocks. With this commit, the hashes of all but yet uninitialized data blocks get checked. The reason for ignoring uninitialized blocks is that they are not actually read from disc but simply generated as an all-zeros block in the driver in order to prevent having to initialize them all to zero in Tresor-Init. That said, the integrity of these blocks cannot be compomised. The according hashes in the meta data remain unset until the data block gets written for the first time. Ref genodelabs#5062

The script tests the use of an encrypted file system that is created and provided via the File Vault. Furthermore the script can be used for test-driving existing File-Vault containers (created with potentially older File-Vault versions) under the current File-Vault version. This is done via the "LX_FS_DIR_TEMPLATE" env variable. Ref genodelabs#5062

* Make command pool a proper module * The command pool used to be kind of a module but it was driven via custom tresor-tester specific code. Now, it becomes a proper module that is driven by the module framework instead. * Move the code for creating and handling the module-execution progress flag into Module_composition::execute_modules as the function is always used with this code surrounding it. * Reorganize files, remove deprecated files * A new class Module_channel is introduced in the module framework and all channel classes inherit from it. With that class in place, the formerly module-specific implementations of the following methods are replaced by new generic implementations in the Module framework: * ready_to_submit_request * submit_request * _peek_completed_request * _drop_completed_request * _peek_generated_request * _drop_generated_request * generated_request_complete * Module requests are now held for the duration of their lifetime at the module they originate from and not, like before, at their target module. As a result, modules can generate new requests inline (without having to wait for the target module), making code much simpler to read, reducing the amount of channel state, and allowing for non-copyable request types. * Introduce a sub-state-machine for securing a superblock in the superblock_control module in order to reduce redundancy. * Some modules, like free_tree, were completely re-designed in order to make them more readable. * Replace all conditional exceptions by using the macros in tresor/assertion.h . * Move methods that are used in multiple modules but that were implemented redundantly in each module to tresor/types.h. * Remove verbosity node and all that was related to it from tresor tester config as the targeted verbosity can be achieved with the VERBOSE_MODULE_COMMUNICATION flag in tresor/verbosity.h . * Extract the aspect of translating the byte-granular I/O-requests to tresor-block requests from the tresor VFS-plugin and move it to a new module called splitter. * Rename the files and interface of the hashing back-end to not reflect the used hashing algorithm/config anymore, while at the same time making the hashing interface strict regarding the used types. * Introduce the NONCOPYABLE macro that makes marking a class noncopyable short and clear. * Replace the former tresor/vfs_utilities.h/.cc with a new tresor/file.h that contains the classes Read_write_file and Write_only_file. These classes significantly simplify the modules crypto, block_io, and trust_anchor by moving the details of file access to a sub-state machine. * The former, rather trivial block allocator module is replaced by a normal object of type Pba_allocator that must be provided by the client of the Sb_initializer (reference in the Sb_initializer_request). Ref genodelabs#5062 tresor: read uninitialized vbas as all zeroes Virtual addresses in a Tresor container that were not yet written by the user should always return a data block that is all-zeroes. This was the concept right from the beginning of the project. However, somehow this aspect either never got implement or got lost along the way. Some context for understanding the commit: The Tresor doesn't initialize the payload data blocks of a container when creating a new container as this would be rather expensive. Instead, it marks the leaf metadata nodes of the virtual-block-device tree (those that reference the payload data blocks in physical address space) with generation 0. Now, this commit ensures that, whenever the virtual-block-device module reads such a generation-0 leaf, instead of asking the block_io and crypto to deliver data from disc, it directly provides the user with 4K of zeroes. Ref genodelabs#5062