From 8ded537a3862439da53696ec078311ee7539bca8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Imobach=20Gonz=C3=A1lez=20Sosa?= Date: Mon, 18 Mar 2019 12:43:19 +0000 Subject: [PATCH] Update AutoYaST related documentation --- doc/autoyast.md | 360 +++++++++++++++++++++--------------------------- 1 file changed, 154 insertions(+), 206 deletions(-) diff --git a/doc/autoyast.md b/doc/autoyast.md index 275a45b129..d6dd10abc4 100644 --- a/doc/autoyast.md +++ b/doc/autoyast.md @@ -1,141 +1,101 @@ -# Implementing AutoYaST partitioning in Storage-NG - +# AutoYaST Support in storage-ng ## About this document -This is a follow-up on [this -document](https://mailman.suse.de/mlarch/SuSE/yast-internal/2017/yast-internal.2017.04/msg00124.html) -sent to the yast-internal mailing list (only accessible within the SUSE -company network). That document categorizes the AutoYaST partitioning use cases -in three levels and states the first two are already close to be covered by just -reusing the existing installation proposal, almost as-is. - -This document describes how some components of that installation proposal could -also be reused to implement many of the features of the third level. - - -## TL;DR - -After finishing an ongoing refactoring of the proposal code, the following -classes could be reused (with small modifications/enhancements) to implement -the so-called level 3 of AutoYaST: `PlannedPartition`, `PlannedLv`, `PlannedVg`, -`PartitionKiller`, `PartitionCreator`, `LvmCreator` and -`SpaceDistributionCalculator`. - -In addition, two similar but new classes should need to be implemented: -`PlannedMd` and `MdCreator`. - -Reusing an existing mount table (fstab) is a different beast. Hard to fit into -the current model and maybe worth dropping. - - -## What we already have +This document describes which components are involved and how they cooperate in +order to create a partitioning proposal for AutoYaST. -The current installation proposal consist basically of two phases: +## Use Cases -* First a list of `PlannedVolume` objects is generated, describing all the - partitions or LVs that are needed to install the system based on the - ProposalSettings. -* Afterwards, a new devicegraph is generated based on the initial one and on - the list of planned volumes. During that process, preexisting partitions and - logical volumes are deleted and resized on demand and the optimal distribution - of partitions is calculated. +When it comes to partitioning, AutoYaST use cases can be categorized in three +different levels: -Each `PlannedVolume` contains both a desired size and a minimal size. So, in the -worst case the whole process is repeated twice (first using the desired sizes -and then falling back to the min). +1. Automatic partitioning. The user does not care about the partitioning and + trusts in AutoYaST to do the right thing. +2. Semi-automatic partitioning. The user would like to set basic some + settings. E.g., a user would like to use LVM but has no idea how the devices + are partitions have to be set. +3. Expert partitioning. The user specifies how the layout should look like. + Anyway, a complete definition is not required and some values could be missing + (AutoYaST should propose a reasonable default). -There is a [pending pull request](https://github.com/yast/yast-storage-ng/pull/179) -(Iván's Hackweek) with a refactoring of the current proposal in which, at the -beginning of every one of the two attempts, the list of planned volumes is -separated into two more specific and explicit lists of `PlannedPartition` and -`PlannedLv` objects. Those objects contain only one target size (so no desired vs -min). The original goal of the PR was to improve readability of many components -but, in addition to that, we found it would also help to implement AutoYaST -specific logic in the proper way. +## Profile Content -The rest of this document assumes the mentioned pull request is finished and -merged and, thus, the `PlannedPartition` and `PlannedLv` classes are there, in -addition to the existing and more generic `PlannedVolume` (and -`PlannedSubvolume`). Of course, the name of some classes referenced on this -document can change (for the best) as part of the mentioned pull request. +An AutoYaST profile can define two different kind of storage settings: -For more information about the internal structure of the installation proposal, -check [proposal.md](./proposal.md). +* [General settings][1], which basically allows to alter the `partitioning` section of the + product features, although only two options are documented (and thus supported). +* [Partitioning layout][2], which defines how the partitioning should look like (partitions, volume + groups, etc.). +Translating this to use cases levels: -## Level 3 of AutoYaST and the installation proposal +* Level 1: no settings at all. +* Level 2: just _general settings_. +* Level 3: _partitioning layout settings_ and, optionally, _general settings_. -[This gist](https://gist.github.com/imobachgs/0f4049c2955b858c0713896210306aa1) -presents many use cases of the so-called level 3 of AutoYaST partitioning, based -on examples found in the AutoYaST documentation. +## Process Overview -The process followed by AutoYaST is quite different from the proposal one, but -the concept of `PlannedPartition`, `PlannedLv` and several auxiliary classes -that take list of those as input to perform operations can be reused to a big -extent. +The AutoYaST partitioning process starts in the [storage_auto client][3], that +lives in the `autoyast2-installation` package. This client relies heavily on the +[AutoinstStorage module][4] which is responsible, among other things, of +importing the AutoYaST profile (via the `#Import` method) and performing the +partitioning (`#Write`). +This document focuses in the `#Import` method, which creates a partitioning +proposal and provides libstorage-ng with the desired *devicegraph*. The +following is just an sketch of the steps of the partitioning process (the rest +of the sections will dig into the details for all of them): -## Adapting the existing code +1. The general settings are [imported][5]. It happens before the + `AutoinstStorage#Import` method is actually called. +2. The settings are [preprocessed][6]. Currently, the only preprocessing which + takes place is replacing `device=ask` with a real device name after asking + the user. Any preprocessing which implies interacting with the user should be + done at this point. +3. Build the proposed [devicegraph][7]. This is the complex part and it involves a + lot of stuff that will be described later. +4. If [any issue][8] is found during step 3, [present][9] it to the user. -### Features that can be added to the existing components/classes +The third step is quite generic and it is completely different depending on +whether a `partitioning` section is present in the profile or not. The +[Y2Autoinstall::StorageProposal][10] class decides which approach to follow. -The following features could, at first sight, be added to the current system -with a relatively low impact in the current structure and philosophy of the -code. +If there is no `partitioning` section, AutoYaST simply relies in the +{Y2Storage::GuidedProposal}. Bear in mind that such a proposal can be influenced +by general settings (leading to level 2). Although this process is quite +complex, it is out of the scope of this document. -* We need to add the `mountby`, `fstopt` and `mkfs_options` attributes to the - `PlannedPartition` and `PlannedLv` classes. The effort to implement them looks - quite reasonable at first sight, although it has been suggested to drop - `fstopt` and `mkfs_options` (which is, of course, even less effort than - implementing them). +However, when the profile contains a `partitioning` section, these are the steps: -* The `partition_nr` tag in the AutoYaST profile is useful for two goals. The - first one if defining a partition to reuse, something already covered by the - existing proposal code. The second goal would be to enforce the usage of a - particular number for newly created partitions, something that would require - some modifications in the current code. +1. Convert the AutoYaST profile -a hash including nested hashes and arrays- + into something better to work with. +2. Associate each drive included in the profile with a real drive. +3. Plan for new stuff by creating a set of *planned devices*. +4. Delete unwanted stuff from the devicegraph. +5. Modify the devicegraph to according to the list of planned devices. -* The effort to implement the `disklabel` property, that allows to specify the - type of partition table to use in each drive, also looks acceptable. Once - again, dropping support for it (which means YaST will always decide which - partition table type to use, as the current proposal does) can also be a - reasonable option. - -### Handling LVM - -The approach to LVM of the level 3 of AutoYaST and the current proposal is -totally different. - -In a normal installation, there is only one VG (that can be a reused one) and -the number and sizes of the PVs are calculated dynamically based on the needs -defined by the `PlannedLv` objects and the restrictions imposed by the -`PlannedPartition` ones. In other words, not all the planned partitions are -known at the beginning of the process, those partitions that will become PVs are -planned while generating the resulting devicegraph. All the LVM-related logic is -encapsulated in a class called `LvmHelper`. +## The AutoYaST level 3 procedure -With AutoYaST the process is more straightforward. The exact list of PVs is -specified in the AutoYaST profile. On the other hand, there can be several VGs -involved. +### Phase one: converting the profile data into proper objects -It makes sense to break the current `LvmHelper` into two components. The first -one will generate `PlannedPartition` objects as described (to become PVs) and -also an object of a new `PlannedVg` class. The second component, that should -probably be called `LvmCreator` for consistency reasons, could take collections -of planned partitions (PVs), planned VGs and planned LVs as input and create -the corresponding structures in the target devicegraph. As a bonus, that would -be more consistent with the rest of the current proposal mechanisms. +As you may know, an AutoYaST profile is basically an XML file which is converted +into a complex hash, including nested hashes and arrays. Working with such a +data structure is not convenient at all so, as a first step, the partitioning +section of the profile is turned into an +{Y2Storage::AutoinstProfile::PartitioningSection} object, which offers a a +better API for this use case. -The new `PlannedVg` and `LvmCreator` classes will be re-used by the AutoYaST -proposal and will have the capabilities needed to honor the following properties -from the AutoYaST profile: `pesize`, `lv_name`, `stripes`, `stripe-size`, -`lvm_group`, `pool`, `used_pool` and `keep_unknown_lv`. +This class does not work in isolation, as each element of the profile is +represented by a different class ({Y2Storage::AutoinstProfile::DriveSection}, +{Y2Storage::AutoinstProfile::PartitionSection}, etc.). +As a side note, the {Y2Storage::AutoinstProfile::PartitioningSection} class is +used when cloning a system too. Check the +{Y2Storage::AutoinstProfile::PartitioningSection.new_from_storage} method for +further details. -## The AutoYaST level 3 procedure - -### Phase one: assigning the drives +### Phase two: assigning the drives The `partitioning` section of the AutoYaST profile is organized into drives containing a list of partitions each. The result must honor that organization. @@ -146,100 +106,88 @@ The matching between a drive and the real disk can be done explicitly in the profile (using the `device` tag) or can be left for AutoYaST to decide. In the latter case, the algorithm used by AutoYaST is dead simple - it just tries to use the first available device that is not explicitly skipped (there is also a -syntax for skipping devices). The documentation states it tries to find a -device that "looks reasonable", but in reality it just uses whatever is not -skipped, with no optimization in place or even without checking if the device is -big enough. - -Thus, it would be easy to implement the same algorithm to decide beforehand the -disk allocating every planned partition. - -The `device` tag can have an undocumented value `ask` that will show a pop-up -asking the user which disk to use for a given device specification. It could be -dropped, although implementing it (initially or at a later point in time) -should not be difficult. - -### Phase two: deleting and resizing old stuff +syntax for skipping devices). The documentation states it tries to find a device +that "looks reasonable", but in reality it just uses whatever is not skipped, +with no optimization in place or even without checking if the device is big +enough. -All delete and resize operations are specified in the AutoYaST profile and, -thus, can be performed in a very early stage. No need to calculate resizing or -deleting on demand during the process like in the proposal procedure. - -As soon as the matching between AutoYaST devices and real disks is known, the -corresponding destructive operations can be performed in the target devicegraph. - -* Partition tables for devices with the `initialize` flag can be deleted right - away from the devicegraph. - -* AutoYaST also makes possible to specify the concrete way in which a given - partition must be resized (resizing LVs is explicitly unsupported). That can - also be implemented as an initial operation in the devicegraph. - -* The `use` tag in the profile can have the following values: `all`, `linux`, - `free` or a number. The first two options are very similar to some options - already implemented in the proposal, so they should be easy to implement. - The fourth one is even more straightforward. The third options means doing - nothing (even easier). All the removal operations can make use of the already - existing class `Proposal::PartitionKiller`. As a last consideration, - partitions marked to be reused in the AutoYaST profile must never be deleted. +The relationship between profile and real drives is keep in an instance of +{Y2Storage::Proposal::AutoinstDrivesMap}. ### Phase three: planning new stuff -After freeing all the space, the profile information can already be used to -create objects of the existing `PlannedPartition`, `PlannedLv` and `PlannedVg` -classes. In addition, a new `PlannedMd` class should be developed to allow the -specification of a future RAID. - -The size of every partition can be specified as a fixed size, as a percentage of -the total size to be used in the disk or with the keywords `max` and `auto`. -Since the disk for each partition and the space to be used in that disk will -be known beforehand, translating those sizes to attributes in the corresponding -`PlannedPartition` objects should be relatively easy. - -Similar reasoning can be applied to the LV sizes. - -During this phase, AutoYaST is also expected to propose any additional partition -required for booting, even if that partition is not present in the profile. The -class `BootRequirementsChecker` can be reused for that purpose. - -### Phase four: making it real - -The last phase consists in creating the new devices and it could, once again, -reuse many of the components of the current proposal. - -* `SpaceDistributionCalculator` can be used to allocate the partitions in the - disks and then `PartitionCreator` can be used to really create those - partitions in the devicegraph. - -* Then a new `MdCreator` class should take care of making the `PlannedMd` real - in the devicegraph. These are the only classes in this regard that would need - to be developed from scratch. - -* Last but not least, `LvmCreator` could be used to create/reuse the planned - VGs and LVs. - - -## Features that are hard to accommodate in the current model - -* As already mentioned by the [original overview -document](https://mailman.suse.de/mlarch/SuSE/yast-internal/2017/yast-internal.2017.04/msg00124.html), -there is an alternative to the AutoYaST `partitioning` section that can be used -to [reuse an existing mount table -(fstab)](https://www.suse.com/documentation/sles-12/singlehtml/book_autoyast/book_autoyast.html#ay.partition_fstab). -That section describes a completely different behavior that is, in fact, -incompatible with the `partitioning` one. As stated in the original document, it -makes sense to try to drop that feature completely. - -* When the size of the partitions in a drive are specified as fixed values (e.g. -"10Gb") but they don't fit in the disk, currently AutoYaST reduces the size of -the biggest partition as much as needed to make all the partitions fit. This -undocumented behavior can be a valid safety measure in several scenarios in -which the profile describes an non achievable situation, but there is probably -no need to keep it, since the combination of fixed sizes, percentages, `max` and -`auto` should be enough to specify a working schema. - -* Another undocumented feature makes possible to specify with total accuracy -(starting and ending block) the region in which a partition must be allocated. -Once again, this looks like a clear candidate for dropping, since it does not -fit into the general idea of letting YaST find the best possible layout and the -current behavior is, in fact, not documented. +With the drives map in-place, the profile information can already be used to +create {Y2Storage::Planned::Device} objects. Those objects are meant to contain +information about the devices that will be created or reused. There are +especialized classes for each device: {Y2Storage::Planned::Partition}, +{Y2Storage::Planned::LvmLv}, {Y2Storage::Planned::LvmVg}, +{Y2Storage::Planned::Md}, {Y2Storage::Planned::Bcache} and +{Y2Storage::Planned::StrayBlkDevice}. + +The class responsible for driving this phase is +{Y2Storage::Proposal::AutoinstDevicesPlanner}. Basically, it goes through the +list of drives contained in the profile creating the corresponding planned +devices according to the `type` (`:CT_DISK`, `:CT_LVM`, etc.). + +However, each type of drive is processed by a different planner class: + +* {Y2Storage::Proposal::AutoinstDiskDevicePlanner} for `:CT_DISK`. +* {Y2Storage::Proposal::AutoinstVgPlanner} for `:CT_LVM`. +* {Y2Storage::Proposal::AutoinstMdPlanner} for `:CT_MD`. +* {Y2Storage::Proposal::AutoinstBcachePlanner} for `:CT_BCACHE`. + +### Phase four: deleting old stuff + +All delete operations are postponed until the list of planned devices is +ready. The reason is that AutoYaST needs to know which devices are going to be +reused to avoid removing them. + +{Y2Storage::Proposal::AutoinstSpaceMaker} is the class responsible for cleaning +up the devicegraph according to the `use` and `initialize` attributes of each +drive section. + +### Phase five: adding the planned stuff to the devicegraph + +The last phase consists on updating the devicegraph to contain the planned +devices. For that purpose, and similarly to the +{Y2Storage::Proposal::AutoinstDevicesPlanner} case, a +{Y2Storage::Proposal::AutoinstDevicesCreator} class exists. It receives the +devicegraph and the list of planned devices. + +The logic to create each plan device, however, is splitted into several classes: + +* {Y2Storage::Proposal::LvmCreator} for {Y2Storage::Planned::LvmVg}. +* {Y2Storage::Proposal::MdCreator} for {Y2Storage::Planned::Md}. +* {Y2Storage::Proposal::BcacheCreator} for {Y2Storage::Planned::Bcache}. + +Note that there are no separate classes for {Y2Storage::Planned::Disk} and +{Y2Storage::Planned::StrayBlkDevice}. The logic for that kind of devices lives +in the {Y2Storage::Proposal::AutoinstDevicesCreator} for historical reasons and, +ideally, it should be extracted. + +## Issues reporting + +It is possible that, given a profile, AutoYaST finds issues when trying to +figure out a the partitioning layout. Some of those issues might be serious +enough to stop the installation; in other cases, just displaying a warning could +be the way to go. + +The {Y2Storage::AutoinstIssues} module features a +{Y2Storage::AutoinstIssues::List} where are problems are registered. After +trying (successfully or not) to create the proposal, AutoYaST displays the list +of problems (if any) to the user. If any of those issues is serious enough, it +will not allow the user to continue. + +{Y2Storage::AutoinstIssues} contains a set of possible issues (all of them +are classes which inherit from {Y2Storage::AutoinstIssues::Issue}). + +[1]: https://doc.opensuse.org/projects/autoyast/#CreateProfile.General.storage "General section documentation" +[2]: https://doc.opensuse.org/projects/autoyast/#CreateProfile.Partitioning "Partitioning documentation" +[3]: https://github.com/yast/yast-autoinstallation/blob/20bf1d0ed6dca9d7bd194308db1baf76fe7312cd/src/clients/storage_auto.rb "storage_auto client" +[4]: https://github.com/yast/yast-autoinstallation/blob/75af746a955be0d755e645da41061715329bcd7a/src/modules/AutoinstStorage.rb "AutoinstStorage module" +[5]: https://github.com/yast/yast-autoinstallation/blob/20bf1d0ed6dca9d7bd194308db1baf76fe7312cd/src/modules/AutoinstStorage.rb#L78 "AutoinstStorage#import_general_settings" +[6]: https://github.com/yast/yast-autoinstallation/blob/20bf1d0ed6dca9d7bd194308db1baf76fe7312cd/src/modules/AutoinstStorage.rb#L321 "AutoinstStorage#preprocessed_settings" +[7]: https://github.com/yast/yast-autoinstallation/blob/20bf1d0ed6dca9d7bd194308db1baf76fe7312cd/src/modules/AutoinstStorage.rb#L255 "AutoinstStorage#build_proposal" +[8]: https://github.com/yast/yast-autoinstallation/blob/20bf1d0ed6dca9d7bd194308db1baf76fe7312cd/src/modules/AutoinstStorage.rb#L276 "AutoinstStorage#valid_proposal?" +[9]: https://github.com/yast/yast-autoinstallation/blob/20bf1d0ed6dca9d7bd194308db1baf76fe7312cd/src/modules/AutoinstStorage.rb#L300 "Present issues to the user" +[10]: https://github.com/yast/yast-autoinstallation/blob/20bf1d0ed6dca9d7bd194308db1baf76fe7312cd/src/lib/autoinstall/storage_proposal.rb "StorageProposal"