From a727a7b3936f0a0819dd7e8a61380dfe99bab0b7 Mon Sep 17 00:00:00 2001 From: Petr Stodulka Date: Thu, 30 Jan 2020 18:37:31 +0100 Subject: [PATCH 1/3] Document practice of LEAPP and LEAPP_DEVEL envars --- docs/source/actor-rhel7-to-rhel8.md | 2 +- docs/source/best-practices.md | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/source/actor-rhel7-to-rhel8.md b/docs/source/actor-rhel7-to-rhel8.md index d40f4420a..7b1038c36 100644 --- a/docs/source/actor-rhel7-to-rhel8.md +++ b/docs/source/actor-rhel7-to-rhel8.md @@ -359,7 +359,7 @@ To flush all saved messages from the repository database, run `snactor messages ## Writing tests for an actor -[Read the tutorial](unit-testing) for writing an running unit and component tests +[Read the tutorial](unit-testing) for writing and running unit and component tests ## Best practices diff --git a/docs/source/best-practices.md b/docs/source/best-practices.md index 43ec6415b..26cfde249 100644 --- a/docs/source/best-practices.md +++ b/docs/source/best-practices.md @@ -151,3 +151,11 @@ In case of [StopActorExecutionError](pydoc/leapp.html#leapp.exceptions.StopActor - [ReportOnly](pydoc/leapp.workflows.html?highlight=FailPhase#leapp.workflows.policies.Policies.Errors.ReportOnly) - do not end the workflow execution at all and continue with logging the issue only You can also use the [StopActorExecution](pydoc/leapp.html#leapp.exceptions.StopActorExecution) and [StopActorExecutionError](pydoc/leapp.html#leapp.exceptions.StopActorExecutionError) exceptions inside a private or shared library. + +## Use the LEAPP and LEAPP\_DEVEL prefixes for new envars + +In case you need to change a behaviour of actor(s) for testing or development purposes - e.g. be able to skip a functionality in your actor - use environment variables. Such environment variables should start with prefix *LEAPP\_DEVEL*. Such variables are not possible to use on production systems without special *LEAPP\_UNSUPPORTED* variable. This prevents users to break their systems by a mistake. + +If you want to enable user to modify behaviour on production systems as well (e.g. enable user increase size of something the actor is producing), use just *LEAPP* prefix and ensure you inform user about the variable if needed. + +In both cases, such environment variables should be mentioned in related commit messages. From e48178bc2bcc47e83332b7ab872c7a4862f5692f Mon Sep 17 00:00:00 2001 From: Petr Stodulka Date: Thu, 20 Feb 2020 22:32:07 +0100 Subject: [PATCH 2/3] Documentations: changes in hierarchy + a new content I realized it's hard to add new content specific to the el7toel8 leapp repository. As well, with new and new content, it's getting harder and harder to find something in the menu as number of items is growing. For that purpose, here is proposed the new structure of documentation that tries to reduce top level menu into categories that should cover other content in future we would like to add. TODO: it's still PoC. Another changes are expected yet. --- .../{ => el7toel8}/actor-rhel7-to-rhel8.md | 18 ++--- docs/source/el7toel8/envars.md | 65 +++++++++++++++++++ .../{ => el7toel8}/inhibit-rhel7-to-rhel8.md | 0 .../el7toel8/leapp-repository-el7toel8.rst | 10 +++ docs/source/faq.md | 8 +-- docs/source/index.rst | 5 +- docs/source/infrastructure.rst | 10 +++ docs/source/leapp-repositories.rst | 14 ++++ docs/source/tutorials.rst | 4 +- 9 files changed, 116 insertions(+), 18 deletions(-) rename docs/source/{ => el7toel8}/actor-rhel7-to-rhel8.md (96%) create mode 100644 docs/source/el7toel8/envars.md rename docs/source/{ => el7toel8}/inhibit-rhel7-to-rhel8.md (100%) create mode 100644 docs/source/el7toel8/leapp-repository-el7toel8.rst create mode 100644 docs/source/infrastructure.rst create mode 100644 docs/source/leapp-repositories.rst diff --git a/docs/source/actor-rhel7-to-rhel8.md b/docs/source/el7toel8/actor-rhel7-to-rhel8.md similarity index 96% rename from docs/source/actor-rhel7-to-rhel8.md rename to docs/source/el7toel8/actor-rhel7-to-rhel8.md index 7b1038c36..19bf394cd 100644 --- a/docs/source/actor-rhel7-to-rhel8.md +++ b/docs/source/el7toel8/actor-rhel7-to-rhel8.md @@ -56,13 +56,13 @@ New actor MyNewActor has been created at /usr/share/leapp-repository/repositorie The main file of the actor is the actor.py in which you’ll write code for logic of the actor. -For further information about how create an actor read this [document](first-actor). +For further information about how create an actor read this [document](../first-actor). ## Including an actor in the RHEL 7 to 8 upgrade process Until now, you have created boilerplate of a new actor and made it visible to Leapp. But, Leapp needs some more information about what to do with the actor. Specifically, in which **“workflow”** and in which **“phase”** the actor should be executed. A workflow is a sequence of phases. The only workflow available now is the one solving the upgrade of RHEL 7 to RHEL 8. Each phase is a set of actors that will be executed one after another before the next phase starts. To find out in which workflow and phase should the actor be executed, Leapp looks for **“tags”**. To be part of RHEL 7 to RHEL 8 upgrade workflow, an actor needs to be tagged with **IPUWorkflowTag**. -The phases of the IPUWorkflow (in order) are: **Facts Collection, Checks, Report, Download, Upgrade RamDisk Preparation, Upgrade RamDisk Start, Late Tests, Preparation, RPM Upgrade, Application Upgrade, Third Party Applications, Finalization** and **First Boot**. Each phase has a specific tag that marks an actor as being part of that phase. You can find descriptions of all the phases and their tags [here](https://github.com/oamg/leapp-repository/blob/master/repos/system_upgrade/el7toel8/workflows/inplace_upgrade.py) and workflow diagram [here](inplace-upgrade-workflow). +The phases of the IPUWorkflow (in order) are: **Facts Collection, Checks, Report, Download, Upgrade RamDisk Preparation, Upgrade RamDisk Start, Late Tests, Preparation, RPM Upgrade, Application Upgrade, Third Party Applications, Finalization** and **First Boot**. Each phase has a specific tag that marks an actor as being part of that phase. You can find descriptions of all the phases and their tags [here](https://github.com/oamg/leapp-repository/blob/master/repos/system_upgrade/el7toel8/workflows/inplace_upgrade.py) and workflow diagram [here](../inplace-upgrade-workflow). For example, if an actor is to be executed within the Checks phase, it needs to be tagged both with IPUWorkflowTag and ChecksPhaseTag. The result after updating the boilerplate would be: @@ -88,7 +88,7 @@ class MyNewActor(Actor): All communication between actors in Leapp is carried out using **“messages”**. An actor can consume or produce messages. A message may contain any data, but the data needs to be in a specific format defined by a **“model”**. If an actor wants to consume a message produced by another actor, it needs to specify the specific model of the consumed messages. Leapp will make sure to execute such an actor only after some message of the specified model was produced by another actor. If no message of the specified model was produced in previous phases or in the current phase, the consuming actor will get no messages of that kind. -For further information about messaging refer to this [document](messaging). +For further information about messaging refer to this [document](../messaging). One of the existing models in Leapp is [ActiveKernelModulesFacts](https://github.com/oamg/leapp-repository/blob/master/repos/system_upgrade/el7toel8/models/activekernelmodulesfacts.py). Messages from this model contain data about the system on which Leapp has been started. For example, it contains installed kernel modules. If an actor wants to perform some action based on existing kernel modules on the system, the actor can get list of these modules by consuming the _ActiveKernelModulesFacts_ messages. By extending the boilerplate, the code could look like this: @@ -132,7 +132,7 @@ The following restrictions apply: change the behavior of any other dialog that any actor might create. - Dialogs can be used only at certain stages of the workflow - ChecksPhase and TargetTransactionChecksPhase. -For more information and real examples please check [dialogs](dialogs). +For more information and real examples please check [dialogs](../dialogs). ### Producing data for other actors and reporting @@ -359,11 +359,11 @@ To flush all saved messages from the repository database, run `snactor messages ## Writing tests for an actor -[Read the tutorial](unit-testing) for writing and running unit and component tests +[Read the tutorial](../unit-testing) for writing and running unit and component tests ## Best practices -Read the best practices [document](best-practices) and [Python guidelines](https://github.com/oamg/leapp-guidelines/blob/master/python-coding-guidelines.md). +Read the best practices [document](../best-practices) and [Python guidelines](https://github.com/oamg/leapp-guidelines/blob/master/python-coding-guidelines.md). ## Contributing actors to the Leapp project @@ -373,7 +373,7 @@ All new content that needs to be part of Leapp release distributed to all users **Before submitting your work for review, make sure you have read and followed the contribution guidelines:** -[Contribution guidelines for writing actors](contributing) +[Contribution guidelines for writing actors](../contributing) This [pull request](https://github.com/oamg/leapp-repository/pull/186) gives a good example of both guidelines-driven actor implementation and thorough test coverage. @@ -381,7 +381,7 @@ This [pull request](https://github.com/oamg/leapp-repository/pull/186) gives a g ### In which existing workflow phase should I place my new actor? -You can decide that based on the description of the phases this information is available in the [code](https://github.com/oamg/leapp-repository/blob/master/repos/system_upgrade/el7toel8/workflows/inplace_upgrade.py) and diagram [here](inplace-upgrade-workflow). Please note that if your actor depends on some message generated by another actor, it cannot be executed in a phase before the phase of such actor. In a similar way, if your actor produces data, it needs to be executed before the actor consuming the data. +You can decide that based on the description of the phases this information is available in the [code](https://github.com/oamg/leapp-repository/blob/master/repos/system_upgrade/el7toel8/workflows/inplace_upgrade.py) and diagram [here](../inplace-upgrade-workflow). Please note that if your actor depends on some message generated by another actor, it cannot be executed in a phase before the phase of such actor. In a similar way, if your actor produces data, it needs to be executed before the actor consuming the data. ### How to stop the upgrade in case my actor finds a problem with the system setup? @@ -414,7 +414,7 @@ Python 2.7+/3.6+, but keep in mind that the resulting code has to be both py2 an Under “tests” folder, an actor can have Python files containing tests that will be executed using PyTest. Leapp provide tests utilities to simulate actor consuming and checking actor production. Please, refer to detailed Leapp documentation about how to write tests. -For further information read: [Writing tests for actors](unit-testing). +For further information read: [Writing tests for actors](../unit-testing). ### How to use libraries or data files in my new actor? diff --git a/docs/source/el7toel8/envars.md b/docs/source/el7toel8/envars.md new file mode 100644 index 000000000..294e8ac49 --- /dev/null +++ b/docs/source/el7toel8/envars.md @@ -0,0 +1,65 @@ +# Environment variables for el7toel8 repository + +Actors in the el7toel8 repository use environment variables specified below. +All these envars use the suggested prefixes specified in +[the best practices document](../best-practices.html#use-the-leapp-and-leapp-devel-prefixes-for-new-envars) +for the leapp project to distinguish their purpose: *production* or *devel* use. + +If the argument for envars below is not specified, it is expected to set `0` +(false) or `1` (true). + +## LEAPP_UNSUPPORTED + +Necessary to use in case you use any envar with the LEAPP_DEVEL prefix +(see the list below). And in case you use the --whitelist-experimental option +for the Leapp tool. + + +## LEAPP_DEVEL_RPMS_ALL_SIGNED + +Leapp will consider all installed pkgs to be signed by RH - that affects +the upgrade process as by default Leapp upgrades only pkgs signed by RH. +Leapp takes care of the RPM transaction (and behaviour of applications) +related to only pkgs signed by Red Hat. What happens with the non-RH signed +RPMs is undefined. + + +## LEAPP_DEVEL_TARGET_RELEASE + +Change the default target RHEL 8 minor version. + + +## LEAPP_DEVEL_SKIP_RHSM + +Do not use `subscription-manager`. Using this option (now) requires creating +a custom actor providing RHEL 8 repos through CustomTargetRepository message. + + +## LEAPP_DEVEL_SKIP_CHECK_OS_RELEASE + +Do not check whether the source RHEL 7 version is the supported one. +E.g. right now Leapp does not allow you to proceed with the upgrade +when you’re not on RHEL 7.6. + + +## LEAPP_DEVEL_DM_DISABLE_UDEV + +Setting the environment variable provides a more convenient +way of disabling udev support in libdevmapper, dmsetup and LVM2 tools globally +without a need to modify any existing configuration settings. +This is mostly useful if the system environment does not use udev. + + +## LEAPP_DEVEL_SOURCE_PRODUCT_TYPE + +By default the upgrade is processed from the GA (general availability) system +using GA repositories. In case you need to do the in-place upgrade from +Beta or HTB system, use the variable to tell which of those you would like +to use. It's case insesitive. The `ga` value is the default. +Expected values: `ga`, `beta`, `htb`. + + +## LEAPP_DEVEL_TARGET_PRODUCT_TYPE + +Analogy to the LEAPP_DEVEL_SOURCE_PRODUCT_TYPE envar, just for the target +system this time. Again, `ga` is the default. diff --git a/docs/source/inhibit-rhel7-to-rhel8.md b/docs/source/el7toel8/inhibit-rhel7-to-rhel8.md similarity index 100% rename from docs/source/inhibit-rhel7-to-rhel8.md rename to docs/source/el7toel8/inhibit-rhel7-to-rhel8.md diff --git a/docs/source/el7toel8/leapp-repository-el7toel8.rst b/docs/source/el7toel8/leapp-repository-el7toel8.rst new file mode 100644 index 000000000..f8e81ccbb --- /dev/null +++ b/docs/source/el7toel8/leapp-repository-el7toel8.rst @@ -0,0 +1,10 @@ +Leapp repository for RHEL 7 to RHEL 8 upgrade +============================================= +This is the official upstream documentation for the leapp repository for in-place upgrade (IPU) from RHEL 7 to RHEL 8. The homepage of the project is `here `_. + +.. toctree:: + :caption: Contents: + + actor-rhel7-to-rhel8 + inhibit-rhel7-to-rhel8 + envars diff --git a/docs/source/faq.md b/docs/source/faq.md index 82a025c07..66de4e068 100644 --- a/docs/source/faq.md +++ b/docs/source/faq.md @@ -23,7 +23,7 @@ Leapp project aims to enable users to modernize their existing workloads without ## How can I get on board with contributing to Leapp? -For the Leapp framework we are currently developing the functionality for the in-place upgrade of RHEL 7 to RHEL 8. You can improve the user experience of the upgrade by creating so called actors for the Leapp framework. We've written a quick guide on how to create such actors for the RHEL 7 to RHEL 8 upgrades: [How to create a Leapp actor for RHEL 7 to 8 upgrade.](actor-rhel7-to-rhel8) +For the Leapp framework we are currently developing the functionality for the in-place upgrade of RHEL 7 to RHEL 8. You can improve the user experience of the upgrade by creating so called actors for the Leapp framework. We've written a quick guide on how to create such actors for the RHEL 7 to RHEL 8 upgrades: [How to create a Leapp actor for RHEL 7 to 8 upgrade.](el7toel8/actor-rhel7-to-rhel8) ## What is an actor and what does it do? @@ -38,13 +38,13 @@ In regards to the upgrades of RHEL 7 to RHEL 8, Leapp should be able to upgrade ## How can I exchange any data between actors? All communication between actors in Leapp is carried out using " messages". An actor can consume or produce messages. A message may contain any data, but the data needs to be in a specific format defined by a "model". If an actor wants to consume a message produced by another actor, it needs to specify the specific model of the consumed messages. Leapp will make sure to execute such an actor only after some message of the specified model was produced by another actor. If no message of the specified model was produced in previous phases or in the current phase, the consuming actor will get no messages of that kind. -Source: [How to create a Leapp actor for RHEL 7 to 8 upgrade](actor-rhel7-to-rhel8) +Source: [How to create a Leapp actor for RHEL 7 to 8 upgrade](el7toel8/actor-rhel7-to-rhel8) ## What do I have to do in order to execute actor I just wrote? If you want to execute just a single actor when developing it, then use the snactor tool. [Here's a tutorial](first-actor) on how to use it. If you want to add your actor to an existing workflow, for example the RHEL 7 to 8 upgrade workflow, then tag your actor with appropriate workflow and phase tags. -Source: [How to create a Leapp actor for RHEL 7 to 8 upgrade](actor-rhel7-to-rhel8) +Source: [How to create a Leapp actor for RHEL 7 to 8 upgrade](el7toel8/actor-rhel7-to-rhel8) ## What should I do if I need to execute multiple actors? Can I somehow ensure the dependencies between them? @@ -78,7 +78,7 @@ It should follow the [Contribution guidelines](contributing) and the [Best pract ## How can I debug my actor? Is there a standard/supported way how to log and get logs from actors/channels? You can run your actor using the snactor tool and printing the output. [See the tutorial](first-actor) on how to use snactor. -Source: [How to create a Leapp actor for RHEL 7 to 8 upgrade](actor-rhel7-to-rhel8) +Source: [How to create a Leapp actor for RHEL 7 to 8 upgrade](el7toel8/actor-rhel7-to-rhel8) ## Are there some technical limitations for an actor? E.g. maximum time execution, size of the input/output, libraries I can use... In case there are, is it possible to specify that the actor needs e.g. longer time for execution? diff --git a/docs/source/index.rst b/docs/source/index.rst index ef4787f8a..3bca62f24 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -12,12 +12,11 @@ Resources not included in this RTD: terminology tutorials - repository-dir-layout + leapp-repositories + infrastructure best-practices test-actors architecture - dependencies - dependencies-leapp-repository inplace-upgrade-workflow contributing python-coding-guidelines diff --git a/docs/source/infrastructure.rst b/docs/source/infrastructure.rst new file mode 100644 index 000000000..505b45dcb --- /dev/null +++ b/docs/source/infrastructure.rst @@ -0,0 +1,10 @@ +Infrastructure +============== +Here you can find documentation related to our CI/CD, packaging, etc. + + +.. toctree:: + :caption: Contents: + + dependencies + dependencies-leapp-repository diff --git a/docs/source/leapp-repositories.rst b/docs/source/leapp-repositories.rst new file mode 100644 index 000000000..8db98569a --- /dev/null +++ b/docs/source/leapp-repositories.rst @@ -0,0 +1,14 @@ +Leapp repositories +================== + +Here you can find all the information related to +`leapp repositories `_, including the documentation +for existing leapp repositories managed by the OS and Application +Modernization Group (`OAMG `_) + + +.. toctree:: + :caption: Contents: + + repository-dir-layout + el7toel8/leapp-repository-el7toel8 diff --git a/docs/source/tutorials.rst b/docs/source/tutorials.rst index 948f585c9..d79d58c59 100644 --- a/docs/source/tutorials.rst +++ b/docs/source/tutorials.rst @@ -7,8 +7,8 @@ Tutorials devenv-install create-repository first-actor - actor-rhel7-to-rhel8 - inhibit-rhel7-to-rhel8 + el7toel8/actor-rhel7-to-rhel8 + el7toel8/inhibit-rhel7-to-rhel8 messaging dialogs repo-linking From d85d1c6b971fad7c2fec92ac7e8db3439d2cc9e9 Mon Sep 17 00:00:00 2001 From: Petr Stodulka Date: Thu, 20 Feb 2020 22:53:30 +0100 Subject: [PATCH 3/3] Document the new leapp-framework capability See the commit d37e2b4 for more info about the capability. --- .../compatibility-with-leapp-repository.md | 59 +++++++++++++++++++ docs/source/infrastructure.rst | 1 + 2 files changed, 60 insertions(+) create mode 100644 docs/source/compatibility-with-leapp-repository.md diff --git a/docs/source/compatibility-with-leapp-repository.md b/docs/source/compatibility-with-leapp-repository.md new file mode 100644 index 000000000..afdf237f2 --- /dev/null +++ b/docs/source/compatibility-with-leapp-repository.md @@ -0,0 +1,59 @@ +# Compatibility between leapp and leapp repositories + +There is a hidden dragon in the split of leapp framework and leapp repositories +projects. The development of features and incompatible changes has different +speed in both projects. We want to release our projects using semantic +versioning and, at the same time, we do not want to release a new version of +the project after each merge of a new feature or incompatible change. +We rather prefer releasing new versions with changelogs and everything +when we agree it's worthwhile. + +But we need a mechanism to be able to synchronize with other projects, when we +provide new functionality in the upstream (master) branch without the need +of immadiate release of the new version of leapp. For these purposes the +`leapp-framework` capability is provided in the framework (python[23]-leapp) rpms. + +## When and how change the capability + +The `leapp-framework` capability has to be changed in case of any change of +provided API (e.g. change in the leapp standard library) or functionality that +could affect actors in leapp repositories. That includes new features as well +as an actor may depend on that in future, and the leapp-repository package +should be able to specify correct framework version that is needed. + +The `leapp-framework` capability uses just `X.Y` versioning: + +- In case of a change breaking previous functionality (e.g. changed a name of a function) provided by the framework, bump `X` and set `Y` to 0. +- In case of a new feature (e.g. new function in stdlib), which doesn't affect already existing actors, bump `Y` + +The value of the capability should be bumped per PR providing all changes. +IOW, when a PR implements two compatible/incompatible changes in several +commits, the value should be bumped only once! + +As well it is recommended to mentioned the value of the capability +(when changed) in the commit msg, so it the tracking is easy. + +## Suggested dependency in leapp-repository rpms +We suggest to set dependency on the `leapp-framework` capability in any rpm +that contains leapp repositories in this way: + +```spec +Requires: leapp-framework >= 1.2, leapp-framework < 2 +``` + +Which means, that you want to install framework of version at least 1.2 but +lower than 2 (e.g. 1.99). + +### Possible issue +There is unfortunately one issue with the above solution. As far as there is just +one rpm providing that capability, it's ok. The problem occurs on systems where +there will be more rpms (of different name) providing the capability. +So in case you are able to install e.g. python2-leapp and python3-leapp rpms +on the system, you could end up with: + +- python2-leapp providing leapp-framework 1.0, and +- python3-leapp providing leapp-framework 3.0 + +which both are broken for leapp repository and the dependency from the point of +rpms is satisfied. This should happen rarely. We suggest you to ensure that you +use such repositories where only one of those rpms exists. diff --git a/docs/source/infrastructure.rst b/docs/source/infrastructure.rst index 505b45dcb..ed558cecd 100644 --- a/docs/source/infrastructure.rst +++ b/docs/source/infrastructure.rst @@ -8,3 +8,4 @@ Here you can find documentation related to our CI/CD, packaging, etc. dependencies dependencies-leapp-repository + compatibility-with-leapp-repository