extention-migration.md

• Feature Name: Salt Extensions Migration
• Start Date: 2022-07-26
• SEP Status: Draft
• SEP PR:
• Salt Issue:

Summary

Salt modules can be distributed as Salt Extensions. This SEP proposes extracting Salt modules into several categories and converting most of them into external Salt Extension repositories.

This allows us to do a few things:

1. Bulk deprecate modules that are no longer used, maintained, or relevant.
2. Release modules independent of Salt versions – decoupling the extension module release cycle from Salt allows for more up-to-date plugins and better backporting. For extensions that are pure Python, only a single artifact will need to be released, rather than dozens per release.
3. Release platform specific versions of Salt that are smaller and more efficient.
4. Speed up PR merges and reviews through faster tests and a smaller core code base.

This SEP proposes that the existing Salt modules be broken up into 3 macro categories. A more detailed description of these categories will be given in the Design section.

1. Core modules – modules that we continue to ship with Salt
2. SaltStack supported modules – OS (Operating Systems) and core application support modules
3. Community modules – plugins which will be deprecated with a request to community contributors to continue maintenance

This SEP also proposes creating a Salt Extension Ecosystem, building functionality to improve the experience for Salt Users, Extension Authors/Maintainers, Contributors, and other community members.

Motivation

When I first created Salt, I wanted to ship a powerful, batteries included automation platform. I wanted to make Salt something that worked out of the box with little to no need for complicated setup or dependency management.

That dream was realized as we built in extensive support for multiple operating systems, APIs, services, and more. Unfortunately, as time elapsed, this model proved to be too high of a burden for the maintenance of Salt.

After a few years I realized that we needed to split out the modules in Salt to improve overall maintainability. But Salt had become so modular, so pluggable, that doing so in an orderly way was a daunting task.

In response to this, technologies were developed to address these concerns. The pop project was built to solve this problem, but it grew in a different direction. In the end, it was agreed that Salt Extensions would work best to separate out the collections of modules into multiple repositories.

Design

The existing Salt modules will be carved up into one of three categories. Each category will be implemented in the following way:

Core Modules

Core Modules will be kept inside the main Salt codebase, and development will be tied to the Salt release cycle.

Supported Modules

Supported modules will be moved to their own repositories within the SaltStack Github organization where they can be maintained separately from the Salt codebase.

Community Modules

Remaining modules will be deprecated from the Salt Core codebase and community members will be able to continue independent maintainership if they are interested. Some plugins are almost exclusively maintained by external corporations – if these corporations wish for formal documentation outlining transfer of ownership it can be handled on a case-by-case basis. The community modules can be hosted either in individual or corporate source control systems, alternatively they can also be hosted in the community run Salt Extensions Github organization, that will operate like the the Salt Formulas Github organization. The criteria to determine which category to place modules in will follow these rules.

Core Modules

1. Required Salt Functionality a. Modules such as state, sys, peer, grains, pillar, etc.
2. Modules critical to Salt’s Multi OS support a. Modules that function across multiple operating systems like cmd and file.

Supported Modules

1. Modules to support specific operating systems traditionally maintained by the core team – such as RedHat, MacOS, Windows, Solaris, etc.
2. Modules to support specific but critical applications, such as Apache, MySQL, etc.
3. Modules created and maintained as part of VMware backed support agreements and contracts.

Community Extension Modules

1. Modules to support specific operating systems traditionally maintained by the OS vendor – such as SUSE, openBSD, NetBSD, etc.
2. Modules to support cloud interfaces, such as AWS, Azure, etc.
3. Modules no longer maintained, or which we suspect are also no longer used or maintained, such as moosefs, qemu_img, etc.

Salt Extension Directory

The Salt Extension Directory will be further designed and developed as a part of the Salt Extension efforts. At a high level, the directory will be a centralized place where users can find extension modules that accomplish their needs. Extension modules should be listed and searchable, with helpful information such as the quality of the documentation, maturity/churn of the extension codebase, number of unresolved issues, etc., to help users make informed decisions about the extension modules they’re consuming, and help potential contributors find communities where their efforts will be welcomed and effective.

Alternatives

Keep doing what we are doing.

Unresolved questions

These rules and guidelines may need better clarification to make the decision process easier and clearer.

Drawbacks

The biggest drawback when creating the Salt Extension Ecosystem will be that in order to maintain or improve support standards will require designing, building, and maintaining the Extension Directory. Separating modules out into a multitude of extensions will also require an increased level of communication and collaboration between the Salt Core team and the Salt Community. It will require better documentation from the Salt Core team on how to get started building extension modules. The level of effort in extracting extension modules is non-trivial. While testing individual pieces will be much more trivial, integration testing on multiple different modules in conjunction with Salt, when necessary, will be more complex. This SEP is the starting point to those efforts. This does spread the maintenance out across multiple repositories, which may add release complexity. Certain parts of the codebase could be neglected. But if they are neglected, then it will be far more visible and manageable. The migration process will be time consuming. But we strongly suspect that the investment time will be quickly offset by dramatic improvements in test performance. This will make adding new code to Salt significantly easier.