[Design] OpenSearch/Dashboards DEB/RPM Distribution Design Document

[peterzhuamazon]

Hi everyone,

We want to give you more details on how we currently experiment/implement the assemble workflows of DEB/RPM packages for the future releases on OpenSearch/Dashboards.

Please feel free to use this as guidance for contributing on DEB/RPM, as well as raising questions about our approach.

Original Draft PR for RPM POC on OpenSearch and Dashboards

We currently have a draft PR for review: #1443.

Thanks.

---

OpenSearch/Dashboards DEB/RPM Distribution Design Document

Introduction

Problem Statement

As of 2021/11, OpenSearch project only publish Tarball and Docker Images as part of the release artifacts. As a result, the community is asking for DEB/RPM packages from us, as it is an easy approach for LINUX users to deploy, run, and test OpenSearch Clusters alongside OpenSearch-Dashboards, without the complicated manual setups on Tarball. After some research, we clearly see the demands from the community and start to investigate the requirements/steps for supporting these new distributions.
Here is the Meta issue to track the progress.

Designed Features (Outdated as of 2022/01/24)

## Designed Features

1. Both .deb and .rpm packages will be provided for users to download and install on their LINUX hosts.
2. The package will include both the Min code as well as all the plugins for both OpenSearch and Dashboards. (Our design will only support the Distribution DEB/RPM not MIN DEB/RPM).
3. The design contains two parts of changes:
   1. Assemble_workflow changes to add DEB/RPM distributions as part of the workflow, setup pre-requisite / requirements before a package is generated, and call Packaging to generate the package.
   2. Packaging changes to include a tool for packaging OpenSearch/Dashboards Distribution/Bundle into a package.
4. Assemble_workflow needs to provide --distribution as an optional argument for the customer to specify which distribution to build. The argument is exposed to assemble.sh . The default value of --distribution is “tar”. We should also support “rpm”, “deb” and “zip” as options of distribution. (“deb” will be an option in later updates)

Requirements (Updated as of 2022/01/24 with more details and milestones)

1. [M1] OpenSearch build process design for RPM generation
   1. The build workflow should generate the contents of the RPM packages as pre-requisite
      1. Re-use the existing build process to generate the OpenSearch/Dashboards min + all of the plugins artifacts for RPM package to use
      2. The artifacts should be built with LINUX platform specified, as we only plan to support RPM package on LINUX
      3. Tweak the build workflow to have a —distribution parameter so the internal API call can show “rpm” text, instead of the default “tar” text, as well as removing tar related files
   2. The assemble workflow should have a function to wrap build artifacts into a RPM package
      1. Just like build workflow, add a --distribution parameter, so that the assemble workflow knows which distribution to create based on the input artifacts from build workflow
      2. The existing distribution generating code of TAR and ZIP should still function after the above change
      3. Add a new block specifically to call the generation code for RPM
         1. The generation code should pull the artifacts from the build workflow to a temporary location
         2. The code will call existing install function to install plugins on min artifacts
         3. After installation, the code will execute a tool or utility to wrap all the content into a RPM package
         4. The code will also add dependencies to the RPM package so that things like JDK and additional libs for plugins can be installed and pulled separately
         5. The code will move the RPM package from the temp location to dist folder
2. [M2] RPM Package Signing
   1. The signing workflow should have a function to sign the RPM package with OpenSearch Project PGP key
      1. Just like the build and the assemble workflow, add a --distribution parameter so that the signing workflow can use rpm toolkit to sign RPM packages
      2. Adding the rpm toolkit call for RPM package to OpenSearch Signer Client, as it is the backend for signing workflow to function correctly
      3. A checksum will also be created after signing is completed to ensure the integrity of the package
3. [M3] RPM Package Design
   1. The RPM package should adhere to below standards and requirements
      1. RPM package name should follow the binary file naming scheme %{name}-%{version}-%{release}.%{architecture}.rpm so we have a clear path for upgrade/re-release (e.g. opensearch-1.2.0-1.x86_64.rpm)
      2. RPM package should be set to be a un-relocatable package, so that users will not get confused and move the content to a different directory, which then cause the software to be unfunctional
      3. RPM package should have its dependencies pulled upon installation, without bundling them in the package
      4. RPM package will have pre/post install/remove scripts to handle min and plugin specific configurations
      5. RPM package should let rpm itself handle the creation and removal of user/group/dir during installation / removal, and remove any of these logic from the pre/post scripts
      6. RPM package should be as slim as possible, without duplicate libs and related dependencies
      7. RPM package should be installed in /opt directory to follow the LINUX standard, and add a symlink to the old directory presented in ODFE for backward compatibility
      8. RPM package should have no reference to Elasticsearch or Kibana
      9. RPM package should redirect its log to /var/log location per LINUX standard
      10. RPM package removal will not remove user data from the LINUX host
      11. RPM package should have a clear description of its content in metadata
      12. RPM package should have a clear label of LICENSE file location, as well as the LICENSE type in its metadata. The LICENSE file should always present on the host once installed
      13. RPM package should have its documentation files labeled as doc, so that users can choose whether they want to install the docs to save space
      14. RPM package should contain process manager related service file, so user can manage the software with ease
      15. RPM package should not couple the opensearch/dashboards process with any additonal plugin services to reduce the complexity and improve the management
      16. RPM package should handle the initialization of certain plugins automatically
4. [M4] RPM Package Publishing
   1. The RPM packages should be published in these two ways:
      1. Publish YUM repository
         1. Deploy the RPM packages into a local YUM repository on a build system
         2. The metadata files for the entire repo will be generated and staged in the repo
         3. The build system should try to install the RPM packages through the local repo and verify the commands running correctly
         4. The build system should attempt to start the service of the installed packages and verify the outcome through process manager of the service (see more details in the test requirements)
         5. Once verified, the entire repo should be uploaded to the dedicated production S3 bucket linked to artfacts.opensearch.org
         6. A repo file is also generated (once every major release) and updated on our website, for user to download and deploy on their host / server
         7. Once deployed, user should be able to use YUM command to install the latest RPM packages from our repository on S3, with the help of CloudFront cache in the front
      2. Publish individual package
         1. Update the link of the RPM package on S3/CloudFront to our website download pages
         2. Users can manually download the RPM package with the link
         3. Users can then install the package manually with YUM/RPM command on their server, or they can deploy the package to their own internal repo and consume with YUM
5. [M5] RPM Package installation
   1. The RPM packages can only be installed on RedHat-Based LINUX distros stated in our compatibility chart
      1. According to the chart, we will only support RHEL/CentOS 7 & 8 + Amazon Linux 2 for RPM
   2. We have two ways to install the package
      1. Install through YUM repository
         1. Users download our repo file through the link we provide on website download page
         2. Users place the repo file manually in /etc/yum.repo.d/ directory
         3. Users can also use yum-config-manager --add-repo repository_url to do so
         4. Users can use yum install command to install our RPM packages
         5. Users can use systemctl related commands to manage the service
      2. Install through manual downloads
         1. Users download the RPM package directly through the links we provide on website download page
         2. Users can use rpm -ivh command to manually install the package on host
         3. Users can use systemctl related commands to manage the service
6. [M6] RPM Validation
   1. The build system should apply tests to validate the quality of the product before releasing to public
      1. Validate package downloading
         1. Use yum-downloader to test the package availability in remote yum repo
         2. Use curl to test the package availability through download links on website
      2. Validate package installing
         1. Install rpm through local yum repository
         2. Install rpm through remote yum repository
         3. Install rpm through installing the package directly without yum
      3. Validate package signing/checkout
         1. Use yum to auto verify the package signature/checksum through repo file
         2. Use rpm to verify downloaded package and its signature/checksum
         3. Ensure our public key is available either local or remote
      4. Validate package running
         1. Use systemd to start/stop the software service, and check whether the log output match the process manager status messages
         2. Test the start time of the services through API and setup a threshold
         3. Executing software binary directly without systemd if systemd could not bring service up in time
      5. Validate package upgrade/downgrade
         1. Users should be able to easily upgrade/downgrade the package
            1. If user use yum to manage the installation
               1. Upgrade:
                  1. yum upgrade to upgrade to latest version
                  2. yum upgrade <package-version> to upgrade to a specific version
               2. Downgrade:
                  1. yum downgrade to downgrade to previous version
                  2. yum downgrade <package-version> to downgrade to a specific version
            2. If user use manual rpm installation
               1. Upgrade:
                  1. User manually download a newer version and use rpm -Uvh to upgrade manually
               2. Downgrade:
                  1. User manually download a older version and use rpm -Uvh —oldpackage to downgrade manually
         2. We need to have automation to replicate these practice for testing purposes
         3. We need to have a matrix showing which version to which version is upgradable/downgradable for shards compatibility reasons

Implementation

Overview (Outdated as of 2022/01/24. See above section for new requirements)

### Overview

We are breaking the design into 2 parts, Assemble Workflow and Packaging.

• Run existing build_workflow to generate Min+Plugins Build Binary for OpenSearch/Dashboards
• Run assemble_workflow with our code changes, and passing --distribution deb/rpm
• The assemble_workflow will call packaging with all the parameters defined in Usage
• packaging will call FPM with all the parameters and generate packages
• Packages will be copied to opensearch-build/dist folder after creation
• The package can now be put into several repositories for customers to consume

Packaging with FPM

As of 2021/11, there are three tools that we can use to generate deb/rpm packages from Min+plugins build binaries. They are gradle with ospackage, Python Setuptools, and FPM. We choose to use FPM as it is still the easiest, most documented, most used tool with the largest package output range support.

Prerequisites

Please make sure that the version of Ruby ~= 2.3.0 and the FPM gem is installed.

Usage

A long command for FPM is called in the assemble workflow including all the details such as service scripts, pre/post scripts, folder structure, architecture, type, and more for fine-grain setup of generating our project DEB/RPM packages.

The inputs of FPM are:

| name               | description                                                             |
|--------------------|-------------------------------------------------------------------------|
| --force            | Force output even if it will overwrite an existing file.                |
| --verbose          | Enable verbose output.                                                  |
| --input-type       | The package type to use as input.                                       |
| --package          | The package file path to output.                                        |
| --output-type      | Type of distribution to build.                                          |
| --name             | Product name (ex: opensearch/opensearch-dashboards).                    |
| --description      | Add a description for this package.                                     |
| --version          | The version to give to the package.                                     |
| --url              | Add a url for this package..                                            |
| --vendor           | Vendor name for this package which is OpenSearch.                       |
| --maintainer       | The maintainer of this package which is OpenSearch.                     |
| --license          | ASL 2.0.                                                                |
| --before-install   | A script to be run before package installation.                         |
| --before-remove    | A script to be run before package removal.                              |
| --after-install    | A script to be run after package installation.                          |
| --after-remove     | A script to be run after package removal.                               |
| --config-files     | Mark a file in the package as being a config file.                      |
| --template-value   | Make 'key' available in script templates.                               |
| --exclude          | Exclude paths matching pattern (shell wildcard globs valid here).       |
| --architecture     | The architecture name. Can be `x86_64` or `aarch64`.                    |                         |

20220204 Update: We have switched from FPM to Rpmbuild for the final implementations, you can see the comments in this post for details: #1545

Assemble a RPM/DEB distribution

Currently, we have an existing assemble workflow implemented with Python. It takes the output from the build step, installs plugins, and assembles a full Distribution/Bundle into a dist folder.

If you have not run the build_workflow yet, here is the command to generate MIN Tarball + Plugin Zips:

# OpenSearch
./build.sh manifests/<version>/opensearch-<version>.yml --distribution rpm --platform linux

# OpenSearch-Dashboards
./build.sh manifests/<version>/opensearch-dashboards-<version>.yml --distribution rpm --platform linux

Once the build_workflow is completed, you can then run the assemble_workflow. A new argument --distribution is introduced to the assemble workflow, so that the distribution can be selected. For now, the input of --distribution can be rpm , tar or zip . The default value is tar.

# OpenSearch
./assemble.sh builds/opensearch/manifest.yml --distribution rpm --platform linux

# OpenSearch-Dashboards
./assemble.sh builds/opensearch-dashboards/manifest.yml --distribution rpm --platform linux

You can then find the generated packages in dist folder of opensearch-build repo:

-rw-rw-r-- 1 opensearch opensearch 2742 Dec 7 23:46 manifest.yml
-rw-rw-r-- 1 opensearch opensearch 447811069 Dec 7 01:01 opensearch-1.2.0-linux-x64.rpm
-rw-rw-r-- 1 opensearch opensearch 213067614 Dec 7 23:46 opensearch-dashboards-1.2.0-linux-x64.rpm

Installation and Usages

Once you have the deb/rpm package, you can install and run the OpenSearch cluster on any Debian/RHEL flavored Linux Distro. Besides, systemd scripts are provided, so that process manager can directly manage the lifecycle of the cluster.

To install deb/rpm:

# Install DEB
sudo dpkg -i opensearch-<version>-linux-x64.deb
sudo dpkg -i opensearch-dashboards-<version>-linux-x64.deb

# Install RPM
sudo rpm -ivh opensearch-<version>-linux-x64.rpm
sudo rpm -ivh opensearch-dashboards-<version>-linux-x64.rpm

To start service:

#start service
sudo systemctl start opensearch
sudo systemctl start opensearch-dashboards

#check status
sudo systemctl status opensearch
sudo systemctl status opensearch-dashboards

#stop service
sudo systemctl stop opensearch
sudo systemctl stop opensearch-dashboards

Once the cluster is up, you can then use the API to validate the state, and start to index your data:
https://opensearch.org/docs/latest/opensearch/popular-api/

Frequently Asked Questions

1. What are the prerequisites to deploy OpenSearch Clusters alongside OpenSearch-Dashboards with DEB/RPM?
   1. You need to have a host of LINUX OS from the compatibility list here.
      2. We will include runtime dependencies such as JAVA and NODEJS so user does not need to install them manually like in ODFE. (Outdated as of 2022/01/24, see Requirements above for information related to adding these as dependencies)
2. What are the steps to deploy OpenSearch Clusters and OpenSearch-Dashboards with DEB/RPM?
   1. Run these commands:
      1. If you install by using package manager directly (Future Steps):
         1. APT: apt install opensearch`` opensearch_dashboards
         2. YUM: yum install opensearch opensearch_dashboards
      2. If you install by downloading the packages from our website:
         1. DEB: dpkg -i opensearch-<VERSION>-linux-<ARCH>.deb opensearch-dashboards-<VERSION>-linux-<ARCH>.deb
         2. RPM: rpm -ivh opensearch-<VERSION>-linux-<ARCH>.rpm opensearch-dashboards-<VERSION>-linux-<ARCH>.rpm
3. Do you support other OS that are not mentioned in this README?
   1. No, we will only support the Linux OS for DEB/RPM on the compatibility list of our website.
4. Why are we not using the existing functionalities within OpenSearch Min code to generate DEB/RPM?
   1. Because those functionalities will only include the Min binary, and lack of all the plugins.
5. What will happen if I do not specify any --distribution when running the assemble_workflow
   1. --distribution is an optional argument exposed to assemble.sh . argparse module helped us to define the argument and automatically generates help and usage messages and issues errors when users give the program invalid arguments. The default is the keyword we used to specify what value should be used when the --distributio is not present. We set default="tar", which means that if --distributio is not present, the workflow will assemble tarball.
6. What should I do if I’d like to have the assembled Distribution/Bundle stored in other directory instead of the root folder?
   1. You can change the output directory in the dist.py file. For now, we use the os.getcwd() to get the current root folder. You can change it to any other directory based on your requirement. The path will be passed into the packaging process.
7. How do I contribute code to the workflow?
   1. We welcome your contribution and recommend you to read the CONTRIBUTING.md as well as the DEVELOPER_GUIDE.md before getting started. You can also create issues here if you have any questions and bug reports. Thanks.

[jimmyjones2]

Will the RPMs be signed, like the ODFE ones were?

[dblock]

Will the RPMs be signed, like the ODFE ones were?

We generally sign everything we release, so let's make sure they are.

[justchris1]

Will the RPMs be signed, like the ODFE ones were?

... and will be they signed with a strong algorithm like SHA256? (Suggestion: they should).

If it isn't it will prevents the use of the rpms / yum repo for install on Redhat / Centos 8 (replacements) when the machine is in FIPS mode, which is quite important for a segment of the user population and not something that can be resolved by 'turning it off'. Without SHA256, the rpm's won't install and there is no way of overriding on the dnf/yum command.

Also, what repo will be they hosted out of? Will the repo have its metadata signed as well?

[dblock]

@peterzhuamazon Do we have documentation on what the signer process produces, keys, etc.? If not let's make sure to add this to this repo.

[jcgraybill]

Please fix the URL on this line - it goes to a document that most of us don't have access to:

You need to have a host of LINUX OS from the compatibility list here.

I'm not following the decision to include copies of the JDK and Node.JS runtime in the deb and rpm packages. You say it's so a user "does not need to install them manually", but the package manager would do this. I'd consider this a strength of package managers, not a burden.

[justchris1]

I'm not following the decision to include copies of the JDK and Node.JS runtime in the deb and rpm packages. You say it's so a user "does not need to install them manually", but the package manager would do this. I'd consider this a strength of package managers, not a burden.

I'm a little confused by that as well. That is why I asked about what repo they will be serving it out of. I fear they will just post a bare RPM without dependencies or meant to fit into the standard package systems. Although, to be honest, I am not holding my breath as I starting asking some of these questions in June of last year (see #27), but no one ever answered. This project hasn't been the easiest to communicate with or the most transparent, unfortunately.

[dblock]

I'm not following the decision to include copies of the JDK and Node.JS runtime in the deb and rpm packages. You say it's so a user "does not need to install them manually", but the package manager would do this. I'd consider this a strength of package managers, not a burden.

I'm a little confused by that as well. That is why I asked about what repo they will be serving it out of. I fear they will just post a bare RPM without dependencies or meant to fit into the standard package systems.

This is the right time to ask ourselves what we want and clarify anything that's unclear. What should we be doing? Is there anything in #1443 that is not doing the right thing already (please CR)?

Although, to be honest, I am not holding my breath as I starting asking some of these questions in June of last year (see #27), but no one ever answered. This project hasn't been the easiest to communicate with or the most transparent, unfortunately.

Nobody was working on RPM then, but we should have been able to answer signing questions in general, apologies. I opened #1502 to document what we do now, which should carry into RPM signing.

[dblock]

Btw, I don't know how to answer "what repo they will be serving it out of". Does anyone? @bbarani or @peternied?

[jcgraybill]

The goals should be to make it easy for anybody who wants to directly download an rpm/deb package and install it, as well as to make it easy for the maintainers of repos to include OpenSearch deb/rpm packages in their repos. I wonder if the assumption that a single file meets all these needs is wrong. A couple of questions:

1. Should we assemble multiple flavors of RPM/DEB files oriented toward each of these uses? Ones intended for consumption by other software repos would ideally use those repos for managing runtime dependencies. However, folks who want an all-in-one package (similar to the .tgz distribution, but in RPM form) might want to get that from the opensearch.org website.
2. Should we provide support/instructions for people who want to add opensearch.org as one of their trusted repos, so they can use rpm/etc directly to pull & update OpenSearch (rather than downloading a file locally and installing from it.)

[justchris1]

This is the right time to ask ourselves what we want and clarify anything that's unclear. What should we be doing? Is there anything in #1443 that is not doing the right thing already (please CR)?

I am not going to pretend to be an expert on how elasticsearch/opensearch builds. I have some rudimentary background in packaging and understand some of the complexities, but not enough to critique the packaging implementation of the shell scripts, python scripts, etc... I can expand some on the 'what' should happen from my viewpoint, however.

I believe we should strive for something similar in how elasticsearch operated: setup some repo in /etc/yum.repos.d/ (although if this wanted to be some already in place like EPEL in the case of redhat, that would be just fine by me) and run the yum/dnf install command with some list of packages (assumedly, opensearch and opensearch-dashboards as appropriate). The yum/dnf command would then process dependencies, install dependent packages, install the indicated opensearch package, and configure startup scripts (like the systemd service files, any required selinux call setups for the paths, etc...). Successive calls to yum/dnf update should install updates as available on the repo server and reload/restart services as appropriate. Calls specifying specific versions of opensearch (like 1.3) or some relation (like most current version <1.3 to stay on the 1.2 series bugfix) should be supported. Updates which change versions should automatically do any migration needed to support the new versions.

All of the above should obviously work for supported platforms substituting relevant commands (like apt-get for yum/dnf). It appears that by the names of the DEB/RPM it seems like Debian and Red Hat flavor distributions are being explicitly supported, which appears sensible, but we should probably make that clear. Support for Red Hat should also include the restrictions for FIPS, which is officially supported by the Red Hat distribution. Pragmatically, I have seen that means that RPMs should be signed with strong algorithms (such as SHA256 instead of, or at least in addition to MD5) and that the repos serving the RPMs should similarly sign their metadata with SHA256 as well (repomd.xml.asc). These are needed to assure the integrity of the metadata which trigger updates and the updates themselves, the need of which should be readily apparent with supply chain attacks becoming prominent in discussions.

[justchris1]

Please feel free to use this as guidance for contributing on DEB/RPM, as well as raising questions about our approach.

@peterzhuamazon It was stated that this would be used for raising questions about the approach. While there appear to be a number of questions raised, it is a bit light on answers. Do you think some will be able to be given?

[jimmyjones2]

Also, what repo will be they hosted out of? Will the repo have its metadata signed as well?

@justchris1 The ODFE repo has its metadata signed...

[justchris1]

Also, what repo will be they hosted out of? Will the repo have its metadata signed as well?

@justchris1 The ODFE repo has its metadata signed...

Is that the repo they are using? There had been a lot of talk of the forums of getting rid of the ODFE nomenclature, so it is unclear if they would use that repo or not. Of course, RPM had been a 1.0 GA feature that kept getting pushed and pushed through time. It wouldn't shock me that since no one seems to have any answers that this just keeps getting pushed to the right. I imagine this isn't of much value to Amazon. Others, like @dblock have also asked, but crickets continue to be the predominant sound heard.

[peterzhuamazon]

Hi All,

We realized there were missing pieces in the initial proposal, which is why we appreciate your feedbacks to the post.
We have since added a lot more details in the requirement section of the original post.

If you search for Requirements in the post, you will find 6 new milestones we created with more details on all the aspects of the RPM package that we need to consider.

Note: The PR associated with initial design is more of a POC code and not ready for production consumption. We will making multiple changes to the RPM distribution generation code corresponding to the milestones listed on the parent issue to meet the quality bar of the RPM artifact.

Please let us know about your thoughts after reading through our new requirements.

Thanks.

@jimmyjones2 @justchris1

[peterzhuamazon]

@jcgraybill @dblock please also take a look in the Requirement section.
We will start to create issues and tasks in this repo and link to the meta issue once we finalize all the details.
Thanks.

[jcgraybill]

Thanks @peterzhuamazon for providing an update with answers to everything that's been provided here. Those requirements and user experience details look good to me.

[opoplawski]

Just one comment - no one at this point should ever install packages with "rpm" directly (and -ivh is particularly bad), but always use "yum" or "dnf". You can install a locally downloaded rpm by specifying the filename, e.g. yum install opensearch-1.3.0-1.x86_64.rpm

[peterzhuamazon]

Close this as our design is already done without any changes.
The artifacts of rpm on 2.0.0-rc1 and 1.3.2 have been released now.
Thanks.

[patcable]

Excited to see RPMs with the latest release! Is there a place where progress towards packaging deb files is tracked?

[peterzhuamazon]

Excited to see RPMs with the latest release! Is there a place where progress towards packaging deb files is tracked?

DEB is out with the latest 2.5.0 release.
https://opensearch.org/downloads.html

Thanks.