[RPM M4] Publish YUM repository #1549

peterzhuamazon · 2022-01-25T23:35:42Z

Tasks	Estimate	Status	Notes	Dependencies
Deploy the RPM packages into a local YUM repository on a build system	3	Complete	The RPM package needs to be ready	see #1549 (comment)
The metadata files for the entire repo will be generated and staged in the repo	2	Complete	Need to figure out whether createrepo is the best way to do this	same as above
The build system should try to install the RPM packages through the ~~local~~ remote repo and verify the commands running correctly	2	Complete (We moved the local yum repo to remote staging yum repo)	The YUM repo needs to be ready	same as above
The build system should attempt to start the service of the installed packages and verify the outcome through process manager of the service (see more details in the test requirements)	2	Please track in #1555		see #1549 (comment)
~~Once verified,~~ the entire repo should be uploaded to the dedicated production S3 bucket linked to artfacts.opensearch.org	2	Complete
A repo file is also generated (once every major release) and updated on our website, for user to download and deploy on their host / server	1	Complete
Once deployed, user can then use YUM command to install the latest RPM packages from our repository on S3, with the help of CloudFront cache in the front	2	Complete

We will attempt to have one repo per major version so that:
- If users want to only install 1.x, they install 1.x yum repo file, and yum update will only bring them latest version in 1.x.
- If users want to upgrade to 2.x, they install 2.x yum repo file, and since they already installed 1.x repo file, they can easily yum upgrade to 2.x, then yum downgrade back to their original 1.x version.
- If users want to install 2.x directly without any old versions on their host, they can install 2.x yum repo file and install the latest version in 2.x.
20220414: The above chart is for staging yum repo which we have already implemented. Production repo will be made manually for now and will implement automation soon. See x64 examples:
- https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.0.0-rc1/latest/linux/x64/rpm/dist/opensearch/opensearch-2.0.0-rc1-staging.repo
- https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.0.0-rc1/latest/linux/x64/rpm/dist/opensearch-dashboards/opensearch-dashboards-2.0.0-rc1-staging.repo

peterzhuamazon · 2022-03-23T23:25:44Z

Yum local repo:

[testrepo]
name=testrepo
baseurl=file:///opt/test
enabled=1
gpgcheck=0

sudo yum install opensearch
Loaded plugins: priorities, remove-with-leaves
1167 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package opensearch.x86_64 0:1.3.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================================================================================================================================================================================
 Package                                                                        Arch                                                                       Version                                                                     Repository                                                                      Size
============================================================================================================================================================================================================================================================================================================================
Installing:
 opensearch                                                                     x86_64                                                                     1.3.0-1                                                                     testrepo                                                                     354 M

Transaction Summary
============================================================================================================================================================================================================================================================================================================================
Install  1 Package

Total download size: 354 M
Installed size: 596 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : opensearch-1.3.0-1.x86_64                                                                                                                                                                                                                                                                                1/1
opensearch-security
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Created opensearch demo certificates in /etc/opensearch/certs
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
  Verifying  : opensearch-1.3.0-1.x86_64                                                                                                                                                                                                                                                                                1/1

Installed:
  opensearch.x86_64 0:1.3.0-1

peterzhuamazon · 2022-03-23T23:32:26Z


% sudo systemctl start opensearch.service

(22-03-23 23:30:07) <0> [/opt]
% sudo systemctl status opensearch.service
● opensearch.service - OpenSearch
   Loaded: loaded (/usr/lib/systemd/system/opensearch.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-03-23 23:30:07 UTC; 7s ago
     Docs: https://opensearch.org/
 Main PID: 25282 (java)
    Tasks: 79
   Memory: 1.2G
   CGroup: /system.slice/opensearch.service
           └─25282 /usr/share/opensearch/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMe...

Mar 23 23:29:55 <>.<> systemd[1]: Starting OpenSearch...
Mar 23 23:29:57 <>.<> systemd-entrypoint[25282]: WARNING: A terminally deprecated method in java.lang.System has been called
Mar 23 23:29:57 <>.<> systemd-entrypoint[25282]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-1.3.0.jar)
Mar 23 23:29:57 <>.<> systemd-entrypoint[25282]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Mar 23 23:29:57 <>.<> systemd-entrypoint[25282]: WARNING: System::setSecurityManager will be removed in a future release
Mar 23 23:29:58 <>.<> systemd-entrypoint[25282]: WARNING: A terminally deprecated method in java.lang.System has been called
Mar 23 23:29:58 <>.<> systemd-entrypoint[25282]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-1.3.0.jar)
Mar 23 23:29:58 <>.<> systemd-entrypoint[25282]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Mar 23 23:29:58 <>.<> systemd-entrypoint[25282]: WARNING: System::setSecurityManager will be removed in a future release
Mar 23 23:30:07 <>.<> systemd[1]: Started OpenSearch.

% curl https://localhost:9200 -u admin:admin --insecure
{
  "name" : "<>",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "jVINa77LSieQakeeuQgRKQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.3.0",
    "build_type" : "rpm",
    "build_hash" : "54c7206d8cba6973d19a814f2705e35ae5cc66db",
    "build_date" : "2022-02-19T17:16:28.462002Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

setiah · 2022-03-25T01:49:47Z

This looks good. In addition, we might need to define more nuances around version upgrade for both minor and major version upgrades. For instance, does a yum update also take care of copying data dir/ from existing to the latest installed version? (i believe not!) Same goes for config and certificates. Without data and config, the new version might start fresh and may not even join the cluster. We may need to document the steps users need to take for minor and major version update via yum.

peterzhuamazon · 2022-03-25T02:23:14Z

This looks good. In addition, we might need to define more nuances around version upgrade for both minor and major version upgrades. For instance, does a yum update also take care of copying data dir/ from existing to the latest installed version? (i believe not!) Same goes for config and certificates. Without data and config, the new version might start fresh and may not even join the cluster. We may need to document the steps users need to take for minor and major version update via yum.

I specifically designed that data folder wont get touch during any upgrade, downgrade, installation.
Also setup config so that any configs that diffs between older and newer will retain the original copy, and move new one to something like config.new.

peterzhuamazon · 2022-04-11T21:58:31Z

I have synced up with @tianleh on the current design of staging yum.

Each build with have 4 yum repo + repofile, os_x64, os_arm64, osd_x64, osd_arm64.
No signing for staging for both repo signing and rpm signing.
When testing upgrade, only testing within major version number. We can test 1.2.4 to 1.3.1, 1.3.0 to 1.3.1, no 1.x to 2.x.
When testing upgrade, install all the repo files that related to the versions in the test, say 1.2.4 to 1.3.1, we will install 1.2.4 * 4repofiles, 1.3.1 * 4repofiles, then install 1.2.4 1st, add some shards, then upgrade 1.3.1, verify state.

More to come.

20220411:

install createrepo in docker image for RPM yum repo support #1955

peterzhuamazon · 2022-04-13T19:56:56Z

We are seeing an issue where rpm itself does not support adding - to separate 3 digit version and its qualifier.
http://ftp.rpm.org/max-rpm/ch-rpm-file-format.html

The version number is normally taken verbatim from the package's version. The only restriction placed on the version is that it cannot contain a dash "-".

Support:

2.0.0alpha1
2.0.0.alpha1
2.0.0~alpha1
2.0.0_alpha1

Not Support:

2.0.0-alpha1

This means the even though the rpm file name shows opensearch-2.0.0-alpha1-linux-x64.rpm.
When install you can only use yum install opensearch or yum install opensearch-2.0.0 not yum install opensearch-2.0.0-alpha1.

Test results:
#1960 (comment)

Thanks.

@CEHENKLE @bbarani @tianleh

dblock · 2022-04-13T20:48:01Z

~~What's the issue? What happens when I yum install opensearch-2.0.0-alpha1?~~

So the tool, rpmbuild doesn't agree with semver? semver/semver#145 ugh

peterzhuamazon · 2022-04-13T20:54:51Z

@dblock rpm does not support dash so its build tool rpmbuild also not supporting it.
You can see a full page of people discussing this.
semver/semver#145

Thanks.

peterzhuamazon · 2022-04-13T21:03:18Z

I would say ~ is the closest to - we can have.
If we have 2.0.0-alpha1 on filename and 2.0.0~alpha1 in rpm metadata then these are supported.

yum install opensearch
yum install opensearch-2.0.0~alpha1
yum install "opensearch-*"
yum install "opensearch-2.0.0*"
yum install "opensearch-2.0.0*alpha1"

And yes this one is not supported at all, because rpm does not have qualifier field and treat 2.0.0~alpha1 its entirety as the version number:

yum install "opensearch-2.0.0"
yum install "opensearch-"

If we do not have ~ or + as you see in above link I provided, there can potentially be a sorting issue in yum repo and cause installation not able to find latest rpm in yum

More explanation: semver/semver#145 (comment)

We also need to check more on this as deb apparently support - (?) according to the discussion.
So if we change rpm we probably need to check if later on deb wants to follow the same or not.

Thanks.

dblock · 2022-04-13T21:05:05Z

My initial reaction is to do this.

peterzhuamazon · 2022-04-13T21:07:26Z

My initial reaction is to do this.

We have some replacements here I think:

~
+
.

Do you think . is the best one here?

dblock · 2022-04-13T21:21:09Z

Personally, I like + because currently we have - and + seems mathematically related. Don't have a strong opinion.

peterzhuamazon · 2022-04-13T21:27:18Z

According to the post seems like . is suited for qualifier while + suited for snapshot.
I actually like to combine . and + so rpm can support both qualifier and snapshot.
Thanks.

peterzhuamazon · 2022-04-13T21:47:57Z

[Enhancement] Make input manifest combine version+qualifier like build/bundle manifest or vise versa #1965

CEHENKLE · 2022-04-13T22:11:02Z

Although there are many, this is not among the hills I am willing to die on ;) Probably not petty enough? ;)

If it's possible, and I think it is, we should do the same thing for both opensearch dashboards and opensearch. I like the idea of . for the qualifier while + for snapshot.

Can we do that for both groups?

peterzhuamazon · 2022-04-20T23:31:12Z

We have completed the staging yum repo but production is not ready, track in #2014.

peterzhuamazon mentioned this issue Jan 25, 2022

[Meta] OpenSearch / OpenSearch Dashboards - Support for RPM distribution #27

Closed

22 tasks

peterzhuamazon added the rpm label Jan 31, 2022

peterzhuamazon changed the title ~~Publish YUM repository~~ [RPM M4] Publish YUM repository Feb 21, 2022

peterzhuamazon self-assigned this Mar 23, 2022

peterzhuamazon assigned tianleh and peterzhuamazon and unassigned peterzhuamazon Apr 11, 2022

tianleh mentioned this issue Apr 11, 2022

install createrepo in docker image for RPM yum repo support #1955

Merged

1 task

peterzhuamazon mentioned this issue Apr 13, 2022

Add yum repo for RPM of OS and OSD in staging #1960

Merged

1 task

This was referenced Apr 13, 2022

[Enhancement] Make input manifest combine version+qualifier like build/bundle manifest or vise versa #1965

Closed

Treat dash as dot in rpm assemble #1966

Merged

peterzhuamazon closed this as completed Apr 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RPM M4] Publish YUM repository #1549

[RPM M4] Publish YUM repository #1549

peterzhuamazon commented Jan 25, 2022 •

edited

peterzhuamazon commented Mar 23, 2022 •

edited

peterzhuamazon commented Mar 23, 2022

setiah commented Mar 25, 2022 •

edited

peterzhuamazon commented Mar 25, 2022 •

edited

peterzhuamazon commented Apr 11, 2022 •

edited

peterzhuamazon commented Apr 13, 2022 •

edited

dblock commented Apr 13, 2022 •

edited

peterzhuamazon commented Apr 13, 2022

peterzhuamazon commented Apr 13, 2022

dblock commented Apr 13, 2022

peterzhuamazon commented Apr 13, 2022 •

edited

dblock commented Apr 13, 2022 •

edited

peterzhuamazon commented Apr 13, 2022

peterzhuamazon commented Apr 13, 2022 •

edited

CEHENKLE commented Apr 13, 2022

peterzhuamazon commented Apr 20, 2022 •

edited

[RPM M4] Publish YUM repository #1549

[RPM M4] Publish YUM repository #1549

Comments

peterzhuamazon commented Jan 25, 2022 • edited

peterzhuamazon commented Mar 23, 2022 • edited

peterzhuamazon commented Mar 23, 2022

setiah commented Mar 25, 2022 • edited

peterzhuamazon commented Mar 25, 2022 • edited

peterzhuamazon commented Apr 11, 2022 • edited

peterzhuamazon commented Apr 13, 2022 • edited

dblock commented Apr 13, 2022 • edited

peterzhuamazon commented Apr 13, 2022

peterzhuamazon commented Apr 13, 2022

dblock commented Apr 13, 2022

peterzhuamazon commented Apr 13, 2022 • edited

dblock commented Apr 13, 2022 • edited

peterzhuamazon commented Apr 13, 2022

peterzhuamazon commented Apr 13, 2022 • edited

CEHENKLE commented Apr 13, 2022

peterzhuamazon commented Apr 20, 2022 • edited

peterzhuamazon commented Jan 25, 2022 •

edited

peterzhuamazon commented Mar 23, 2022 •

edited

setiah commented Mar 25, 2022 •

edited

peterzhuamazon commented Mar 25, 2022 •

edited

peterzhuamazon commented Apr 11, 2022 •

edited

peterzhuamazon commented Apr 13, 2022 •

edited

dblock commented Apr 13, 2022 •

edited

peterzhuamazon commented Apr 13, 2022 •

edited

dblock commented Apr 13, 2022 •

edited

peterzhuamazon commented Apr 13, 2022 •

edited

peterzhuamazon commented Apr 20, 2022 •

edited