Support git and stable salt-bootstrap on Gentoo. by ijansky · Pull Request #1500 · saltstack/salt-bootstrap

ijansky · 2020-09-26T17:47:37Z

What does this PR do?

This PR fixes and improves Salt boostrap on Gentoo. The current implementation is outdated, does not support the git method, and does not have any tests on Gentoo.

What issues does this PR fix or reference?

It fixes #1499.

Previous Behavior

It was not possible to bootstrap Salt using the "git" installation method. See "bootstrap-salt.sh git master" fails on Gentoo #1499.
It was not possible to install a specific stable version - only one (unpredictable) version was supported.
It was not possible to install the 2019.x version.
No tests on Gentoo.

New Behavior

The "git" installation method is fully supported.
It is possible to install any version available in Gentoo portage using the "stable" method - at the moment 2019.2, 3000.3, and 3001.1 versions.
Salt correctly installs on Gentoo running either OpenRC or systemd.
Added tests on Gentoo.

s0undt3ch · 2020-09-27T08:06:39Z

Coyld you please also add Gentoo here?
That will add it to our CI process.

Thanks! By the way, awesome work!

myii · 2020-09-27T09:09:36Z

Previously, we tried to use the bootstrap to build pre-salted images for Gentoo but weren't successful. I've since tried with this PR's branch and I was able to build images (based on gentoo/stage3:systemd) for:

git-master-py3
- Was able to complete the process for this and have it pushed by our salt-image-builder to Docker Hub (currently my fork):
  - https://hub.docker.com/r/myii/salt-master-py3/tags (tag gentoo-stage3-systemd)
stable-3001-py3
stable-3000.3-py3

Thanks for providing this, @ijansky!

ijansky · 2020-09-27T10:02:25Z

Thanks! I am glad to help.

@s0undt3ch PR was updated with the CI configuration. It adds quite a few tests to the pipeline. Is that ok or would a subset be more convenient?

@myii this is the next step. :) I will raise a PR with a Gentoo support to the https://gitlab.com/saltstack-formulas/infrastructure/salt-image-builder repository after this PR gets merged. You can check out the work in my fork https://gitlab.com/ijansky/salt-image-builder/-/tree/gentoo-support

myii · 2020-09-27T10:18:17Z

@myii this is the next step. :) I will raise a PR with a Gentoo support to the https://gitlab.com/saltstack-formulas/infrastructure/salt-image-builder repository after this PR gets merged. You can check out the work in my fork https://gitlab.com/ijansky/salt-image-builder/-/tree/gentoo-support

@ijansky Nice work, it will be good to compare the two efforts. Here's a rough outline of what I've used:

https://gitlab.com/myii/salt-image-builder/-/compare/98e41413608c2e9ec581fb53bb669acb17c81899...b0318db31b3b0949bc9e2b7db09f118b8c897b98
- Ignore the massive changes to .gitlab-ci.yml, since I commented out all of the other builds, so that I could focus on Gentoo.

Note that building stable has worked out much slower and with more of our tests failing. But we can pick up that conversation in the repo itself once the time comes.

ijansky · 2020-09-27T10:51:38Z

+    emerge ${EMERGE_FLAGS} "${@}"; return $?
 }

 __gentoo_pre_dep() {


Several tests failed due to the 20 minutes per build limit. Some time could be saved by using emerge-webrsync instead of emerge --sync, but that may not be sufficient.

Should we add a emerge-webrsync to the kitchen config file for Gentoo?
At least its up to date when the test container is created....

I like the idea of adding emerge-webrsync to the provision_command section, It is in the PR now.

Unfortunately it does not solve the problem. I went through the logs of one of the successful runs and it turned out that kitchen-docker runs emerge --sync when creating the container - see https://github.com/test-kitchen/kitchen-docker/blob/24a54be8dece9d095e0f9285c509def1c186f088/lib/kitchen/docker/helpers/dockerfile_helper.rb#L81-L88

I guess that the solution is to replace emerge --sync by emerge-webrsync in kitchen-docker. I will play with it and open a PR against that code to see if it can be replaced or not.

EDIT: PR to kitchen-docker repository - test-kitchen/kitchen-docker#373

With the changes in test-kitchen/kitchen-docker#373, more Gentoo builds succeed, but some of them still get cancelled. I guess that the only two options are either to skip running Gentoo tests in the pipeline or raise the 20 minutes build limit. What do you think, @s0undt3ch?

We actually need to wait for the kitchen-docker release or install from git on the Gemfile

Hmm, let's bump it to 40 minutes then 😢

I am afraid that 45 minutes is closer to what these builds need.

GIT and Salt >= 3000 ~ 17 minutes

GIT and Salt 2019 ~ 40-45 minute

Stable and Salt >= 3000 ~ 35-40 minutes

Stable and Salt 2019 ~ 4045 minutes

We could drop 2019, since this one takes the most time - it needs to build Python 3.6.

Can't drop 2019, it's still supported...

@s0undt3ch I was told by various SaltStack employees that the support is only until Phase 3 support ends on the following page:

https://www.saltstack.com/product-support-lifecycle/

So it appears that 2019.2 is only supported for two more days. Or is there something I'm misssing?

I suppose the same doesn't apply for the salt-bootstrap, which probably needs to continue until the Extended life support ends.

I made the job timeout configurable with the default set to 20 minutes unless overridden in a per distro map. Gentoo jobs have the timeout set to 45 minutes at the moment.

Good news is that all jobs succeeded and most Gentoo jobs finished in less than 20 minutes with just a few exceptions. When the support for 2019 gets dropped at some point in the future, the timeout could be decreased to around 30 minutes.

Let me know if I should squash all commits into one to keep the history cleaner.

s0undt3ch

This is what I was referring to

s0undt3ch · 2020-09-30T04:38:43Z

Thank You!

ijansky force-pushed the bootstrap-gentoo branch from d4af0c6 to 48c0c86 Compare September 26, 2020 17:50

Support git and stable salt-bootstrap on Gentoo.

7ae1dfc

ijansky force-pushed the bootstrap-gentoo branch from 48c0c86 to 7ae1dfc Compare September 26, 2020 18:06

s0undt3ch approved these changes Sep 27, 2020

View reviewed changes

ijansky added 2 commits September 27, 2020 11:39

Add Gentoo to the CI process.

c59da85

Merge branch 'develop' into bootstrap-gentoo

ea7fa19

ijansky commented Sep 27, 2020

View reviewed changes

s0undt3ch reviewed Sep 27, 2020

View reviewed changes

Comment thread .kitchen.yml

Comment thread .kitchen.yml

ijansky added 5 commits September 27, 2020 15:44

Run emerge-webrsync when provisioning a Gentoo container.

a14e893

Test pipelines with kitchen-docker from its GitHub master.

821cc2a

Install Python 3.6 only when building Salt 2019 (stable).

898f394

Install Python 3.6 only when building Salt 2019 (stable+git).

771b813

Make CI job timeout configurable and raise timeout for Gentoo.

7fa4efb

s0undt3ch merged commit fd57a0a into saltstack:develop Sep 30, 2020

ijansky deleted the bootstrap-gentoo branch September 30, 2020 19:35

Uh oh!

Conversation

ijansky commented Sep 26, 2020

What does this PR do?

What issues does this PR fix or reference?

Previous Behavior

New Behavior

Uh oh!

s0undt3ch commented Sep 27, 2020

Uh oh!

myii commented Sep 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ijansky commented Sep 27, 2020

Uh oh!

myii commented Sep 27, 2020

Uh oh!

ijansky Sep 27, 2020

Choose a reason for hiding this comment

Uh oh!

s0undt3ch Sep 27, 2020

Choose a reason for hiding this comment

Uh oh!

ijansky Sep 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ijansky Sep 28, 2020

Choose a reason for hiding this comment

Uh oh!

s0undt3ch Sep 28, 2020

Choose a reason for hiding this comment

Uh oh!

s0undt3ch Sep 28, 2020

Choose a reason for hiding this comment

Uh oh!

ijansky Sep 28, 2020

Choose a reason for hiding this comment

Uh oh!

s0undt3ch Sep 28, 2020

Choose a reason for hiding this comment

Uh oh!

myii Sep 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ijansky Sep 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

s0undt3ch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

s0undt3ch commented Sep 30, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

myii commented Sep 27, 2020 •

edited

Loading

ijansky Sep 27, 2020 •

edited

Loading

myii Sep 28, 2020 •

edited

Loading

ijansky Sep 29, 2020 •

edited

Loading