Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support git and stable salt-bootstrap on Gentoo. #1500

Merged
merged 8 commits into from Sep 30, 2020

Conversation

ijansky
Copy link
Contributor

@ijansky ijansky commented Sep 26, 2020

What does this PR do?

This PR fixes and improves Salt boostrap on Gentoo. The current implementation is outdated, does not support the git method, and does not have any tests on Gentoo.

What issues does this PR fix or reference?

It fixes #1499.

Previous Behavior

  • It was not possible to bootstrap Salt using the "git" installation method. See "bootstrap-salt.sh git master" fails on Gentoo #1499.
  • It was not possible to install a specific stable version - only one (unpredictable) version was supported.
  • It was not possible to install the 2019.x version.
  • No tests on Gentoo.

New Behavior

  • The "git" installation method is fully supported.
  • It is possible to install any version available in Gentoo portage using the "stable" method - at the moment 2019.2, 3000.3, and 3001.1 versions.
  • Salt correctly installs on Gentoo running either OpenRC or systemd.
  • Added tests on Gentoo.

@s0undt3ch
Copy link
Member

Coyld you please also add Gentoo here?
That will add it to our CI process.

Thanks! By the way, awesome work!

@myii
Copy link
Contributor

myii commented Sep 27, 2020

Previously, we tried to use the bootstrap to build pre-salted images for Gentoo but weren't successful. I've since tried with this PR's branch and I was able to build images (based on gentoo/stage3:systemd) for:

  • git-master-py3
  • stable-3001-py3
  • stable-3000.3-py3

Thanks for providing this, @ijansky!

@ijansky
Copy link
Contributor Author

ijansky commented Sep 27, 2020

Thanks! I am glad to help.

@s0undt3ch PR was updated with the CI configuration. It adds quite a few tests to the pipeline. Is that ok or would a subset be more convenient?

@myii this is the next step. :) I will raise a PR with a Gentoo support to the https://gitlab.com/saltstack-formulas/infrastructure/salt-image-builder repository after this PR gets merged. You can check out the work in my fork https://gitlab.com/ijansky/salt-image-builder/-/tree/gentoo-support

@myii
Copy link
Contributor

myii commented Sep 27, 2020

@myii this is the next step. :) I will raise a PR with a Gentoo support to the https://gitlab.com/saltstack-formulas/infrastructure/salt-image-builder repository after this PR gets merged. You can check out the work in my fork https://gitlab.com/ijansky/salt-image-builder/-/tree/gentoo-support

@ijansky Nice work, it will be good to compare the two efforts. Here's a rough outline of what I've used:

Note that building stable has worked out much slower and with more of our tests failing. But we can pick up that conversation in the repo itself once the time comes.

# emerge currently won't write to files that aren't there, so we need to ensure their presence
touch /etc/portage/package.accept_keywords /etc/portage/package.keywords /etc/portage/package.license /etc/portage/package.unmask /etc/portage/package.use
# shellcheck disable=SC2086
emerge ${EMERGE_FLAGS} "${@}"; return $?
}

__gentoo_pre_dep() {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Several tests failed due to the 20 minutes per build limit. Some time could be saved by using emerge-webrsync instead of emerge --sync, but that may not be sufficient.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add a emerge-webrsync to the kitchen config file for Gentoo?
At least its up to date when the test container is created....

Copy link
Contributor Author

@ijansky ijansky Sep 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea of adding emerge-webrsync to the provision_command section, It is in the PR now.

Unfortunately it does not solve the problem. I went through the logs of one of the successful runs and it turned out that kitchen-docker runs emerge --sync when creating the container - see https://github.com/test-kitchen/kitchen-docker/blob/24a54be8dece9d095e0f9285c509def1c186f088/lib/kitchen/docker/helpers/dockerfile_helper.rb#L81-L88

I guess that the solution is to replace emerge --sync by emerge-webrsync in kitchen-docker. I will play with it and open a PR against that code to see if it can be replaced or not.

EDIT: PR to kitchen-docker repository - test-kitchen/kitchen-docker#373

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the changes in test-kitchen/kitchen-docker#373, more Gentoo builds succeed, but some of them still get cancelled. I guess that the only two options are either to skip running Gentoo tests in the pipeline or raise the 20 minutes build limit. What do you think, @s0undt3ch?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually need to wait for the kitchen-docker release or install from git on the Gemfile

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, let's bump it to 40 minutes then 😢

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am afraid that 45 minutes is closer to what these builds need.

  • GIT and Salt >= 3000 ~ 17 minutes
  • GIT and Salt 2019 ~ 40-45 minute
  • Stable and Salt >= 3000 ~ 35-40 minutes
  • Stable and Salt 2019 ~ 4045 minutes

We could drop 2019, since this one takes the most time - it needs to build Python 3.6.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't drop 2019, it's still supported...

Copy link
Contributor

@myii myii Sep 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@s0undt3ch I was told by various SaltStack employees that the support is only until Phase 3 support ends on the following page:

So it appears that 2019.2 is only supported for two more days. Or is there something I'm misssing?


I suppose the same doesn't apply for the salt-bootstrap, which probably needs to continue until the Extended life support ends.

Copy link
Contributor Author

@ijansky ijansky Sep 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made the job timeout configurable with the default set to 20 minutes unless overridden in a per distro map. Gentoo jobs have the timeout set to 45 minutes at the moment.

Good news is that all jobs succeeded and most Gentoo jobs finished in less than 20 minutes with just a few exceptions. When the support for 2019 gets dropped at some point in the future, the timeout could be decreased to around 30 minutes.

Let me know if I should squash all commits into one to keep the history cleaner.

Copy link
Member

@s0undt3ch s0undt3ch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what I was referring to

.kitchen.yml Show resolved Hide resolved
.kitchen.yml Show resolved Hide resolved
@s0undt3ch s0undt3ch merged commit fd57a0a into saltstack:develop Sep 30, 2020
@s0undt3ch
Copy link
Member

Thank You!

@ijansky ijansky deleted the bootstrap-gentoo branch September 30, 2020 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

"bootstrap-salt.sh git master" fails on Gentoo
3 participants