Upgrade to Salt 3000.2 python2 #2504

TeddyAndrieux · 2020-04-30T12:30:06Z

Component:

'salt'

Context:

Two CVE were found on the Salt side that has a huge security impact, Salt version 3000.2 include these fixes.

https://docs.saltstack.com/en/latest/topics/releases/3000.2.html#salt-3000-2-release-notes
https://labs.f-secure.com/advisories/saltstack-authorization-bypass

Summary:

In order to fix this CVE ASAP do not wait to have a full python3 solution for MetalK8s and use the python2 version of Salt 3000.2.

To support Salt 3000.2:

Handle "non-string" version in our custom package manager Salt module
Apply tojson filter when dumping certs from Salt mine to avoid unicode invalid certificate
Changes mine send and mine functions in pillar to do not use kwargs as it does not work in Salt 3000.2 (3000.1 - Converting module.run mine.send to new module format saltstack/salt#56584)
backport 0dbcbd1 in 2.0
Remove token_type from kubernetes_rbac salt Auth as we cannot provide additional kwargs for auth in Salt 3000.2 (saltstack/salt@3dbe8dc)

Fixes: #650

Package version may not be a string but an int or a float, so just cast the versions to string before comparing them

In salt we push CA content to the mine but in newer Salt version (>= 2019) if we not use the `tojson` filter on it we get unicode lines so on each line of the resulting certificate you will have `u'<content>'` which is not a valid certificate

In Salt '3000.2' we can't provide kwargs for mine call, so we need to use potitional arguments in pillar and for `module.run` with `mine.send` we need to use potitional arguments to provide the mine name, as `func` get renamed to `name` but this arguments is gave to the function behind, which make the mine function call fail. Sees saltstack/salt#56584

bert-e · 2020-04-30T12:30:09Z

Hello teddyandrieux,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2020-04-30T12:30:11Z

Branches have diverged

This pull request's source branch improvement/GH-650-upgrade-to-salt-3000 has diverged from
development/2.6 by more than 50 commits.

To avoid any integration risks, please re-synchronize them using one of the
following solutions:

Merge origin/development/2.6 into improvement/GH-650-upgrade-to-salt-3000
Rebase improvement/GH-650-upgrade-to-salt-3000 onto origin/development/2.6

Note: If you choose to rebase, you may have to ask me to rebuild
integration branches using the reset command.

bert-e · 2020-04-30T12:30:26Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.1/improvement/GH-650-upgrade-to-salt-3000 with contents from improvement/GH-650-upgrade-to-salt-3000
and development/2.1.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.1/improvement/GH-650-upgrade-to-salt-3000 origin/development/2.1
 $ git merge origin/improvement/GH-650-upgrade-to-salt-3000
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.1/improvement/GH-650-upgrade-to-salt-3000

Sometimes we use a complex structure as an interpolated value in an SLS file. Whilst this works, this is merely by accident, and Salt 2019.2 changes the behaviour: complex values would be spliced in using their `unicode` representation, including `u""` markers. As such, use a proper `tojson` filter in the Jinja templates instead. See: #650 See: #650 See: https://docs.saltstack.com/en/latest/topics/releases/2019.2.0.html#non-backward-compatible-change-to-yaml-renderer (cherry picked from commit 0dbcbd1)

bert-e · 2020-04-30T14:55:15Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.4/improvement/GH-650-upgrade-to-salt-3000 with contents from w/2.3/improvement/GH-650-upgrade-to-salt-3000
and development/2.4.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.4/improvement/GH-650-upgrade-to-salt-3000 origin/development/2.4
 $ git merge origin/w/2.3/improvement/GH-650-upgrade-to-salt-3000
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.4/improvement/GH-650-upgrade-to-salt-3000

bert-e · 2020-04-30T15:26:58Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.5/improvement/GH-650-upgrade-to-salt-3000 with contents from w/2.4/improvement/GH-650-upgrade-to-salt-3000
and development/2.5.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.5/improvement/GH-650-upgrade-to-salt-3000 origin/development/2.5
 $ git merge origin/w/2.4/improvement/GH-650-upgrade-to-salt-3000
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.5/improvement/GH-650-upgrade-to-salt-3000

bert-e · 2020-05-04T08:58:08Z

History mismatch

Merge commit #19ef077932b64e9be95ba7496b7905cba27b982a on the integration branch
w/2.1/improvement/GH-650-upgrade-to-salt-3000 is merging a branch which is neither the current
branch improvement/GH-650-upgrade-to-salt-3000 nor the development branch
development/2.1.

It is likely due to a rebase of the branch improvement/GH-650-upgrade-to-salt-3000 and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

TeddyAndrieux · 2020-05-04T08:58:42Z

/reset

bert-e · 2020-05-04T08:58:50Z

Reset complete

I have successfully deleted this pull request's integration branches.

bert-e · 2020-05-04T08:58:56Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.1/improvement/GH-650-upgrade-to-salt-3000 with contents from improvement/GH-650-upgrade-to-salt-3000
and development/2.1.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.1/improvement/GH-650-upgrade-to-salt-3000 origin/development/2.1
 $ git merge origin/improvement/GH-650-upgrade-to-salt-3000
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.1/improvement/GH-650-upgrade-to-salt-3000

Salt auth not longer support additional kwargs for custom auth in Salt 3000.2, so we cannot provide `token_type` instead consider it's a Bearer auth if a `token` is provided and a Basic auth if we have a `password` See: saltstack/salt@3dbe8dc

Change salt-master image to salt 3000.2 and download salt-minion version 3000.2 Fixes: #650

bert-e · 2020-05-04T09:05:27Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.4/improvement/GH-650-upgrade-to-salt-3000 with contents from w/2.3/improvement/GH-650-upgrade-to-salt-3000
and development/2.4.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.4/improvement/GH-650-upgrade-to-salt-3000 origin/development/2.4
 $ git merge origin/w/2.3/improvement/GH-650-upgrade-to-salt-3000
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.4/improvement/GH-650-upgrade-to-salt-3000

bert-e · 2020-05-04T09:25:49Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.5/improvement/GH-650-upgrade-to-salt-3000 with contents from w/2.4/improvement/GH-650-upgrade-to-salt-3000
and development/2.5.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.5/improvement/GH-650-upgrade-to-salt-3000 origin/development/2.5
 $ git merge origin/w/2.4/improvement/GH-650-upgrade-to-salt-3000
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.5/improvement/GH-650-upgrade-to-salt-3000

bert-e · 2020-05-04T09:54:03Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.6/improvement/GH-650-upgrade-to-salt-3000 with contents from w/2.5/improvement/GH-650-upgrade-to-salt-3000
and development/2.6.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.6/improvement/GH-650-upgrade-to-salt-3000 origin/development/2.6
 $ git merge origin/w/2.5/improvement/GH-650-upgrade-to-salt-3000
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.6/improvement/GH-650-upgrade-to-salt-3000

bert-e · 2020-05-04T13:27:43Z

History mismatch

Merge commit #604a2922455cebaaa856ccf2158521dda075da45 on the integration branch
w/2.5/improvement/GH-650-upgrade-to-salt-3000 is merging a branch which is neither the current
branch improvement/GH-650-upgrade-to-salt-3000 nor the development branch
development/2.5.

It is likely due to a rebase of the branch improvement/GH-650-upgrade-to-salt-3000 and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

bert-e · 2020-05-04T13:41:04Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

TeddyAndrieux · 2020-05-04T14:00:26Z

/help

bert-e · 2020-05-04T14:00:28Z

Help page

The following options and commands are available at this time.

Options

name	description	privileged	authored
➡️ bypass_leader_approval	Bypass the pull request leaders' approval	⭐
➡️ create_pull_requests	Allow the creation of integration pull requests.
➡️ unanimity	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
➡️ approve	Instruct Bert-E that the author has approved the pull request.		✍️
➡️ bypass_jira_check	Bypass the Jira issue check	⭐
➡️ wait	Instruct Bert-E not to run until further notice.
➡️ bypass_peer_approval	Bypass the pull request peers' approval	⭐
➡️ bypass_commit_size	Bypass the check on the size of the changeset `TBA`	⭐
➡️ bypass_build_status	Bypass the build and test status	⭐
➡️ after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
➡️ bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
➡️ no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
➡️ bypass_author_approval	Bypass the pull request author's approval	⭐

Commands

name	description	privileged
➡️ force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
➡️ build	Re-start a fresh build `TBA`
➡️ retry	Re-start a fresh build `TBA`
➡️ clear	Remove all comments from Bert-E from the history `TBA`
➡️ help	Print Bert-E's manual in the pull request.
➡️ reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.
➡️ status	Print Bert-E's current status in the pull request `TBA`

bert-e · 2020-05-04T14:00:39Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

TeddyAndrieux · 2020-05-04T14:51:31Z

/create_pull_requests

bert-e · 2020-05-04T14:51:53Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/GH-650-upgrade-to-salt-3000 into
development/2.0
w/2.1/improvement/GH-650-upgrade-to-salt-3000 will be merged into development/2.1 (pull request INTEGRATION [PR#2504 > development/2.1] Upgrade to Salt 3000.2 python2 #2514)
w/2.2/improvement/GH-650-upgrade-to-salt-3000 will be merged into development/2.2 (pull request INTEGRATION [PR#2504 > development/2.2] Upgrade to Salt 3000.2 python2 #2515)
w/2.3/improvement/GH-650-upgrade-to-salt-3000 will be merged into development/2.3 (pull request INTEGRATION [PR#2504 > development/2.3] Upgrade to Salt 3000.2 python2 #2516)
w/2.4/improvement/GH-650-upgrade-to-salt-3000 will be merged into development/2.4 (pull request INTEGRATION [PR#2504 > development/2.4] Upgrade to Salt 3000.2 python2 #2517)
w/2.5/improvement/GH-650-upgrade-to-salt-3000 will be merged into development/2.5 (pull request INTEGRATION [PR#2504 > development/2.5] Upgrade to Salt 3000.2 python2 #2518)
w/2.6/improvement/GH-650-upgrade-to-salt-3000 will be merged into development/2.6 (pull request INTEGRATION [PR#2504 > development/2.6] Upgrade to Salt 3000.2 python2 #2519)

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3

Follow integration pull requests if you would like to be notified of
build statuses by email.

The following options are set: create_pull_requests

bert-e · 2020-05-04T14:51:55Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following options are set: create_pull_requests

gdemonet · 2020-05-05T09:02:42Z

salt/_auth/kubernetes_rbac.py

+            '"password" at the same time'
+        )
+        return False
+    if not token and (not password or not username):


Don't change it, but in the future, you can make this boolean expression more readable with:

if not (token or password and username): # Or with parentheses if not (token or (password and username)):

An underrated pattern:

if not any([ token, username and password, ]): blah()

Like this one, it's even more readable :)

While strictly speaking less efficient (so could make a difference in 'hot' loops and such), any and all often make multiple or'ed or and'ed boolean expressions easier to grasp.

gdemonet

LGTM! Let's fix the upgrade and downgrade issues in a follow-up PR (can you open tickets so we can track them please?)

TeddyAndrieux · 2020-05-05T15:14:36Z

/approve

bert-e · 2020-05-05T15:14:59Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.0
✔️ development/2.1
✔️ development/2.2
✔️ development/2.3
✔️ development/2.4
✔️ development/2.5
✔️ development/2.6

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: create_pull_requests, approve

bert-e · 2020-05-05T16:08:22Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.0
✔️ development/2.1
✔️ development/2.2
✔️ development/2.3
✔️ development/2.4
✔️ development/2.5
✔️ development/2.6

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3

Please check the status of the associated issue GH-650.

Goodbye teddyandrieux.

TeddyAndrieux added 3 commits April 30, 2020 13:48

salt: metalk8s_packages_manager module handle non-string version

4ba21c1

Package version may not be a string but an int or a float, so just cast the versions to string before comparing them

salt: Apply tojson filter on CA advertised

9b69bd3

In salt we push CA content to the mine but in newer Salt version (>= 2019) if we not use the `tojson` filter on it we get unicode lines so on each line of the resulting certificate you will have `u'<content>'` which is not a valid certificate

TeddyAndrieux requested a review from a team April 30, 2020 12:30

TeddyAndrieux changed the base branch from development/2.6 to development/2.0 April 30, 2020 12:30

TeddyAndrieux force-pushed the improvement/GH-650-upgrade-to-salt-3000 branch 2 times, most recently from 203aa01 to b01769f Compare April 30, 2020 14:50

TeddyAndrieux force-pushed the improvement/GH-650-upgrade-to-salt-3000 branch from b01769f to b296660 Compare May 4, 2020 08:58

TeddyAndrieux added 2 commits May 4, 2020 11:02

build: Upgrade Salt to 3000.2

6e9bbfd

Change salt-master image to salt 3000.2 and download salt-minion version 3000.2 Fixes: #650

TeddyAndrieux force-pushed the improvement/GH-650-upgrade-to-salt-3000 branch from b296660 to 6e9bbfd Compare May 4, 2020 09:02

TeddyAndrieux marked this pull request as ready for review May 4, 2020 14:51

gdemonet reviewed May 5, 2020

View reviewed changes

gdemonet approved these changes May 5, 2020

View reviewed changes

bert-e merged commit 6e9bbfd into development/2.0 May 5, 2020

bert-e deleted the improvement/GH-650-upgrade-to-salt-3000 branch May 5, 2020 16:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to Salt 3000.2 python2 #2504

Upgrade to Salt 3000.2 python2 #2504

TeddyAndrieux commented Apr 30, 2020 •

edited

Loading

bert-e commented Apr 30, 2020

bert-e commented Apr 30, 2020

bert-e commented Apr 30, 2020

bert-e commented Apr 30, 2020

bert-e commented Apr 30, 2020

bert-e commented May 4, 2020

TeddyAndrieux commented May 4, 2020

bert-e commented May 4, 2020

bert-e commented May 4, 2020

bert-e commented May 4, 2020

bert-e commented May 4, 2020

bert-e commented May 4, 2020

bert-e commented May 4, 2020

bert-e commented May 4, 2020

TeddyAndrieux commented May 4, 2020

bert-e commented May 4, 2020

bert-e commented May 4, 2020

TeddyAndrieux commented May 4, 2020

bert-e commented May 4, 2020

bert-e commented May 4, 2020

gdemonet May 5, 2020

NicolasT May 5, 2020

gdemonet May 5, 2020

NicolasT May 5, 2020

gdemonet left a comment

TeddyAndrieux commented May 5, 2020

bert-e commented May 5, 2020

bert-e commented May 5, 2020

Upgrade to Salt 3000.2 python2 #2504

Upgrade to Salt 3000.2 python2 #2504

Conversation

TeddyAndrieux commented Apr 30, 2020 • edited Loading

bert-e commented Apr 30, 2020

Hello teddyandrieux,

bert-e commented Apr 30, 2020

Branches have diverged

bert-e commented Apr 30, 2020

Conflict

bert-e commented Apr 30, 2020

Conflict

bert-e commented Apr 30, 2020

Conflict

bert-e commented May 4, 2020

History mismatch

TeddyAndrieux commented May 4, 2020

bert-e commented May 4, 2020

Reset complete

bert-e commented May 4, 2020

Conflict

bert-e commented May 4, 2020

Conflict

bert-e commented May 4, 2020

Conflict

bert-e commented May 4, 2020

Conflict

bert-e commented May 4, 2020

History mismatch

bert-e commented May 4, 2020

Waiting for approval

TeddyAndrieux commented May 4, 2020

bert-e commented May 4, 2020

Help page

Options

Commands

bert-e commented May 4, 2020

Waiting for approval

TeddyAndrieux commented May 4, 2020

bert-e commented May 4, 2020

Integration data created

bert-e commented May 4, 2020

Waiting for approval

gdemonet May 5, 2020

Choose a reason for hiding this comment

NicolasT May 5, 2020

Choose a reason for hiding this comment

gdemonet May 5, 2020

Choose a reason for hiding this comment

NicolasT May 5, 2020

Choose a reason for hiding this comment

gdemonet left a comment

Choose a reason for hiding this comment

TeddyAndrieux commented May 5, 2020

bert-e commented May 5, 2020

In the queue

bert-e commented May 5, 2020

TeddyAndrieux commented Apr 30, 2020 •

edited

Loading