Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

salt: 2019.2.0 -> 3000.2 #85561

Merged
merged 1 commit into from May 3, 2020
Merged

salt: 2019.2.0 -> 3000.2 #85561

merged 1 commit into from May 3, 2020

Conversation

@Flakebi
Copy link
Member

Flakebi commented Apr 19, 2020

Motivation for this change

salt-minion service failed for me since a while with the following error, so I upgraded salt itself and python from 2 to 3.
The error is gone now and minion and master work as expected.

Process Process-1:
Traceback (most recent call last):
  File "/nix/store/mdnyphb6385xrcx2qw2g43bq81x1p67m-python-2.7.17/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/nix/store/mdnyphb6385xrcx2qw2g43bq81x1p67m-python-2.7.17/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/scripts.py", line 157, in minion_process
    minion.start()
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/cli/daemons.py", line 348, in start
    self.minion.tune_in()
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/minion.py", line 1053, in tune_in
    self._bind()
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/minion.py", line 953, in _bind
    self.event = salt.utils.event.get_event('minion', opts=self.opts, io_loop=self.io_loop)
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/utils/event.py", line 146, in get_event
    raise_errors=raise_errors)
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/utils/event.py", line 274, in __init__
    self.connect_pub()
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/utils/event.py", line 386, in connect_pub
    io_loop=self.io_loop
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/transport/ipc.py", line 264, in __new__
    client.__singleton_init__(io_loop=io_loop, socket_path=socket_path)
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/transport/ipc.py", line 666, in __singleton_init__
    socket_path, io_loop=io_loop)
  File "/nix/store/lxxxwhm55201l8w524bhzqxi3ln364qb-salt-2019.2.0/lib/python2.7/site-packages/salt/transport/ipc.py", line 292, in __singleton_init__
    self.unpacker = msgpack.Unpacker(encoding=encoding)
TypeError: __init__() got an unexpected keyword argument 'encoding'
Exception AttributeError: AttributeError("'IPCMessageSubscriber' object has no attribute '_refcount_lock'",) in <bound method IPCMessageSubscriber.__del__ of <salt.transport.ipc.IPCMessageSubscriber object at 0x7a276f80fed0>> ignored
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Copy link
Contributor

aneeshusa left a comment

Tested this out locally and seems to be working for me! Thanks for this, been waiting a while for Salt to release this update.

pkgs/tools/admin/salt/default.nix Outdated Show resolved Hide resolved
pkgs/tools/admin/salt/default.nix Outdated Show resolved Hide resolved
@Flakebi Flakebi force-pushed the Flakebi:salt branch from a02469c to 42ed66f Apr 21, 2020
@ofborg ofborg bot requested a review from aneeshusa Apr 21, 2020
Copy link
Contributor

drewrisinger left a comment

Diff LGTM, better formatting than before. Minor note on repetition of version in msgpack.

Builds locally with nixpkgs-review
(note: no tests run other than build)

https://github.com/NixOS/nixpkgs/pull/85561
1 package built:
salt
pkgs/tools/admin/salt/default.nix Outdated Show resolved Hide resolved
@Flakebi Flakebi force-pushed the Flakebi:salt branch from 42ed66f to de08daf Apr 29, 2020
@Flakebi Flakebi force-pushed the Flakebi:salt branch from de08daf to 435713e Apr 29, 2020
@Flakebi
Copy link
Member Author

Flakebi commented Apr 29, 2020

Salt 3000.2 and 2019.2.4 are out, fixing two CVEs: https://docs.saltstack.com/en/latest/topics/releases/3000.2.html

@drewrisinger
Copy link
Contributor

drewrisinger commented Apr 29, 2020

In that case, I suggest:

  • this PR should be renamed 2019.2.0 -> 3000.2
  • After this is merged, create 2x new PR with backports to [19.09] and [20.03] (see CONTIBUTING.md)

I'm not sure if the backports should be of 2019.2.4 or 3000.2. I don't know enough about this package to know how much breaking changes there are b/w 2019 & 3000. @Mic92, thoughts?

@Flakebi Flakebi changed the title salt: 2019.2.0 -> 3000.1 salt: 2019.2.0 -> 3000.2 Apr 29, 2020
@Mic92
Copy link
Contributor

Mic92 commented Apr 30, 2020

I would backport 2019.2.4. Salt is non-trivial software.

@mweinelt
Copy link
Member

mweinelt commented May 3, 2020

This is now being exploited in the wild. saltstack/salt#57057

@Flakebi
Copy link
Member Author

Flakebi commented May 3, 2020

Thanks for the reference @mweinelt, I’m lucky that I run salt in a vpn only.

I tried to backport 2019.2.4 but it does work neither for 20.03 nor for 19.09:

builder for '/nix/store/zpn8dbhf03rr94lc0n1lcb6mkr7ydp29-salt-2019.2.4.drv' failed with exit code 1; last 10 log lines:
  Finished executing setuptoolsInstallPhase
  installing
  Executing pipInstallPhase
  /build/salt-2019.2.4/dist /build/salt-2019.2.4
  DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
  Processing ./salt-2019.2.4-py2-none-any.whl
  Requirement already satisfied: futures>=2.0 in /nix/store/04wmd8vzgvm9das2hf5sr2c84k30v29s-python2.7-futures-3.2.0/lib/python2.7/site-packages (from salt==2019.2.4) (3.2.0)
  Collecting PyYAML<5.1 (from salt==2019.2.4)
    ERROR: Could not find a version that satisfies the requirement PyYAML<5.1 (from salt==2019.2.4) (from versions: none)
  ERROR: No matching distribution found for PyYAML<5.1 (from salt==2019.2.4)
cannot build derivation '/nix/store/mj2nlpk5w2026z69c2zmdhgb8661m8j6-salt-pepper-0.7.5.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/h12n1kdfniibcsf6hjv419ncnqry2q1c-env.drv': 2 dependencies couldn't be built
[0 built (1 failed), 128 copied (348.4 MiB), 78.1 MiB DL]
error: build of '/nix/store/h12n1kdfniibcsf6hjv419ncnqry2q1c-env.drv' failed
2 packages failed to build:
pepper salt

3000.2 builds and seems to work fine in 19.09 (didn’t test thoroughly).
Does anyone know if the salt-minion in 20.03 and 19.09 actually works?

@mweinelt
Copy link
Member

mweinelt commented May 3, 2020

Does anyone know if the salt-minion in 20.03 and 19.09 actually works?

Yep, is outdated and seems to work.

❯ git reset --hard release-20.03
HEAD is now at c0137ebba71 Merge pull request #86604 from LibreCybernetics/backport-keybase-upgrades
❯ nix-build -A salt
these paths will be fetched (9.22 MiB download, 47.56 MiB unpacked):
  /nix/store/02ra46sj0kh4alkz76cd3vnfmcdprd5b-salt-2019.2.0
copying path '/nix/store/02ra46sj0kh4alkz76cd3vnfmcdprd5b-salt-2019.2.0' from 'https://cache.nixos.org'...
/nix/store/02ra46sj0kh4alkz76cd3vnfmcdprd5b-salt-2019.2.0
❯ cd result/bin
❯ ls
salt  salt-api  salt-call  salt-cloud  salt-cp  salt-key  salt-master  salt-minion  salt-proxy  salt-run  salt-ssh  salt-syndic  salt-unity  spm
❯ ./salt-minion --version
/nix/store/02ra46sj0kh4alkz76cd3vnfmcdprd5b-salt-2019.2.0/lib/python2.7/site-packages/salt/scripts.py:198: DeprecationWarning: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date.  Salt will drop support for Python 2.7 in the Sodium release or later.
salt-minion 2019.2.0 (Fluorine)

For some reason v2019.2.4 requires an older PyYAML version, ours is at 5.2 already, but I can't see that requirement in their repository.

@drewrisinger
Copy link
Contributor

drewrisinger commented May 3, 2020

@Flakebi
Copy link
Member Author

Flakebi commented May 3, 2020

Interesting that 2019.2.0 works on 20.03, thanks @mweinelt, I thought it might suffer from the same bug as on unstable.

salt 2019.2.4 requires an old PyYaml as specified here.
I made prs with overrides for 19.09 and 20.03 in #86651 and #86653.

@mweinelt
Copy link
Member

mweinelt commented May 3, 2020

LGTM.

❯ nixpkgs-review pr 85561
$ git -c fetch.prune=false fetch --force https://github.com/NixOS/nixpkgs master:refs/nixpkgs-review/0 pull/85561/head:refs/nixpkgs-review/1
remote: Enumerating objects: 44, done.
remote: Counting objects: 100% (44/44), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 53 (delta 34), reused 40 (delta 34), pack-reused 9
Unpacking objects: 100% (53/53), 13.36 KiB | 285.00 KiB/s, done.
From https://github.com/NixOS/nixpkgs
   0dc3e485a11..af62dc53559  master               -> refs/nixpkgs-review/0
   06e55a145b3..435713eac22  refs/pull/85561/head -> refs/nixpkgs-review/1
$ git worktree add /home/hexa/.cache/nixpkgs-review/pr-85561/nixpkgs af62dc53559c419d570df5bf9367a2e5d0c2a1d4
Preparing worktree (detached HEAD af62dc53559)
Updating files: 100% (21630/21630), done.
HEAD is now at af62dc53559 kismet: 2016-07-R1 -> 2020-04-R2
$ git merge --no-commit 435713eac2237abc1708497cf71a621fc49cc5ea
Automatic merge went well; stopped before committing as requested
$ nix build --no-link --keep-going --option build-use-sandbox relaxed -f /home/hexa/.cache/nixpkgs-review/pr-85561/build.nix
[4 built, 13 copied (61.0 MiB), 2.2 MiB DL]
https://github.com/NixOS/nixpkgs/pull/85561
1 package built:
salt

$ nix-shell /home/hexa/.cache/nixpkgs-review/pr-85561/shell.nix
these paths will be fetched (0.05 MiB download, 0.28 MiB unpacked):
  /nix/store/d27ihabal1lgajnjj56i0kg02fnavs5a-bash-interactive-4.4-p23-dev
copying path '/nix/store/d27ihabal1lgajnjj56i0kg02fnavs5a-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...

[nix-shell:~/.cache/nixpkgs-review/pr-85561]$ ./results/salt/bin/
salt                  .salt-call-wrapped    .salt-cp-wrapped      .salt-master-wrapped  .salt-proxy-wrapped   .salt-ssh-wrapped     .salt-unity-wrapped
salt-api              salt-cloud            salt-key              salt-minion           salt-run              salt-syndic           .salt-wrapped
.salt-api-wrapped     .salt-cloud-wrapped   .salt-key-wrapped     .salt-minion-wrapped  .salt-run-wrapped     .salt-syndic-wrapped  spm
salt-call             salt-cp               salt-master           salt-proxy            salt-ssh              salt-unity            .spm-wrapped

[nix-shell:~/.cache/nixpkgs-review/pr-85561]$ ./results/salt/bin/salt-minion --version
/nix/store/bghqfj3cdk2s5qwzvl6kf4sq1vab8id9-salt-3000.2/lib/python3.7/site-packages/salt/ext/tornado/httputil.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  class HTTPHeaders(collections.MutableMapping):
salt-minion 3000.2
    
[nix-shell:~/.cache/nixpkgs-review/pr-85561]$ ./results/salt/bin/salt-master --version
/nix/store/bghqfj3cdk2s5qwzvl6kf4sq1vab8id9-salt-3000.2/lib/python3.7/site-packages/salt/ext/tornado/httputil.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  class HTTPHeaders(collections.MutableMapping):
salt-master 3000.2
@bhipple
Copy link
Contributor

bhipple commented May 3, 2020

Result of nixpkgs-review pr 85561 1

1 package built:
- salt
@bhipple
bhipple approved these changes May 3, 2020
@bhipple bhipple merged commit f2b48ae into NixOS:master May 3, 2020
16 checks passed
16 checks passed
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="435713e"; rev="435713eac2237abc1708497cf71a621fc49cc5ea"; } ./pkgs/t
Details
grahamcofborg-eval-lib-tests nix-build --arg pkgs import ./. {} ./lib/tests/release.nix
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="435713e"; rev="435713eac2237abc1708497cf71a621fc49cc5ea"; } ./nixos/
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="435713e"; rev="435713eac2237abc1708497cf71a621fc49cc5ea"; } ./nixos/
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="435713e"; rev="435713eac2237abc1708497cf71a621fc49cc5ea"; } ./nixos/
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="435713e"; rev="435713eac2237abc1708497cf71a621fc49cc5ea"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="435713e"; rev="435713eac2237abc1708497cf71a621fc49cc5ea"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="435713e"; rev="435713eac2237abc1708497cf71a621fc49cc5ea"; } ./pkgs/t
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
salt, salt.passthru.tests on aarch64-linux Success
Details
salt, salt.passthru.tests on x86_64-linux Success
Details
@Flakebi Flakebi deleted the Flakebi:salt branch May 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants
You can’t perform that action at this time.