Showing progress with 'salt' command

[kfdm]

One very minor advantage Ansible has for new users is that when you run a command, ansible-playbook it gives you progress about what steps are executing.

If you run the command on the minion side with salt-call, you can get some general output by adding -l info though it's a touch noisy if you don't know what you're looking for.

If you add state_events: True to your master configuration, then you can view the general progress by running salt-run state.events though this can also be a touch noisy.

If you run salt ... there is only the final report when everything is finished.

What would it take to be able to more easily watch the progress of a running state? Is this something that would make sense as an extension to the salt command or perhaps building it as a salt-run command would be better.

The local client provides a cmd_async command that returns a jobid which could be used to filter events

• salt/job/<JID>/new
• salt/job/<JID>/ret/<MID>
• salt/job/<JID>/prog/<MID>/<RUN NUM>
  though perhaps there is a slight delay between getting the jid from cmd_async and being able to subscribe to salt.modules.state.event

Are there perhaps any other race conditions that one would need to be aware of?

Copied from: https://groups.google.com/d/msgid/salt-users/afa89988-d358-40de-a451-9417bdb90206%40googlegroups.com.

[garethgreenaway]

@kfdm Thanks for the report. There is a progress out put module that can be used to show some progress during a Salt calls. It requires that the python progressbar library is installed where you're running Salt, eg. on the master.
https://docs.saltstack.com/en/latest/ref/output/all/salt.output.progress.html#module-salt.output.progress

Real time output from the minion is tricky since they don't return their output to the master until the run has completed.

[kfdm]

--progress is one way to show that the command is still running, but I'm curious about what it would take to show more details about what is running. Unlike ansible the commands are not run in sync (some minions will finish before others) but for a proof-of-concept I'm not worried about that.

While it would require state_events: True to be set, my initial brainstorm was something similar to this

salt '*' state.apply
salt/job/<JID>/prog/A/1 file.manage /path/to/foo
salt/job/<JID>/prog/B/1 file.manage /path/to/foo
salt/job/<JID>/prog/A/2 service.running bar
salt/job/<JID>/prog/A/3 service.running baz
salt/job/<JID>/prog/B/2 service.running bar
...
Normal output.highstate

Perhaps I should try a quick implementation as a salt runner to test this idea a bit more.

[kedare]

I confirm this is more than needed.
We have some huge highstate (and on Windows...) that can take between 30 minutes and 1h to apply, it's very frustrating to not have any progress report

[dbernadett]

I have the same need.

[Oloremo]

+1

[adam12b1]

+1! And I would say, even within the current architectural constraints on getting realtime data back from minions, it would still be better than nothing to at least have a "Now running [state] on [target]..." or similar update from the Salt master as it moves through its work... although, hm, I guess that's still currently only known on the minion, eh?

[Mapel88]

+1

[bmaeck]

+1

[pthirumalachar]

+1. + if this could report not only the state transitions, but could also capture the STD outs/warning of the underlying processes being run, is very much needed. And having the ability to list only the delta output would be very helpful

[digitalformula]

+1

[damon-atkins]

The minion is polled to see if the job is running. But it does not return where it is up to. state_events: True is most likely the best way. You can push event onto slack or similar as well.

[digitalformula]

The minion is polled to see if the job is running. But it does not return where it is up to. state_events: True is most likely the best way. You can push event onto slack or similar as well.

Out of interest - how difficult would it be to add some sort of progress bar? I understand that there are reasons why SaltStack doesn't do that now, and that's 100% fine, but if there was a command line switch that showed something as simple as "Running state 3/17", that would be huge.

salt-call state.apply blender.install --progress

My use case is the cloning of source from Github, which then needs to be built (Blender, specifically). I've got it all working without issue but there's zero output until the end, when the state application has either succeeded or failed.

[damon-atkins]

find_job response should lead to how many minions are working on it.

[tdonovic]

+1 States (when im testing) and cant see why the state times out until 20 min later is really slowing things up. Being able to tail the logs to the master that executed the state would make my life incredibly easy

[amunoz951]

+1 - we have salt-calls that can take hours and it's extremely helpful to see stdout in realtime

[BeehiveSystems]

+1 - we have salt-calls that can take hours and it's extremely helpful to see stdout in realtime

Same here, this would be a very useful feature.

[nsu700]

+1 , get used to the ansible progress output, and find it not safe enough to run a salt script without any progress and eventually throw me an error, killing me

[gigi206]

+1

[rawkode]

I'd love it if we could get real-time output when using salt-call --local.

Could this be implemented?

[klelifo]

+1

[damon-atkins]

find_job response should lead to how many minions are working on it. Maybe even return the number of states completed.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[Oloremo]

not stale

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[KevoSoftworks]

+1 We have a number of systems with high resource usage peaks Being able to monitor the progress of applying states would benefit our workflow during these periods.

[theunsa]

+1 fwiw

[hemantaltair]

+1 otherwise it's very difficult to go on a minion and check logs which is not practical when you have 1000 of infrastructure.

[SJay3]

+1 We need a progress for salt

[MalloZup]

+1

[network-shark]

+1

[alexkolomolo]

+999 This is really needed for anyone wanting to use salt seriously to manage their infra.

[amunoz951]

Our organization already abandoned salt in part for this reason. Reliability being the other.

…

On Fri, Dec 11, 2020 at 11:50 AM Kolomolo123 ***@***.***> wrote: +999 This is really needed for anyone wanting to use salt seriously to manage their infra. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#46665 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEXQNJKDDFKFAHTAZX55HM3SUJZZPANCNFSM4EW4AVYQ> .

[dmulyalin]

+1

Thoughts: might be possible to listen to events on master and write salt execution module in a way that logs will be emitted on event bus or fire events explicitly throughout module execution to indicate the progress. At the same time having a runner running on salt master to listen to this events or an output module that will be listening to such an events showing the progress.

IMHO, capability to subscribe to such progress events using Job ID sounds as useful feature as well.

[xyzst]

Would like to add my interest for this feature. I have a long running job on a minion, and I have to manually check the minion itself if the job has stalled or not.

[auphofBSF]

I Have observed in minion tail target-logs -f and journalctl -f rich sources of logging that could be useful in providing a more automated progress feedback., is there any work current or in planning at Saltstack to provide some progress reporting in running a particular salt state

I often find my salt-ssh <target> state.apply <some state> failing with what seems like an ssh timeout. however I have been able to monitor the to see the state.apply is still functioning, utilizing logs and journalctl

The following is a typical response I get , which is not very helpful

# Response from salt-ssh during a long running state
[ERROR   ] JSON Render failed for:
Connection to <target> closed by remote host.
[ERROR   ] Expecting value: line 1 column 1 (char 0)

[sticky-note]

+ 1 to have this kind of real time reporting feature in SaltStack

[raqua]

This is needed!

[Jibun-no-Kage]

THIS IS NEEDED!

[oijkn]

I use Gitlab-ci to deploy projects with Saltstack but effectively the real time events is missing. When the state.apply finished, then the full report appears.

Can you add this feature please?

[damon-atkins]

The first step is the salt-minion to record more information in the job file. I think it would need a SEP.

[damon-atkins]

What information do people want?
X/28 tasks complete
Or
List of all task ID and status e.g. Running/InProgress, Stable/Same/Identical, Changed/Mocked, Skipped, Failed

I do not think it should output the full data, however if it did then the user could control how much is displayed. However it would increase overhead.

[alexkolomolo]

X/28 tasks complete would be a great!!

…

On Sun, Nov 27, 2022, at 2:24 AM, Damon Atkins wrote: What information do people want? X/28 tasks complete Or List of all task ID and status e.g. Running/InProgress, Stable/Same/Identical, Changed/Mocked, Skipped, Failed I do not think it should output the result, it should be suitable for an interface. — Reply to this email directly, view it on GitHub <#46665 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGQGDKBA6NZMAPUZIVJEEP3WKKZ4TANCNFSM4EW4AVYQ>. You are receiving this because you commented.Message ID: ***@***.***>

[oijkn]

What is the X/28 tasks complete?

[floridop]

Hi and thanks for looking into this. It is one of my main reasons to use ansible instead on machines that can be reached directly via ssh.

It would be nice to have something like
machineX/stateY status

At least printed out when states have been completed or failed or anything, however they did

It would of course be nicer to know which task in the state is being processed, but for me is more about debugging in that case.

[winternet-studio]

This is definitely needed. Why aren't the devs picking up on this?

What information do people want? X/28 tasks complete Or List of all task ID and status e.g. Running/InProgress, Stable/Same/Identical, Changed/Mocked, Skipped, Failed

Not jut the number of tasks completed, but like an event log for each task that is being done. Output should include the minion name, the current task ID that is running, the time stamp, the event type (started/finished/failed/etc) and maybe a few other key information that could be available and useful. It should be sort of an abbreviated version of the report that is shown after the command is complete - but should all fit in one line per event. I guess that line could even include "task X of 28" (for the given minion) in abbreviated format "X/28", to show the total progress.

Best would be to see this on the salt master where the command is executed, but even if I would have to run a command on the minions to see this info I would be very happy. Even if we would simply need to just do a tail -f on a log file I would be exhilarated!! Seems like writing to a log file at least would be something very easy to implement!

[raqua]

I hacked the log myself hooking to the events, so this is how it looks. That is what I would like it to look.

13:36:33 Starting Salt...
13:36:33 Started at: Tue May  9 13:36:33 CEST 2023
13:36:33 
13:36:41 Preparing ...
13:36:59    1/396       dev --> Install Nginx
13:36:59    2/396       dev --> Setup Nginx configuration file
13:37:00    3/396       dev --> Setup Exite configuration file for Nginx
13:37:00    4/396       dev --> Setup main.key for Nginx
13:37:00    5/396       dev --> Setup main.crt for Nginx
13:37:01    6/396       dev --> Setup Inbound.xsl for Nginx
13:37:01    7/396       dev --> Setup Outbound.xsl for Nginx
13:37:01    8/396       dev --> Setup inbound-outbound.xsd for Nginx
13:37:01    9/396       dev --> Start Nginx
13:37:02   10/396       dev --> Install Haproxy
13:37:02   11/396       dev --> Setup haproxy configuration file
13:37:02   12/396       dev --> Set up exite.pem certificate for haproxy
13:37:02   13/396       dev --> Ensure Haproxy is running
13:37:02   14/396       dev --> Ensure Haproxy is running
13:37:03   15/396       dev --> Ensure "tree" package is installed
13:37:03   16/396       dev --> Cron job that removes old log files
13:37:03   17/396       dev --> Ensure there is no hourly triggered log rotation
13:37:03   18/396       dev --> Ensure there is no daily triggered log rotation
13:37:03   19/396       dev --> Remove old log rotation for containers logs
13:37:04   20/396       dev --> Add default Linux log rotation caller and logger script
13:37:04   21/396       dev --> Cron job that will trigger default Linux log rotation on exact time
13:37:04   22/396       dev --> Add log rotation caller and logger script
13:37:04   23/396       dev --> Cron job that will trigger log rotation on exact time
13:37:04   24/396       dev --> Install bind
13:37:05   25/396       dev --> Setup Bind configuration file
13:37:05   26/396       dev --> Setup Bind internal overrides zone file
13:37:06   27/396       dev --> Install Java
13:37:06   28/396       dev --> Start docker
13:37:06   29/396       dev --> Installing Docker python module
13:37:06   30/396       dev --> Add startDockers.sh script
13:37:06   31/396       dev --> Ensure Jenkins user exists
.....

My solution has some issues, namely that it only shows what was already finished and not what goes on and that the number of expected events are only guessed based on previous run, which is highly incorrect and only makes sense if one does test run before.