enter: start default services before executing subcommand #257

rebornplusplus · 2023-07-07T12:35:50Z

The --run option in the enter command should ensure that the default services have all started before executing the subcommand, as specified in the help info.

This, however, does not seem to be the case as the current implementation only sends a request to the API to start the default services and immediately afterwards, executes the subcommand. This can have undesired behaviour. For example, the following fails:

$ pebble enter --run stop srv1
2023-06-01T09:52:52.516Z [pebble] Started daemon.
2023-06-01T09:52:52.546Z [pebble] POST /v1/services 29.059682ms 202
2023-06-01T09:52:52.546Z [pebble] Started default services with change 386.
2023-06-01T09:52:52.565Z [pebble] Service "srv1" starting: <cmd..>
2023-06-01T09:52:52.584Z [pebble] POST /v1/services 37.51026ms 202
error: cannot perform the following tasks:
- Stop service "srv1" (cannot stop service while starting)

It may also happen that the stop subcommand is executed, before the service hasn't even been initiated/started.

$ pebble enter --run stop srv99
2023-06-22T05:46:02.604Z [pebble] Started daemon.
2023-06-22T05:46:02.619Z [pebble] Service "srv55" starting: <cmd..>
2023-06-22T05:46:02.629Z [pebble] POST /v1/services 23.863368ms 202
2023-06-22T05:46:02.630Z [pebble] Started default services with change 9.
2023-06-22T05:46:02.637Z [pebble] Service "srv1" starting: <cmd..>
2023-06-22T05:46:02.646Z [pebble] POST /v1/services 14.761043ms 202
2023-06-22T11:46:02+06:00 INFO Service "srv99" has never been started.
...

This commit should ensure that the default services have all started, before executing the subcommand. It does so by ensuring the "change" made by the "autostart" request is ready, before running the subcommand. The status of the "change" can be "Error" once it's ready; the subcommand will still be executed after.

Add a new option ``hideProgress`` in ``waitMixin`` to hide the progress, while waiting for changes to be ready.

The --run option in the enter command should ensure that the default services have all started before executing the subcommand, as specified in the help info. This, however, does not seem to be the case as the current implementation only sends a request to the API to start the default services and immediately afterwards, executes the subcommand. This can have undesired behavior. For example, the following fails: $ pebble enter --run stop srv1 2023-06-01T09:52:52.516Z [pebble] Started daemon. 2023-06-01T09:52:52.546Z [pebble] POST /v1/services 29.059682ms 202 2023-06-01T09:52:52.546Z [pebble] Started default services with change 386. 2023-06-01T09:52:52.565Z [pebble] Service "srv1" starting: <cmd..> 2023-06-01T09:52:52.584Z [pebble] POST /v1/services 37.51026ms 202 error: cannot perform the following tasks: - Stop service "srv1" (cannot stop service while starting) It may also happen that the stop subcommand is executed, before the service hasn't even been initiated/started. $ pebble enter --run stop srv99 2023-06-22T05:46:02.604Z [pebble] Started daemon. 2023-06-22T05:46:02.619Z [pebble] Service "srv55" starting: <cmd..> 2023-06-22T05:46:02.629Z [pebble] POST /v1/services 23.863368ms 202 2023-06-22T05:46:02.630Z [pebble] Started default services with change 9. 2023-06-22T05:46:02.637Z [pebble] Service "srv1" starting: <cmd..> 2023-06-22T05:46:02.646Z [pebble] POST /v1/services 14.761043ms 202 2023-06-22T11:46:02+06:00 INFO Service "srv99" has never been started. This commit should ensure that the default services have all started, before executing the subcommand. It does so by ensuring the "change" made by the "autostart" request is ready, before running the subcommand. The status of the "change" can be "Error" once it's ready; the subcommand will still be executed after.

internals/cli/cmd_run.go

According to the feedback from @virtustom, log the error messages from ``waitMixin.wait()`` if there are any.

rebornplusplus · 2023-07-10T11:54:01Z

@benhoyt @niemeyer -- will you please take a look at this PR? Thanks. :)

PS. The tests are failing randomly. I would appreciate it very much if any of you could re-trigger the workflow.

niemeyer · 2023-07-10T15:24:06Z

PS. The tests are failing randomly. I would appreciate it very much if any of you could re-trigger the workflow.

I'd appreciate even more if the tests could be fixed so they don't fail randomly.

internals/cli/cmd_run.go

benhoyt · 2023-07-11T01:04:17Z

PS. The tests are failing randomly. I would appreciate it very much if any of you could re-trigger the workflow.

I'd appreciate even more if the tests could be fixed so they don't fail randomly.

Agreed. However, note that Fred Lotter is working on that (these are the servstate failures). So I've manually re-run them for now.

internals/cli/cmd_run.go

Exit with non-zero error code if waiting for the "change" (of default services start) to be ready returns any errors.

benhoyt

Added a comment, but looks reasonable to me. Is it possible to add a test for this?

internals/cli/cmd_run.go

This commit adds a new test which checks that default services start before executing subcommand.

Go 1.14 does not support os.Readfile(), use ioutil.Readfile() instead.

woky

LGTM

internals/cli/cmd_enter_test.go

Instead of writing errors directly to os.Stderr in rcmd.Run, write them to cli.Stderr which eventually points to os.Stderr; but useful for testing purposes.

Test if default service start errors are properly reported and if pebble exits with non-zero error codes. This is only tested for the following command, for now: pebble enter --run [enter-OPTIONS..] <subcommand> [args..]

cjdcordeiro

thanks for fixing this behavior @rebornplusplus. The overall behavioural changes to the enter startup sequence with --run LGTM!

My only comment is about this thread:

Somewhat related: #167 -- I wonder if pebble run should also wait for services to be started successfully before starting up (or if not waiting, exit if they don't start properly). It seems like it'd be better for that to be a hard error. But maybe that's a separate PR/discussion...

Concerning pebble run, I think it's better to exit and report if the services don't start properly. In enter, we need the wait because the subcommand which follows needs to be run after the services have started. That may not be the case for run though.

Also, I am not sure and please correct me if I am wrong, but it seemed to be that if a startup-enabled service failed to start with an error, the services that followed in the queue were not started as well. So I think it's best to report any errors while starting up and exit afterwards.

Afaiu and according to our discussions, 4098a71 makes the Pebble daemon exit if 1 service fails to start correctly. IMO this is NOT the right behaviour, and even if it was, it deserves some discussion. The rationale is that Pebble services can be completely independent (unless one comes after or before another), and thus the failure of one service should not jeopardize the execution of another. I suggest dropping the hard error from that commit (having extra logging is still nice though).

Moreover, --run is supposed to mimic pebble run's behaviour and thus this PR would introduce a discrepancy amongst the two (as alluded by @benhoyt in his comment above).

@benhoyt @rebornplusplus please let me know if you disagree. If so, we should probably have a chat about it. IMO the right approach is to actually address #167.

benhoyt · 2023-07-21T04:31:25Z

@cjdcordeiro Yep, that makes sense and I agree. We should at least have more discussion if we're going to change that behaviour, and (separately) address #167.

This reverts commit 1281c96 and reverts the hard-error behaviour introduced in 4098a71. Pebble should no longer, at least not due to this commit, exit with non-zero error code if waiting for the "change" (of default services start) to be ready returns any errors.

rebornplusplus · 2023-07-24T12:28:36Z

@benhoyt @rebornplusplus please let me know if you disagree. If so, we should probably have a chat about it. IMO the right approach is to actually address #167.

@cjdcordeiro Yep, that makes sense and I agree. We should at least have more discussion if we're going to change that behaviour, and (separately) address #167.

Sounds good, let's discuss that separately. Meanwhile, I have reverted related changes in c01f729.

cjdcordeiro

Thanks for the changes ;)

rebornplusplus · 2024-03-07T11:20:17Z

Hiya, this has been idle for some time. I have updated the PR to be in-line with the master branch. Please take another look at your convenience. Thanks!

internals/cli/cmd_run.go

internals/cli/wait.go

internals/cli/cmd_enter_test.go

benhoyt

One more suggested simplification / improvement. Sorry for the back and forth -- let me know what you think.

internals/cli/cmd_run.go

internals/cli/cmd_enter_test.go

benhoyt

Looks good, thanks. I've tested locally with both pebble run and pebble enter as well.

feat: hide progress of changes to be ready

37e791e

Add a new option ``hideProgress`` in ``waitMixin`` to hide the progress, while waiting for changes to be ready.

rebornplusplus force-pushed the rocks-696/subcommand-waits branch 2 times, most recently from c9e980b to 151bffd Compare July 7, 2023 13:01

rebornplusplus force-pushed the rocks-696/subcommand-waits branch from 151bffd to daeec0e Compare July 10, 2023 08:37

rebornplusplus marked this pull request as ready for review July 10, 2023 08:41

woky suggested changes Jul 10, 2023

View reviewed changes

internals/cli/cmd_run.go Outdated Show resolved Hide resolved

fix(enter): log default services' start errors

67b6eab

According to the feedback from @virtustom, log the error messages from ``waitMixin.wait()`` if there are any.

rebornplusplus requested a review from woky July 10, 2023 10:56

woky approved these changes Jul 10, 2023

View reviewed changes

cjdcordeiro requested review from niemeyer and benhoyt July 10, 2023 11:56

niemeyer requested changes Jul 10, 2023

View reviewed changes

internals/cli/cmd_run.go Show resolved Hide resolved

benhoyt reviewed Jul 11, 2023

View reviewed changes

internals/cli/cmd_run.go Outdated Show resolved Hide resolved

internals/cli/cmd_run.go Show resolved Hide resolved

fix(enter): stop pebble if default services cannot start

4098a71

Exit with non-zero error code if waiting for the "change" (of default services start) to be ready returns any errors.

rebornplusplus requested review from niemeyer and benhoyt July 13, 2023 04:38

benhoyt approved these changes Jul 17, 2023

View reviewed changes

internals/cli/cmd_run.go Show resolved Hide resolved

rebornplusplus added 2 commits July 19, 2023 15:59

fix(enter): add test for subcommand waiting

b38f0d0

This commit adds a new test which checks that default services start before executing subcommand.

fix: os.Readfile is not supported in 1.14

03d5a68

Go 1.14 does not support os.Readfile(), use ioutil.Readfile() instead.

rebornplusplus requested review from woky and benhoyt July 19, 2023 11:14

woky approved these changes Jul 19, 2023

View reviewed changes

woky reviewed Jul 19, 2023

View reviewed changes

internals/cli/cmd_enter_test.go Outdated Show resolved Hide resolved

rebornplusplus added 2 commits July 20, 2023 10:09

fix: write to cli.Stderr instead of os.Stderr

0d01e0c

Instead of writing errors directly to os.Stderr in rcmd.Run, write them to cli.Stderr which eventually points to os.Stderr; but useful for testing purposes.

fix(enter): test default service start error reporting

1281c96

Test if default service start errors are properly reported and if pebble exits with non-zero error codes. This is only tested for the following command, for now: pebble enter --run [enter-OPTIONS..] <subcommand> [args..]

rebornplusplus requested a review from woky July 20, 2023 04:31

woky approved these changes Jul 20, 2023

View reviewed changes

cjdcordeiro suggested changes Jul 20, 2023

View reviewed changes

rebornplusplus requested a review from cjdcordeiro July 24, 2023 12:28

cjdcordeiro approved these changes Jul 24, 2023

View reviewed changes

benhoyt approved these changes Jul 24, 2023

View reviewed changes

Merge branch 'master' into backup/subcommand-waits

236ac73

rebornplusplus requested review from benhoyt and cjdcordeiro March 7, 2024 11:19

benhoyt added the 24.04 Planned for the 24.04 cycle label Mar 13, 2024

benhoyt requested changes Mar 14, 2024

View reviewed changes

rebornplusplus added 4 commits March 19, 2024 16:26

Merge branch 'master' into rocks-696/subcommand-waits

c044a42

fix: revert changes of wait.go

b17082e

test(enter): simplify commands, drop shell usage

9d57ed8

fix: remove extra channel, refactor logs

753eb86

rebornplusplus requested a review from benhoyt March 20, 2024 21:14

benhoyt requested changes Mar 20, 2024

View reviewed changes

internals/cli/cmd_run.go Outdated Show resolved Hide resolved

internals/cli/cmd_run.go Outdated Show resolved Hide resolved

internals/cli/cmd_enter_test.go Show resolved Hide resolved

fix: wait for autostart to be ready either-way

b55bbf8

rebornplusplus requested a review from benhoyt March 21, 2024 05:24

benhoyt approved these changes Mar 21, 2024

View reviewed changes

benhoyt merged commit 2f9fdc1 into canonical:master Mar 21, 2024
15 checks passed

benhoyt mentioned this pull request Mar 21, 2024

Pebble doesn't report errors starting startup-enabled services #167

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enter: start default services before executing subcommand #257

enter: start default services before executing subcommand #257

rebornplusplus commented Jul 7, 2023 •

edited

rebornplusplus commented Jul 10, 2023 •

edited

niemeyer commented Jul 10, 2023

benhoyt commented Jul 11, 2023 •

edited

benhoyt left a comment

woky left a comment

cjdcordeiro left a comment

benhoyt commented Jul 21, 2023

rebornplusplus commented Jul 24, 2023

cjdcordeiro left a comment

rebornplusplus commented Mar 7, 2024

benhoyt left a comment

benhoyt left a comment

enter: start default services before executing subcommand #257

enter: start default services before executing subcommand #257

Conversation

rebornplusplus commented Jul 7, 2023 • edited

rebornplusplus commented Jul 10, 2023 • edited

niemeyer commented Jul 10, 2023

benhoyt commented Jul 11, 2023 • edited

benhoyt left a comment

Choose a reason for hiding this comment

woky left a comment

Choose a reason for hiding this comment

cjdcordeiro left a comment

Choose a reason for hiding this comment

benhoyt commented Jul 21, 2023

rebornplusplus commented Jul 24, 2023

cjdcordeiro left a comment

Choose a reason for hiding this comment

rebornplusplus commented Mar 7, 2024

benhoyt left a comment

Choose a reason for hiding this comment

benhoyt left a comment

Choose a reason for hiding this comment

rebornplusplus commented Jul 7, 2023 •

edited

rebornplusplus commented Jul 10, 2023 •

edited

benhoyt commented Jul 11, 2023 •

edited