Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

salt: Do not restart salt-minion process in background #3059

Merged
merged 1 commit into from
Jan 21, 2021
Merged

salt: Do not restart salt-minion process in background #3059

merged 1 commit into from
Jan 21, 2021

Conversation

TeddyAndrieux
Copy link
Collaborator

@TeddyAndrieux TeddyAndrieux commented Jan 21, 2021 •
edited
Loading

Component:

'salt'

Context:

It happen time to time, especially during upgrade when reconfiguring the salt-minion process on a node, the salt-minion do not restart properly so we end the upgrade with a salt-minion process stopped
With in log An instance already running

Summary:

Time to time when restarting the salt-minion process, restart take a bit
of time (more than 10s) so we have 2 salt command trying to start the
salt-minion process at the same time and sometime it happens that at the
end the salt-minion is not running at all An instance is already running.
Because the salt-call --local service.restart salt-minion first try to
restart the salt-minion process and then service.running seems to
happen when salt-minion process is actually not yet restarted.
To fix do not call the service.restart command in background so that
we wait for this to finish before running the service.running state.
NOTE: That this bg: True is not needed here since anyway in this SLS
we wait for the salt-minion process to restart

@TeddyAndrieux
salt: Do not restart salt-minion process in background
0c3ae0f 
Time to time when restarting the salt-minion process, restart take a bit
of time (more than 10s) so we have 2 salt command trying to start the
salt-minion process at the same time and sometime it happens that at the
end the salt-minion is not running at all `An instance is already running`.
Because the `salt-call --local service.restart salt-minion` first try to
restart the salt-minion process and then `service.running` seems to
happen when salt-minion process is actually not yet restarted.
To fix do not call the `service.restart` command in background so that
we wait for this to finish before running the `service.running` state.
NOTE: That this `bg: True` is not needed here since anyway in this SLS
we wait for the salt-minion process to restart
@TeddyAndrieux TeddyAndrieux added kind:bug Something isn't working topic:flakiness Some test are flaky and cause CI to do transient failing topic:lifecycle Issues related to upgrade or downgrade of MetalK8s complexity:medium Something that requires one or few days to fix labels Jan 21, 2021
@TeddyAndrieux TeddyAndrieux added this to the MetalK8s 2.7.1 milestone Jan 21, 2021
@TeddyAndrieux TeddyAndrieux requested a review from a team January 21, 2021 10:52
@bert-e
Copy link
Contributor

bert-e commented Jan 21, 2021

Hello teddyandrieux,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

@bert-e
Copy link
Contributor

bert-e commented Jan 21, 2021

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

  • development/1.0
  • development/1.1
  • development/1.2
  • development/1.3
  • development/2.0
  • development/2.1
  • development/2.2
  • development/2.3
  • development/2.4
  • development/2.5
  • development/2.6

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

@bert-e
Copy link
Contributor

bert-e commented Jan 21, 2021

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • one peer

Peer approvals must include at least 1 approval from the following list:

@TeddyAndrieux TeddyAndrieux force-pushed the bugfix/salt-upgrade-may-fail-to-restart-minion-process branch from 4e9d64e to 0c3ae0f Compare January 21, 2021 10:52
@bert-e
Copy link
Contributor

bert-e commented Jan 21, 2021

History mismatch

Merge commit #4e9d64e8e14f85ac6055a1d5c1a28e347c847a4b on the integration branch
w/2.8/bugfix/salt-upgrade-may-fail-to-restart-minion-process is merging a branch which is neither the current
branch bugfix/salt-upgrade-may-fail-to-restart-minion-process nor the development branch
development/2.8.

It is likely due to a rebase of the branch bugfix/salt-upgrade-may-fail-to-restart-minion-process and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

@TeddyAndrieux
Copy link
Collaborator Author

/reset

@bert-e
Copy link
Contributor

bert-e commented Jan 21, 2021

Reset complete

I have successfully deleted this pull request's integration branches.

@bert-e
Copy link
Contributor

bert-e commented Jan 21, 2021

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

  • development/1.0
  • development/1.1
  • development/1.2
  • development/1.3
  • development/2.0
  • development/2.1
  • development/2.2
  • development/2.3
  • development/2.4
  • development/2.5
  • development/2.6

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

@bert-e
Copy link
Contributor

bert-e commented Jan 21, 2021

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • one peer

Peer approvals must include at least 1 approval from the following list:

@TeddyAndrieux
Copy link
Collaborator Author

/approve

@bert-e
Copy link
Contributor

bert-e commented Jan 21, 2021

Build failed

The build for commit did not succeed in branch w/2.8/bugfix/salt-upgrade-may-fail-to-restart-minion-process.

The following options are set: approve

@bert-e
Copy link
Contributor

bert-e commented Jan 21, 2021

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

  • ✔️ development/2.7

  • ✔️ development/2.8

The following branches will NOT be impacted:

  • development/1.0
  • development/1.1
  • development/1.2
  • development/1.3
  • development/2.0
  • development/2.1
  • development/2.2
  • development/2.3
  • development/2.4
  • development/2.5
  • development/2.6

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

  • Any commit you add on the source branch will trigger a new cycle after the
    current queue is merged.
  • Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

@bert-e
Copy link
Contributor

bert-e commented Jan 21, 2021

I have successfully merged the changeset of this pull request
into targetted development branches:

  • ✔️ development/2.7

  • ✔️ development/2.8

The following branches have NOT changed:

  • development/1.0
  • development/1.1
  • development/1.2
  • development/1.3
  • development/2.0
  • development/2.1
  • development/2.2
  • development/2.3
  • development/2.4
  • development/2.5
  • development/2.6

Please check the status of the associated issue None.

Goodbye teddyandrieux.

@bert-e bert-e merged commit 0c3ae0f into development/2.7 Jan 21, 2021
@bert-e bert-e deleted the bugfix/salt-upgrade-may-fail-to-restart-minion-process branch January 21, 2021 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexandre-allard alexandre-allard alexandre-allard approved these changes

@gdemonet gdemonet gdemonet approved these changes

Assignees
No one assigned
Labels
complexity:medium Something that requires one or few days to fix kind:bug Something isn't working topic:flakiness Some test are flaky and cause CI to do transient failing topic:lifecycle Issues related to upgrade or downgrade of MetalK8s
Projects
None yet
Milestone
MetalK8s 2.7.1
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@TeddyAndrieux @bert-e @alexandre-allard @gdemonet