Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

check timestamps before waiting for agent rollout (backport #5698) #5705

Merged
merged 1 commit into from
Apr 30, 2024

Conversation

mergify[bot]
Copy link

@mergify mergify bot commented Apr 29, 2024

IMPORTANT: Please do not create a Pull Request without creating an issue first.

Problem:

PR attempts to address random upgrade fails where the upgrade fails due to the following error:

Wait for cluster settling down...
CAPI cluster fleet-local/local is provisioned (current generation: 838).
cluster.fleet.cattle.io/local patched
Error from server (NotFound): deployments.apps "fleet-agent" not found

This is caused by a delay in rollout of fleet-agent, where the rollout happens when the check for fleet-agent rollout status is running, and deployment gets recreated.

Solution:

The fix attempts to use original fleet-agent creation timestamp before fleet cluster object is patched, and ensures new fleet-agent deployment has a newer creation timestamp before rollout status is checked.

Related Issue:

Test plan:


This is an automatic backport of pull request #5698 done by [Mergify](https://mergify.com).

@mergify mergify bot added the conflicts label Apr 29, 2024
Copy link
Author

mergify bot commented Apr 29, 2024

Cherry-pick of cc1c2ad has failed:

On branch mergify/bp/v1.2/pr-5698
Your branch is up to date with 'origin/v1.2'.

You are currently cherry-picking commit cc1c2adb.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   package/upgrade/upgrade_manifests.sh

no changes added to commit (use "git add" and/or "git commit -a")

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

(cherry picked from commit cc1c2ad)

fixed rebase conflicts
@starbops starbops merged commit b121a2a into v1.2 Apr 30, 2024
4 checks passed
@mergify mergify bot deleted the mergify/bp/v1.2/pr-5698 branch April 30, 2024 02:40
@w13915984028
Copy link
Member

Hit issue #5712 (comment), and need a further fix.

@@ -1367,6 +1369,23 @@ rules:
EOF
}

fleet_agent_timestamp(){
wait_for_deployment cattle-fleet-local-system fleet-agent &> /dev/null
time=$(kubectl get deploy -n cattle-fleet-local-system fleet-agent -o json | jq -r .metadata.creationTimestamp)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kubectl get deploy may fail because the above wait_for_deployment just success on the old deployment

with a fix on #5732

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants