Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

werf dismiss stuck after deploy custom chart with minio dependency #60

Open
1 task done
kozharskyad opened this issue Jun 5, 2024 · 1 comment
Open
1 task done

Comments

@kozharskyad
Copy link

kozharskyad commented Jun 5, 2024

Before proceeding

  • I didn't find a similar issue

Version

2.3.3

How to reproduce

  1. Environment:
    Hardware:
    Apple MacBook Pro, M3 Pro, 18 Gb RAM
    Software:
    macOS Sonoma 14.5
$ k3d --version|head -1
k3d version v5.6.3
$ kubectl version|tail -1
Server Version: v1.29.5+k3s1
  1. Add custom hosts record in your OS:
$ echo '127.0.0.1 k3s-default' | sudo tee -a /etc/hosts
  1. Run k3s cluster via k3d:
k3d cluster create --api-port 6550 -p '9080:80@loadbalancer' -p '9443:443@loadbalancer' -a 3 -s 3 -i rancher/k3s:v1.29.5-k3s1 --registry-create k3s-default:0.0.0.0:6000 --k3s-arg '--disable=traefik@server:*'
  1. Unpack werf-stuck-bug.tgz archive and change current working directory to directory that unpacked from archive

  2. Export environment variables:

export WERF_DEV="true"
export WERF_INSECURE_REGISTRY="true"
export WERF_PLATFORM="linux/amd64"
export WERF_REPO="k3s-default:6000/werf-stuck-bug"
export WERF_TELEMETRY="0"
  1. Run werf converge

  2. After converge success, wait for MinIO Tenant is green healthStatus:

kubectl get -n werf-stuck-bug tenant/minio -o jsonpath='{ .status.healthStatus } ' -w

Cltr+C after green status appears.

  1. Run werf dismiss. Adding --debug flag possible, because it not affects result.

Result

$ werf dismiss --debug
146.409375ms Version: v2.3.3
162.783417ms Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 version"
165.75725ms  git version 2.45.2
166.14175ms  Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 --git-dir /Users/a.kozharsky/Develop/comita/werf-stuck-bug/.git      ↵
166.15775ms  rev-parse"
168.896833ms Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 --git-dir /Users/a.kozharsky/Develop/comita/werf-stuck-bug/.git      ↵
168.904458ms rev-parse --git-dir"
171.773667ms /Users/a.kozharsky/Develop/comita/werf-stuck-bug/.git
171.88875ms  Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 show-ref --head"
174.988ms    c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb HEAD
174.995ms    c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb refs/heads/_werf-dev
175.001458ms c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb refs/heads/master
175.494417ms Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 worktree list --porcelain"
179.782292ms worktree /Users/a.kozharsky/Develop/comita/werf-stuck-bug
179.786167ms HEAD c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb
179.79525ms  branch refs/heads/master
179.799708ms 
179.836167ms worktree /Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97a7d1e8a198c1a22c/worktree
179.8455ms   HEAD c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb
179.848375ms branch refs/heads/_werf-dev
179.853667ms 
179.925708ms ┌ Switch work tree /Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97a7d1e8a198c1a22 ...
179.999667ms │ Work tree dir:                                                                                                                                ↵
180.007167ms │ /Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97a7d1e8a198c1a22c/worktree
180.03975ms  │ Commit: c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb
180.092042ms │ Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 checkout --force --detach c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb"
189.729542ms │ HEAD сейчас на c2438c4 Initial commit
190.237333ms │ Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 reset --hard c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb"
195.869958ms │ Указатель HEAD сейчас на коммите c2438c4 Initial commit
196.087417ms │ Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 --work-tree                                                        ↵
196.094667ms │ /Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97a7d1e8a198c1a22c/worktree clean    ↵
196.110792ms │ -d -f -f -x"
200.376ms    │ ┌ Sync submodules in work tree "/Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97 ...
200.441417ms │ │ Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 submodule sync --recursive"
239.399625ms │ └ Sync submodules in work tree "/Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713 ... (0.04 seconds)
239.44475ms  │ 
239.450208ms │ ┌ Update submodules in work tree "/Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b ...
239.529708ms │ │ Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 submodule update --checkout --force --init --recursive"
277.515958ms │ └ Update submodules in work tree "/Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b7 ... (0.04 seconds)
277.972417ms │ 
277.974417ms │ Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 --work-tree                                                        ↵
277.985958ms │ /Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97a7d1e8a198c1a22c/worktree          ↵
278.003375ms │ submodule foreach --recursive git -c core.autocrlf=false -c gc.auto=0 reset --hard"
315.508458ms │ Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 --work-tree                                                        ↵
315.515792ms │ /Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97a7d1e8a198c1a22c/worktree          ↵
315.533167ms │ submodule foreach --recursive git -c core.autocrlf=false -c gc.auto=0 clean -d -f -f -x"
350.3195ms   └ Switch work tree /Users/a.kozharsky/.werf/local_cache/git_worktrees/9/local/0b9145a4e250807fd6b5911103964b713de1439ed5379b97 ... (0.17 seconds)
350.606375ms 
350.607167ms Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 branch --list _werf-dev"
354.74675ms    _werf-dev
354.895ms    Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 checkout _werf-dev"
359.78125ms  Переключились на ветку «_werf-dev»
360.215708ms Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 merge-base --is-ancestor c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb    ↵
360.235667ms _werf-dev"
364.097917ms Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 rev-parse _werf-dev"
367.441ms    c2438c4ae2ddd9befe4557dbcaec9bc4d36cb7fb
367.621208ms Running command "/opt/homebrew/bin/git -c core.autocrlf=false -c gc.auto=0 --work-tree /Users/a.kozharsky/Develop/comita/werf-stuck-bug add     ↵
367.628125ms --dry-run --ignore-missing --pathspec-from-file=- --pathspec-file-nul"
376.218958ms -- giterminism_manager.NewManager: projectDir="/Users/a.kozharsky/Develop/comita/werf-stuck-bug"                                                ↵
376.231042ms localGitRepo.WorkTreeDir="/Users/a.kozharsky/Develop/comita/werf-stuck-bug"
383.879792ms Using werf config render file: /private/var/folders/k7/l591ptpx0lz3kgd5cvhtcm3c0000gn/T/werf-config-render-653589138
384.933125ms 
384.933625ms Using namespace: werf-stuck-bug
384.948708ms Using release: werf-stuck-bug
426.596708ms uninstall: Deleting werf-stuck-bug
509.210458ms Starting delete for "console" ServiceStarting delete for "operator" ServiceStarting delete for "sts" ServiceStarting delete for "minio-operator" DeploymentStarting delete for "console" DeploymentStarting delete for "console-sa-binding" ClusterRoleBindingStarting delete for "minio-operator-binding" ClusterRoleBindingStarting delete for "console-sa-role" ClusterRoleStarting delete for "minio-operator-role" ClusterRoleStarting delete for "policybindings.sts.min.io" CustomResourceDefinitionStarting delete for "miniojobs.job.min.io" CustomResourceDefinitionStarting delete for "tenants.minio.min.io" CustomResourceDefinitionStarting delete for "console-env" ConfigMapStarting delete for "console-sa-secret" SecretStarting delete for "minio" SecretStarting delete for "minio-config" SecretStarting delete for "console-sa" ServiceAccountStarting delete for "minio-operator" ServiceAccountStarting delete for "minio" TenantWaiting for resources elimination: services/console, services/operator, services/sts, deployments/minio-operator, deployments/console, clus ...

Output stucks in this moment. All resources successfully removed, but Werf not going to stop and continue waiting.

Result stability: 9/10

Expected result

Werf successfully undeploy all resources and exit.

Additional information

Not sure if this may be cause stuck situation:

$ kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n minio
...
NAME                                            HOLDER                           AGE
lease.coordination.k8s.io/minio-operator-lock   minio-operator-9799b649c-4phlp   5m14s
...
@ilya-lesikov
Copy link
Member

Thank you, will be solved as part of #68

@ilya-lesikov ilya-lesikov transferred this issue from werf/werf Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants