Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes 1.30 regression: runtime error: invalid memory address or nil pointer dereference on server-side diff #18548

Closed
3 tasks done
luispabon opened this issue Jun 7, 2024 · 3 comments · Fixed by #18840
Labels
bug Something isn't working

Comments

@luispabon
Copy link

luispabon commented Jun 7, 2024

Checklist:

  • I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
  • I've included steps to reproduce the bug.
  • I've pasted the output of argocd version.

Describe the bug

Since upgrading to kubernetes 1.30, when syncing an Application that require server side diffs, like Kyverno (official chart) or MongoDB Percona Operator (official chart), the application remains in UNKNOWN HEAD state and no resources are synced or displayed in the UI.

This works with no issues on kubernetes 1.27. This is happening on a pristine EKS 1.30 environment we're testing.

Cluster cache clearing does not help.

The logs for argo's application controller:

Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 217 [running]:
runtime/debug.Stack()
\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()
\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1475 +0x54
panic({0x36a5000?, 0x71fcb20?})
\t/usr/local/go/src/runtime/panic.go:920 +0x270
k8s.io/apimachinery/pkg/util/managedfields.(*GvkParser).Type(...)
\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/managedfields/gvkparser.go:43
github.com/argoproj/gitops-engine/pkg/diff.removeWebhookMutation(0xc00590e018, 0xc003b8a8a0, 0x0, {0x5106960?, 0x3dc10bf?})
\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240416142647-fbecbb86e412/pkg/diff/diff.go:210 +0xfd
github.com/argoproj/gitops-engine/pkg/diff.serverSideDiff(0xc003b8a5f0, 0xc003b8a8a0, {0xc0094e9898, 0x8, 0x8})
\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240416142647-fbecbb86e412/pkg/diff/diff.go:176 +0x2bd
github.com/argoproj/gitops-engine/pkg/diff.ServerSideDiff(0xc003b8a8a0?, 0xc0094e9898?, {0xc0094e9898?, 0x8?, 0xc00a05ab40?})
\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240416142647-fbecbb86e412/pkg/diff/diff.go:138 +0x2a
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc003a2e788, 0xc003a2e738, {0xc0094e9898, 0x8, 0x8})
\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240416142647-fbecbb86e412/pkg/diff/diff.go:88 +0x1d6
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc009b2e800, 0x2c, 0x410805?}, {0xc009b2e600, 0x2c?, 0x58?}, {0xc0094e9898, 0x8, 0x8})
\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240416142647-fbecbb86e412/pkg/diff/diff.go:814 +0x132
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc000b5be40?, 0xc00738cb00?, 0x1e?}, {0xc007748600?, 0xc0073c6a10?, 0x7?}, {0x511bba0, 0xc0028cc180?})
\t/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:318 +0x74b
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc00029fc00, 0xc005d5f800, 0xc00844a480, {0xc00738a400, 0x2, 0x2}, {0xc000a34540?, 0x2, 0x2}, 0x0, ...)
\t/go/src/github.com/argoproj/argo-cd/controller/state.go:684 +0x3e1f
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc00088efc0)
\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1586 +0x1223
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:824 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:157 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x50c44c0, 0xc0009bc9f0}, 0x1, 0xc000ac4cc0)
\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:158 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x40?, 0x4a2f280?)
\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:135 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x0?, 0x0?)
\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:92 +0x1e
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 99
\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:823 +0x89b

To Reproduce

Expected behavior

Argo syncs normally

Screenshots

image

Version

Tested on 3 versions (no argo cli available)

{
    "Version": "v2.10.1+a79e0ea",
    "BuildDate": "2024-02-14T17:37:43Z",
    "GitCommit": "a79e0eaca415461dc36615470cecc25d6d38cefb",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.3",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.14.0+g3fc9f4b",
    "KubectlVersion": "v0.26.11",
    "JsonnetVersion": "v0.20.0"
}
{
    "Version": "v2.11.2+25f7504",
    "BuildDate": "2024-05-23T13:32:13Z",
    "GitCommit": "25f7504ecc198e7d7fdc055fdb83ae50eee5edd0",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.9",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.14.4+g81c902a",
    "KubectlVersion": "v0.26.11",
    "JsonnetVersion": "v0.20.0"
}
{
    "Version": "v2.11.3+3f344d5",
    "BuildDate": "2024-06-06T08:42:00Z",
    "GitCommit": "3f344d54a4e0bbbb4313e1c19cfe1e544b162598",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.9",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.14.4+g81c902a",
    "KubectlVersion": "v0.26.11",
    "JsonnetVersion": "v0.20.0"
}

Logs

time="2024-06-06T22:16:03Z" level=info msg="Notifying 1 settings subscribers: [0xc000c5c360]"
time="2024-06-06T22:17:53Z" level=info msg="Refreshing app status (normal refresh requested), level (3)" application=argocd/kyverno
time="2024-06-06T22:17:53Z" level=info msg="Refreshing app status (comparison expired, requesting refresh. reconciledAt: 2024-06-06 22:13:14 +0000 UTC, expiry: 3m0s), level (2)" application=argocd/app-of-apps
time="2024-06-06T22:17:53Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: argocd)" application=argocd/app-of-apps
time="2024-06-06T22:17:53Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: kyverno)" application=argocd/kyverno
time="2024-06-06T22:17:54Z" level=info msg="GetRepoObjs stats" application=argocd/app-of-apps build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=806 unmarshal_ms=806 version_ms=0
time="2024-06-06T22:17:54Z" level=info msg="Skipping auto-sync: most recent sync already to ee06bf8a32319b2caace0f540f9bcb7e384a34f3" application=argocd/app-of-apps
time="2024-06-06T22:17:54Z" level=info msg="Updated health status: Progressing -> Healthy" application=app-of-apps dest-namespace=argocd dest-server="https://kubernetes.default.svc" reason=ResourceUpdated spec="{&ApplicationSource{RepoURL:https://my.gitlab/my/repo.git,Path:.,TargetRevision:HEAD,Helm:nil,Kustomize:nil,Directory:&ApplicationSourceDirectory{Recurse:true,Jsonnet:ApplicationSourceJsonnet{ExtVars:[]JsonnetVar{},TLAs:[]JsonnetVar{},Libs:[],},Exclude:,Include:*/**/application.yaml,},Plugin:nil,Chart:,Ref:,} {https://kubernetes.default.svc argocd  false} platform &SyncPolicy{Automated:&SyncPolicyAutomated{Prune:false,SelfHeal:false,AllowEmpty:false,},SyncOptions:[],Retry:nil,ManagedNamespaceMetadata:nil,} [] [] <nil> []}" type=Normal
time="2024-06-06T22:17:54Z" level=info msg="Update successful" application=argocd/app-of-apps
time="2024-06-06T22:17:54Z" level=info msg="Reconciliation completed" application=argocd/app-of-apps dedup_ms=1 dest-name= dest-namespace=argocd dest-server="https://kubernetes.default.svc" diff_ms=31 fields.level=2 git_ms=807 health_ms=1 live_ms=0 patch_ms=27 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=905
time="2024-06-06T22:17:54Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/app-of-apps
time="2024-06-06T22:17:54Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: argocd)" application=argocd/app-of-apps
time="2024-06-06T22:17:54Z" level=info msg="GetRepoObjs stats" application=argocd/app-of-apps build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=26 unmarshal_ms=26 version_ms=0
time="2024-06-06T22:17:54Z" level=info msg="Skipping auto-sync: most recent sync already to ee06bf8a32319b2caace0f540f9bcb7e384a34f3" application=argocd/app-of-apps
time="2024-06-06T22:17:54Z" level=info msg="No status changes. Skipping patch" application=argocd/app-of-apps
time="2024-06-06T22:17:54Z" level=info msg="Reconciliation completed" application=argocd/app-of-apps dedup_ms=0 dest-name= dest-namespace=argocd dest-server="https://kubernetes.default.svc" diff_ms=6 fields.level=1 git_ms=27 health_ms=3 live_ms=1 patch_ms=0 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=65
time="2024-06-06T22:17:55Z" level=info msg="GetRepoObjs stats" application=argocd/kyverno build_options_ms=0 helm_ms=119 plugins_ms=0 repo_ms=0 time_ms=1543 unmarshal_ms=1424 version_ms=0
time="2024-06-06T22:17:55Z" level=info msg="Applying resource ClusterRoleBinding/kyverno in cluster: https://172.16.0.1:443, namespace: " dry-run=server manager=argocd-controller serverSideApply=true serverSideDiff=true
time="2024-06-06T22:17:55Z" level=info msg="Reconciliation completed" application=argocd/kyverno dest-name= dest-namespace=kyverno dest-server="https://kubernetes.default.svc" fields.level=3 patch_ms=0 setop_ms=0 time_ms=1969
time="2024-06-06T22:17:55Z" level=error msg="Recovered from panic: runtime error: invalid memory address or nil pointer dereference\ngoroutine 202 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1466 +0x54\npanic({0x3610e20?, 0x70a6c60?})\n\t/usr/local/go/src/runtime/panic.go:920 +0x270\nk8s.io/apimachinery/pkg/util/managedfields.(*GvkParser).Type(...)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/managedfields/gvkparser.go:43\ngithub.com/argoproj/gitops-engine/pkg/diff.removeWebhookMutation(0xc004ab6018, 0xc0049c8930, 0x0, {0x5011620?, 0x3d291bd?})\n\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:210 +0xfd\ngithub.com/argoproj/gitops-engine/pkg/diff.serverSideDiff(0xc0049c88d0, 0xc0049c8930, {0xc00616b8b0, 0x8, 0x8})\n\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:176 +0x2bd\ngithub.com/argoproj/gitops-engine/pkg/diff.ServerSideDiff(0xc0049c8930?, 0xc00616b8b0?, {0xc00616b8b0?, 0x8?, 0xc002a394a0?})\n\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:138 +0x2a\ngithub.com/argoproj/gitops-engine/pkg/diff.Diff(0xc0049c8228, 0xc0049c8220, {0xc00616b8b0, 0x8, 0x8})\n\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:88 +0x1d6\ngithub.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc00403b400, 0x2c, 0xc0034697f8?}, {0xc00403b200, 0x2c?, 0xc003469860?}, {0xc00616b8b0, 0x8, 0x8})\n\t/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:814 +0x132\ngithub.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc000be5600?, 0xc002f6db80?, 0x1e?}, {0xc0008d8000?, 0xc003027110?, 0x7?}, {0x50256f0, 0xc00618c2c0?})\n\t/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:310 +0x725\ngithub.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc000a44b60, 0xc000cd6400, 0xc0064dab40, {0xc005db37a0, 0x2, 0x2}, {0xc00145c460?, 0x2, 0x2}, 0x0, ...)\n\t/go/src/github.com/argoproj/argo-cd/controller/state.go:649 +0x3dfa\ngithub.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc0009c36c0)\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1577 +0x1223\ngithub.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:815 +0x25\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:157 +0x33\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000b06660?, {0x4fd0200, 0xc000dd40c0}, 0x1, 0xc0000cd4a0)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:158 +0xaf\nk8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000c38fb0?, 0x3b9aca00, 0x0, 0x0?, 0xc000c38f10?)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:135 +0x7f\nk8s.io/apimachinery/pkg/util/wait.Until(0xc000c38fd0?, 0xa77805?, 0xc00042cfc0?)\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:92 +0x1e\ncreated by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 83\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:814 +0x7ce\n"

@luispabon luispabon added the bug Something isn't working label Jun 7, 2024
@luispabon luispabon changed the title Kubernetes 1.30: runtime error: invalid memory address or nil pointer dereference on server-side diff Kubernetes 1.30 regression: runtime error: invalid memory address or nil pointer dereference on server-side diff Jun 7, 2024
@luispabon
Copy link
Author

luispabon commented Jun 7, 2024

Seems similar to #18020 which was addressed #18154 recently, but it either didn't make it to argocd 2.11.3 or this is caused by something else.

But adding IncludeMutationWebhook=true in addition to server side diff to compare-options seems to workaround the problem successfully

@crenshaw-dev
Copy link
Collaborator

crenshaw-dev commented Jun 7, 2024

When gitops engine initializes a cluster cache, it uses k8s libraries to load a GVKParser based on the cluster's OpenAPI v2 doc. In k8s 1.30, the OpenAPI doc lists the same GVK for some resources multiple times (by suffixing a _v2 to the name). The library function that initializes the GVKParser returns an error when it encounters multiple OpenAPI models for the same GVK. gitops engine logs the error and sets the cluster cache's gvkparser field to nil. When server side diff attempts to use the gvkparser, it throws a nil ref error.

The duplicated GVKs in the OpenAPI doc is a problem that k8s has had before. It was fixed a few months ago but seems to have been reintroduced in 1.30.

I'm looking into it more next week.

@pgier
Copy link

pgier commented Jun 7, 2024

I'm not sure if this is the same issue, but I'm seeing something similar. I'm also on AWs EKS but kubernetes version 1.28. This causes the refresh button to spin indefinitely. Tried invalidating the cache, but it didn't help. Only happens with serverside diff set to true.

Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 194 [running]:
runtime/debug.Stack()
    /usr/local/Cellar/go/1.22.3/libexec/src/runtime/debug/stack.go:24 +0x5e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()
    /Users/paulgier/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1485 +0x54
panic({0x38035c0?, 0x7719a60?})
    /usr/local/Cellar/go/1.22.3/libexec/src/runtime/panic.go:770 +0x132
k8s.io/apimachinery/pkg/util/managedfields.(*GvkParser).Type(...)
    /Users/paulgier/go/pkg/mod/k8s.io/apimachinery@v0.29.2/pkg/util/managedfields/gvkparser.go:43
github.com/argoproj/gitops-engine/pkg/diff.removeWebhookMutation(0xc00201cb98, 0xc0021ed458, 0x0, {0x5559be8?, 0x3f35167?})
    /Users/paulgier/go/src/github.com/argoproj/gitops-engine/pkg/diff/diff.go:213 +0xfd
github.com/argoproj/gitops-engine/pkg/diff.serverSideDiff(0xc0021ed3d8, 0xc0021ed458, {0xc00ab71898, 0x8, 0x8})
    /Users/paulgier/go/src/github.com/argoproj/gitops-engine/pkg/diff/diff.go:179 +0x2bd
github.com/argoproj/gitops-engine/pkg/diff.ServerSideDiff(0x55129c0?, 0xc00019e048?, {0xc00ab71898?, 0x2b?, 0xc00ab716c8?})
    /Users/paulgier/go/src/github.com/argoproj/gitops-engine/pkg/diff/diff.go:141 +0x2a
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc0021ec290, 0xc0021ec298, {0xc00ab71898, 0x8, 0x8})
    /Users/paulgier/go/src/github.com/argoproj/gitops-engine/pkg/diff/diff.go:89 +0x319
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc000190d88, 0x57, 0x798e400?}, {0xc000b3db08, 0x57?, 0xc00300df80?}, {0xc00ab71898, 0x8, 0x8})
    /Users/paulgier/go/src/github.com/argoproj/gitops-engine/pkg/diff/diff.go:824 +0x132
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc0009e8848?, 0x3c70f40?, 0x1e?}, {0xc004a96908?, 0xc00ac532d0?, 0x6?}, {0x55705a0, 0xc004ead740})
    /Users/paulgier/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:318 +0x74e
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc0005bcb60, 0xc004d5d408, 0xc000a23688, {0xc00d194a40, 0x1, 0x1}, {0xc00ac53260, 0x1, 0x1}, 0x0, ...)
    /Users/paulgier/go/src/github.com/argoproj/argo-cd/controller/state.go:684 +0x3d3f
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc000cf61c0)
    /Users/paulgier/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1595 +0x1183
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
    /Users/paulgier/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:833 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
    /Users/paulgier/go/pkg/mod/k8s.io/apimachinery@v0.29.2/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001183890, {0x5515ca0, 0xc001191530}, 0x1, 0xc0010e6a20)
    /Users/paulgier/go/pkg/mod/k8s.io/apimachinery@v0.29.2/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc001183890, 0x3b9aca00, 0x0, 0x1, 0xc0010e6a20)
    /Users/paulgier/go/pkg/mod/k8s.io/apimachinery@v0.29.2/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
    /Users/paulgier/go/pkg/mod/k8s.io/apimachinery@v0.29.2/pkg/util/wait/backoff.go:161
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 100
    /Users/paulgier/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:832 +0x865

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants