bug: failed to decrypt previous encrypted key, status code 400

[zt4123]

Current Behavior

I deployed apisix and apisix ingress controller on GCP kubernetes cluster. In pod log for apisix-ingress-controller, there are always errors about "failed to create ssl: unexpected status code 400; error message: {"error_msg":"failed to decrypt previous encrypted key"}"

Expected Behavior

No such errors.

Error Logs

2024-03-22T01:54:46+08:00 error apisix/ssl.go:139 failed to create ssl: unexpected status code 400; error message: {"error_msg":"failed to decrypt previous encrypted key"}

2024-03-22T01:54:46+08:00 error apisix/apisix_tls.go:179 failed to sync SSL to APISIX {"error": "unexpected status code 400; error message: {"error_msg":"failed to decrypt previous encrypted key"}\n", "errorCauses": [{"error": "unexpected status code 400"}, {"error": "error message: {"error_msg":"failed to decrypt previous encrypted key"}\n"}], "ssl": {"id":"8db0ab63","snis":["gke-sea1-pragma-dev-apisix-dashboard.concentrix.com"],"cert":"-----BEGIN CERTIFICATE-----\r\nMIIFeTCCBP6gAwIBAgIQDAD9d20jevNIsWSOM3QKtjAKBggqhkjOPQQDAzBWMQsw\r\nCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMTAwLgYDVQQDEydEaWdp\r\nQ2VydCBUTFMgSHlicmlkIEVDQyBTSEEzODQgMjAyMCBDQTEwHhcNMjMwNjA2MDAw\r\nMDAwWhcNMjQwNzA1MjM1OTU5WjBwMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2Fs\r\naWZvcm5pYTEQMA4GA1UEBxMHRnJlbW9udDEfMB0GA1UEChMWQ29uY2VudHJpeCBD\r\nb3Jwb3JhdGlvbjEZMBcGA1UEAwwQKi5jb25jZW50cml4LmNvbTBZMBMGByqGSM49\r\nAgEGCCqGSM49AwEHA0IABLdwc14ZsyTqHeAWrBksbuqqUpOHTNsRl0ZReJvLquVb\r\ndIlxCTDkKWWBCDCe8kC9fsYR5r2vGj3TWXtwJWsPlNKjggOSMIIDjjAfBgNVHSME\r\nGDAWgBQKvAgpF4ylOW16Ds4zxy6z7fvDejAdBgNVHQ4EFgQUcSI3ZsxDkoOxSu16\r\nhTW7tZNMyNIwKwYDVR0RBCQwIoIQKi5jb25jZW50cml4LmNvbYIOY29uY2VudHJp\r\neC5jb20wDgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEF\r\nBQcDAjCBmwYDVR0fBIGTMIGQMEagRKBChkBodHRwOi8vY3JsMy5kaWdpY2VydC5j\r\nb20vRGlnaUNlcnRUTFNIeWJyaWRFQ0NTSEEzODQyMDIwQ0ExLTEuY3JsMEagRKBC\r\nhkBodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vRGlnaUNlcnRUTFNIeWJyaWRFQ0NT\r\nSEEzODQyMDIwQ0ExLTEuY3JsMD4GA1UdIAQ3MDUwMwYGZ4EMAQICMCkwJwYIKwYB\r\nBQUHAgEWG2h0dHA6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzCBhQYIKwYBBQUHAQEE\r\neTB3MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdpY2VydC5jb20wTwYIKwYB\r\nBQUHMAKGQ2h0dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydFRMU0h5\r\nYnJpZEVDQ1NIQTM4NDIwMjBDQTEtMS5jcnQwCQYDVR0TBAIwADCCAX0GCisGAQQB\r\n1nkCBAIEggFtBIIBaQFnAHcA7s3QZNXbGs7FXLedtM0TojKHRny87N7DUUhZRnEf\r\ntZsAAAGIj+50AAAABAMASDBGAiEAiPVe7X9Fgw6x+A5xb+xXKKrxiEHHRMCrsndI\r\nxrpzVUUCIQC+54rTQryylaHCWgDtXap3N0XUYfCmMWzJWrFwCE5KfwB1AEiw42va\r\npkc0D+VqAvqdMOscUgHLVt0sgdm7v6s52IRzAAABiI/udCwAAAQDAEYwRAIgXrXf\r\n+lyTEp+BxDvqSYgOLogRqTwZLjnUl3xpkkhD6dUCIDo7Fgx90AgdYQHGfSyYW5ue\r\nGmnbtn8WWazf6MmX0eaFAHUA2ra/az+1tiKfm8K7XGvocJFxbLtRhIU0vaQ9MEjX\r\n+6sAAAGIj+5z6QAABAMARjBEAiBiQ2aRojIFTGKtEh1LaE7u//XYoW7hPCSsVMKN\r\nhel2WQIgMS+r70gKodGSohlc/zLIArKukObwV2tkmTcXtJdzZigwCgYIKoZIzj0E\r\nAwMDaQAwZgIxAPKpY9qB+WzjowQT+S065L7wuiNgA2y5THh892oVKeMz/UJm94aM\r\nF0AGTRb6wTpVLQIxALQP5QisAeSVfpqWAbKmX6XgxeLn6fKGYg4VgYRDbDPCMSe2\r\nbDsIKBes7Cu1KB4ebQ==\r\n-----END CERTIFICATE-----\r\n","key":"Bag Attributes\r\n Microsoft Local Key set: \r\n localKeyID: 01 00 00 00 \r\n friendlyName: te-2d33dfef-2403-4eb5-9dfb-a25900162c4c\r\n Microsoft CSP Name: Microsoft Software Key Storage Provider\r\nKey Attributes\r\n X509v3 Key Usage: 80 \r\n-----BEGIN PRIVATE KEY-----\r\nMIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgfehuLux8Peq8nE/j\r\nLmmU09MMF8dvKgcPM3ScYxCp1zKhRANCAAS3cHNeGbMk6h3gFqwZLG7qqlKTh0zb\r\nEZdGUXiby6rlW3SJcQkw5CllgQgwnvJAvX7GEea9rxo901l7cCVrD5TS\r\n-----END PRIVATE KEY-----\r\n","status":1,"labels":{"managed-by":"apisix-ingress-controller","meta_secret_name":"concentrix-com","meta_secret_namespace":"ingress-apisix"}}}
2024-03-22T01:54:46+08:00 warn apisix/apisix_tls.go:279 sync ApisixTls failed, will retry {"object": {"Type":4,"Object":{"Key":"ingress-apisix/apisix-dashboard","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":null}, "error": "unexpected status code 400; error message: {"error_msg":"failed to decrypt previous encrypted key"}\n", "errorCauses": [{"error": "unexpected status code 400"}, {"error": "error message: {"error_msg":"failed to decrypt previous encrypted key"}\n"}]}

Steps to Reproduce

1. Deploy apisix and apisix-ingress-controller on GCP k8s by using helm chart apisix "2.6.0" and apisix-ingress-controller "0.14.0"
2. run kubectl logs -f or run kubectl describe apisixtls apisix-admin-api, will see errors.

Environment

• APISIX version (run apisix version):
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):