P12 authentication issue using external Elasticsearch

[saowu]

https://skywalking.apache.org/docs/skywalking-swck/v0.9.0/examples/storage/
I refer to the above article and use external Elasticsearch, but the following error appears

apiVersion: v1
kind: Secret
metadata:
  name: skywalking-storage
  namespace: skywalking-swck-system
data:
  password: 123
  storage.p12: >-
    MIIJpgIBAzCCCWwGCSqGSIb3DQEHAaCCCV0EgglZMIIJVTCCA/cGCSqGSIb3DQEHBqCCA
type: Opaque

2024-03-29 09:27:29,821 org.apache.skywalking.oap.server.library.module.BootstrapFlow 43 [main] INFO  [] - start the provider default in prometheus-fetcher module.
2024-03-29 09:27:29,822 org.apache.skywalking.oap.server.library.module.BootstrapFlow 43 [main] INFO  [] - start the provider kubernetes in cluster module.
2024-03-29 09:27:30,063 org.apache.skywalking.oap.server.library.module.BootstrapFlow 43 [main] INFO  [] - start the provider default in event-analyzer module.
2024-03-29 09:27:30,064 org.apache.skywalking.oap.server.library.module.BootstrapFlow 43 [main] INFO  [] - start the provider elasticsearch in storage module.
2024-03-29 09:27:30,066 io.kubernetes.client.informer.cache.Controller 102 [informer-controller-V1Pod] INFO  [] - informer#Controller: ready to run resync & reflector runnable
2024-03-29 09:27:30,066 io.kubernetes.client.informer.cache.Controller 111 [informer-controller-V1Pod] INFO  [] - informer#Controller: resync skipped due to 0 full resync period
2024-03-29 09:27:30,070 io.kubernetes.client.informer.cache.ReflectorRunnable 84 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO  [] - class io.kubernetes.client.openapi.models.V1Pod#Start listing and watching...
2024-03-29 09:27:30,132 org.apache.skywalking.oap.server.starter.OAPServerBootstrap 60 [main] ERROR [] - [9.1.0-f1f519c] keystore password was incorrect
org.apache.skywalking.oap.server.library.module.ModuleStartException: keystore password was incorrect
	at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.StorageModuleElasticsearchProvider.start(StorageModuleElasticsearchProvider.java:236) ~[storage-elasticsearch-plugin-9.1.0.jar:9.1.0]
	at org.apache.skywalking.oap.server.library.module.BootstrapFlow.start(BootstrapFlow.java:46) ~[library-module-9.1.0.jar:9.1.0]
	at org.apache.skywalking.oap.server.library.module.ModuleManager.init(ModuleManager.java:60) ~[library-module-9.1.0.jar:9.1.0]
	at org.apache.skywalking.oap.server.starter.OAPServerBootstrap.start(OAPServerBootstrap.java:44) [server-starter-9.1.0.jar:9.1.0]
	at org.apache.skywalking.oap.server.starter.OAPServerStartUp.main(OAPServerStartUp.java:23) [server-starter-9.1.0.jar:9.1.0]
Caused by: java.io.IOException: keystore password was incorrect
	at sun.security.pkcs12.PKCS12KeyStore.engineLoad(Unknown Source) ~[?:?]
	at sun.security.util.KeyStoreDelegator.engineLoad(Unknown Source) ~[?:?]
	at java.security.KeyStore.load(Unknown Source) ~[?:?]
	at org.apache.skywalking.library.elasticsearch.ElasticSearchBuilder.build(ElasticSearchBuilder.java:167) ~[library-elasticsearch-client-9.1.0.jar:9.1.0]
	at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.connect(ElasticSearchClient.java:149) ~[library-client-9.1.0.jar:9.1.0]
	at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.StorageModuleElasticsearchProvider.start(StorageModuleElasticsearchProvider.java:231) ~[storage-elasticsearch-plugin-9.1.0.jar:9.1.0]
	... 4 more
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
	at sun.security.pkcs12.PKCS12KeyStore.engineLoad(Unknown Source) ~[?:?]
	at sun.security.util.KeyStoreDelegator.engineLoad(Unknown Source) ~[?:?]
	at java.security.KeyStore.load(Unknown Source) ~[?:?]
	at org.apache.skywalking.library.elasticsearch.ElasticSearchBuilder.build(ElasticSearchBuilder.java:167) ~[library-elasticsearch-client-9.1.0.jar:9.1.0]
	at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.connect(ElasticSearchClient.java:149) ~[library-client-9.1.0.jar:9.1.0]
	at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.StorageModuleElasticsearchProvider.start(StorageModuleElasticsearchProvider.java:231) ~[storage-elasticsearch-plugin-9.1.0.jar:9.1.0]
	... 4 more
2024-03-29 09:27:30,144 io.kubernetes.client.informer.cache.Controller 164 [informer-controller-V1Pod] ERROR [] - [9.1.0-f1f519c] DefaultController#processLoop get interrupted null
java.lang.InterruptedException: null
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(Unknown Source) ~[?:?]
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) ~[?:?]
	at io.kubernetes.client.informer.cache.DeltaFIFO.pop(DeltaFIFO.java:318) ~[client-java-14.0.0.jar:?]
	at io.kubernetes.client.informer.cache.Controller.processLoop(Controller.java:162) ~[client-java-14.0.0.jar:?]
	at io.kubernetes.client.informer.cache.Controller.run(Controller.java:130) ~[client-java-14.0.0.jar:?]
	at java.lang.Thread.run(Unknown Source) [?:?]

I guess the p12 certificate key is not changed successfully

2024-03-29 09:44:34,959 org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader 118 [main] INFO  [] - Provider=elasticsearch config=namespace has been set as 
2024-03-29 09:44:34,959 org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader 118 [main] INFO  [] - Provider=elasticsearch config=clusterNodes has been set as 192.168.211.245:9200
2024-03-29 09:44:34,959 org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader 118 [main] INFO  [] - Provider=elasticsearch config=protocol has been set as https
2024-03-29 09:44:34,960 org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader 118 [main] INFO  [] - Provider=elasticsearch config=connectTimeout has been set as 3000
2024-03-29 09:44:34,960 org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader 118 [main] INFO  [] - Provider=elasticsearch config=socketTimeout has been set as 30000
2024-03-29 09:44:34,960 org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader 118 [main] INFO  [] - Provider=elasticsearch config=responseTimeout has been set as 15000
2024-03-29 09:44:34,961 org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader 118 [main] INFO  [] - Provider=elasticsearch config=numHttpClientThread has been set as 0
2024-03-29 09:44:34,961 org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader 118 [main] INFO  [] - Provider=elasticsearch config=user has been set as elastic
2024-03-29 09:44:34,961 org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader 118 [main] INFO  [] - Provider=elasticsearch config=password has been set as 123
2024-03-29 09:44:34,961 org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader 118 [main] INFO  [] - Provider=elasticsearch config=trustStorePath has been set as /skywalking/p12/storage.p12
2024-03-29 09:44:34,962 org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader 118 [main] INFO  [] - Provider=elasticsearch config=trustStorePass has been set as skywalking  
2024-03-29 09:44:34,962 org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader 118 [main] INFO  [] - Provider=elasticsearch config=secretsManagementFile has been set as

#oap Deployment ENV
        - name: SW_STORAGE_ES_SSL_JKS_PATH
          value: /skywalking/p12/storage.p12
        - name: SW_STORAGE_ES_SSL_JKS_PASS   #No p12 password is changed. Procedure，Always skywalking
          value: skywalking

[wu-sheng]

You could use this debugging tool to verify whether the config is applied.

https://skywalking.apache.org/docs/main/latest/en/debugging/config_dump/

And check whether your system variable is set correctly. In theory, there is nothing special for various config keys.