Automating security index backup

[zalseryani]

Current Environment.

• we have opensearch deployed behind nginx ingress on kubernetes cluster,

• opensearch is deployed with helm charts,

• opensearch-dashboards is also deployed with helm charts.

• Since opensearch is deployed behind a proxy and we have ssl termination on nginx controllers, @dblock supported me previously by suggesting to update http: true, to http: false in order to be able to reach opensearch, as when http is set to true, and using the nginx ingress with the request, we are making plain text requests to opensearch that is configured to receive a secure requests,

The Issue

• Since the documentation suggests excluding the security index from snapshots, and provides another way of taking a security configuration backup using the securityadmin.sh, I am facing the following case:

• while http: false , and executing the securityadmin.sh script I get the following error

Security Admin v7
Will connect to localhost:9200 ... done
ERR: An unexpected IOException occured: Unrecognized SSL message, plaintext connection?
Trace:
java.io.IOException: Unrecognized SSL message, plaintext connection?
	at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:958)
	at org.opensearch.client.RestClient.performRequest(RestClient.java:332)
	at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
	at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:572)
	at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:162)
Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
	at java.base/sun.security.ssl.SSLEngineInputRecord.bytesInCompletePacket(SSLEngineInputRecord.java:145)
	at java.base/sun.security.ssl.SSLEngineInputRecord.bytesInCompletePacket(SSLEngineInputRecord.java:64)
	at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:612)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
	at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
	at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:279)
	at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:333)
	at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:545)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Thread.java:833)

• The only way to make securityadmin.sh script works is to enable http: true again, and restart opensearch, and this can not be automated when we want to take a backup for security configuration.

Questions:

1- Why securityadmin.sh script is related to the following configuration http: true/false ?
2- Does this have any other workaround to allow executing the securityadmin.sh script without having to modify the configuration from http: false to http: true when having opensearch deployed behind nginx ingress on kubernetes cluster ?

Amazing efforts being done on this project and I would gladly ask your help and support regarding this matter,
Thank you.

[zalseryani]

Solved by using nginx-ingress annotation nginx.ingress.kubernetes.io/backend-protocol: HTTPS at opensearch ingress resource and in this case, nginx controller will communicate https with the backend (opensearch) and it will trust the certificate of the backend,

• Note: This is will make you have double encryption and decryption on each request because first request towards your ingress-nginx controller will have ssl termination on the controller endpoint itself, then the controller will communicate with opensearch https and the controller job to trust the certificate of opensearch server if you are using a self-signed certificates for http

But, I believe this should be solved from opensearch side not from the ingress controller, as in this case, we need to have all our applications communicate with opensearch with https rather than http, even if they are in the same kubernetes cluster where we could use opensearch serviceName.Namespace host.

Please correct me if I am wrong, and advise other proper solution if available.

Thank you.

[prudhvigodithi]

[Untriage]
Hey can you please share the securityadmin.sh command, I usually run as follows inside the pod (where localhost is https).

/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd "/usr/share/opensearch/config/opensearch-security" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv 

Also when you mention http: true, to http: false, you mean adding a flag as plugins.security.ssl.http.enabled=false?

WIth nginx can you try with the following annotations and yes it would have double encryption, one at the reverse proxy level (nginx) and other with OpenSearch.

    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "false"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

Adding @bbarani @peterzhuamazon @rishabh6788 @gaiksaya