Fix readiness probe script to let xpack.security.http.ssl.client_authentication be set to required

[ryanhockstad]

It looks like currently the xpack.security.http.ssl.client_authentication setting won't work if it is set to 'required.' This is caused when the readiness_probe.sh tries to ping https://localhost:9200, but it doesn't present a cert or key file. The fix is to simply set the --key and --cert flags of the curl request. This still works when ECK is configured to use custom certs because the operator always puts them in the /usr/share/elasticsearch/config/http-certs/ directory, and it also continue to work even when SSL is disabled because the cert files will still be there.

It's worth pointing out that this setting isn't listed in the Settings Managed by ECK doc page, which is why I decided to fix it.

[pebrc]

Thanks for your PR! I think the problem I see with this is that changing the readiness probe alone won't put us into a position to enable client certificate verification. I think we would also have to at least adjust the HTTP client used by the operator to present a certificate to still be able to reconcile clusters that have xpack.security.http.ssl.client_authentication enabled.

[ryanhockstad]

Oh yeah, that makes sense. I only tested this out on a 3 node cluster to see if it would spin up, but I didn't think about the operator making requests to the cluster.

That also makes me realize the other CRDs would need to be edited to present certs when connecting to ES, too.

Yeah so this is a way bigger lift than I realized at first. Should we edit this doc page to have xpack.security.http.ssl.client_authentication as a setting users shouldn't change, then?

[pebrc]

Yes I think we can do two things. We can update the documentation to highlight this limitation and we can open an issue to address this in a more comprehensive manner. Is there a specific use case you had where you strictly required client certificate verification or was this just an experiment?

[ryanhockstad]

I had a client who wanted client certificate verification enabled, but they were ok moving on without it when I realized ECK doesn't support it. After that I was just interested in figuring out how we could get it to work.