Sharing example of using tokens + custom ssl certs

[jordansissel]

I am still feeling like it's too hard to do what I feel are common things with this client.

In order to use tokens, you have to implement a custom RoundTripper which is pretty deep and awkward within Go, especialy if you're a developer who just wants to "use elasticsearch" and move on; implementing a custom RoundTripper or custom Transport is tedious.

In order to use custom ssl certs, you have to modify the TLSClientConfig of the transport. This is still pretty deep within net/http as compared to the abstraction layer "elasticsearch client library" ; deeper than I'd prefer.

I was unable to find any solid examples of setting both custom headers (for tokens) and setting custom CA certs.

To that end, here's what I found works; this code requires Go 1.13 for http.Transport.Clone()

package main

import (
	"crypto/x509"
	"fmt"
	"github.com/elastic/go-elasticsearch/v8"
	"io/ioutil"
	"log"
	"net/http"
	"os"
)

type CustomRoundTripper struct {
	http.RoundTripper
	Custom func(*http.Request) (*http.Response, error)
}

func (crt CustomRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
	return crt.Custom(req)
}

func main() {
	esurl := os.Args[1]
	token := os.Args[2]
	cafile := os.Args[3]

	transport := http.DefaultTransport.(*http.Transport).Clone()

	var err error
	transport.TLSClientConfig.RootCAs, err = x509.SystemCertPool()

	b, err := ioutil.ReadFile(cafile)
	if err != nil {
		log.Printf("Error reading additional ca certs in path %s / error %s", cafile, err)
		os.Exit(1)
	}

	ok := transport.TLSClientConfig.RootCAs.AppendCertsFromPEM(b)
	if !ok {
		log.Printf("No certs were found in the additional ca cert path: %s", cafile)
		os.Exit(1)
	}

	crt := CustomRoundTripper{
		Custom: func(req *http.Request) (*http.Response, error) {
			req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", token))
			return transport.RoundTrip(req)
		},
	}

	config := elasticsearch.Config{
		Addresses: []string{esurl},
		Transport: crt,
	}

	es, err := elasticsearch.NewClient(config)
	if err != nil {
		log.Printf("Error creating ES client: %s", err)
		os.Exit(1)
	}

	resp, err := es.Ping()
	if err != nil {
		fmt.Printf("Error pinging ES: %s\n", err)
		os.Exit(1)
	}

	fmt.Printf("Response: %s\n", resp)

}

Could we provide some kind of helper function in this library to help users generate their custom Transport? Something like:

elasticsearch.NewClient(elasticsearch.Config{
    Transport: happyHttpHelper{
        Headers: { ... },
        AdditionalCARoots: ...
    }
})

Especially as tokens are becoming more prominent in Elasticsearch (API tokens, etc), it would improve things if we made it less code to achieve this; especially if that "less code" removes areas of Go that, I assume, people do not commonly travel (DIY Transport implementations, etc)

[karmi]

Thanks for opening the issue, and for providing the example, Jordan!

While I still think that providing a custom http.Transport is the correct way to go for many situations where the underlying HTTP client needs to be customized, I can see your argument that eg. passing a root CA or a set of client certificates should be made easier.

I'll describe some suggestions below.

APIKey

The Elasticsearch specific APIKey authentication you mention is already supported by the elasticsearch.Config.APIKey option.

Bearer/Token Authentication

The request for supporting token/bearer authentication has been discussed in #84. Note that the actual implementation is five lines. As I've explained in the discussion, I considered a bearer/token authentication a "non-standard environment", and therefore a custom transport as an appropriate solution. I was wrong about that, since the bearer/token authentication is supported by Elasticsearch itself.

On the other hand, the request can be rephrased as: “How can I set HTTP headers for all requests easily?” Some official clients do support setting custom headers in the client configuration, some don't. I've started a discussion around this in the team.

Note that it's possible to pass custom HTTP headers to each API call manually, and this might even be an attractive option for something like an authentication token, since those tend to expire, and thus it would be possible to handle the renewal process in a routine outside the client.

However, I do see value in making it easy to set "global" HTTP headers in the client initialization, and passing them to API requests automatically. This could be quite trivially implemented by adding a field directly to the configuration struct: Headers http.Header. These would then be automatically propagated to the request during estransport.Client.Perform(), in a similar way how the user agent is set. What do you think about this option?

There's a separate discussion going on about supporting the Token configuration directly, but I would like to discuss this "global headers" option in this ticket.

TLS and Certificates

The situation with TLS and certificates is a bit different. Usually the clients defer to passing a configuration to the underlying library, but with Go, you do have to make a little ceremony around it, as you've demonstrated in your example.

I'm open to discussing how to facilitate passing CAs and certs to the client directly, and remove the friction with ioutil.ReadFile, TLSClientConfig.RootCAs.AppendCertsFromPEM, and so on.

Note I'm far from being an expert in TLS, but as far as I see, there are at least two pieces to make it work well, following eg. the https://venilnoronha.io/a-step-by-step-guide-to-mtls-in-go tutorial as a blueprint:

• The CA file
• The client certificates

Therefore, we would to provide configuration options (or helpers) for both. What I'm thinking about is something like:

package elasticsearch

type Config struct {
	// ...
	CertPEM []byte
	KeyPEM  []byte
}

The client would then takes these values, and performed the appropriate tls.X509KeyPair() and (*x509.CertPool).AppendCertsFromPEM().

I would like to keep these values as []byte — or, alternatively, io.Reader —, rather than eg. strings, to abstract away the source. Granted, this would mean that the calling code would still need to do something like ioutil.ReadAll() — or os.Open —, as you do in your example, but I think it's an acceptable price to pay.

I'm not 100% about the inner mechanics, but that's something to explore once the implementation starts. What's your opinion on this approach?

[jordansissel]

However, I do see value in making it easy to set "global" HTTP headers in the client initialization

+1 this is what I've settled on today, since my use case is temporary/utility tooling and my API keys/tokens would expire a while after I'm done using whatever tooling.

type Config struct {

This example makes sense to me. I know there's always going to be a balance between simplicity and that point when simplicity makes hard things too difficult, but for this specific use case, and my experience with TLS, most common settings from tls.Config will be:

• Certificates (client certificates+key, needed when doing PKI realm or other client verification)
• RootCAs (for adding/removing trusted CA certs)

Less common:

• MinVersion/MaxVersion -- for setting SSL/TLS version constraints. Some customers, for example, require "TLS 1.2 or greater" or similar constraints
• InsecureSkipVerify -- for times when we want to remove the most of the "S" from "TLS" ?

I've grown some fondness for the way you designed esapi with a kind of builder pattern, like Bulk(..., Bulk.WithIndex(...), Bulk.WithWhatever(...)

I wonder about a transport builder that would support this somehow:

es := elasticsearch.NewClient(elasticsearch.NewTransport(
  // TLS
  estransport.Transport.WithAdditionalRootCAs(pem []byte),
  estransport.Transport.WithClientCertificate(foo x509.Certificate),
  estransport.Transport.WithMinVersion(tls.VersionTLS12),
  estransport.Transport.WithInsecureSkipVerify(true),

  // Headers; global for all requests.
  estransport.Transport.WithHeaders(foo http.Header), 
  estransport.Transport.WithBasicAuth(user, pass string),
  estransport.Transport.WithApiKey(id, token string),
  ...
)

Thoughts? My objective above is to present an API which presents what I think some users are thinking, "I need to " reducing to one item in the code. Compared with today, setting TLS min version requires deeper thinking and learning how to use the crypto/tls and net/http packages (probably not enjoyable haha)

[jordansissel]

Here's an updated "custom roundtripper" example that moves header handling into the roundtripper itself:

https://gist.github.com/jordansissel/692e26a5307deb15e6cdaf39e3f2953d

[karmi]

I've added the option to pass custom certificate authorities to the client in #126, and also added an executable example to _examples/security. Let me know if it solves the original problem.