Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add otel tracing implementation support #2578

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
27 changes: 19 additions & 8 deletions docs/reference/plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -430,18 +430,11 @@ func (g *geoip) Match(r *http.Request) bool {
}
```

## OpenTracing plugins
## Tracing plugins

The tracers, except for `noop`, are built as Go Plugins. A tracing plugin can
be loaded with `-opentracing NAME` as parameter to skipper.

Implementations of OpenTracing API can be found in the
https://github.com/skipper-plugins/opentracing repository.

All plugins must have a function named `InitTracer` with the following signature

func([]string) (opentracing.Tracer, error)

The parameters passed are all arguments for the plugin, i.e. everything after the first
word from skipper's -opentracing parameter. E.g. when the -opentracing parameter is
`mytracer foo=bar token=xxx somename=bla:3` the "mytracer" plugin will receive
Expand All @@ -452,6 +445,24 @@ as arguments.

The tracer plugin implementation is responsible to parse the received arguments.

Tracer plugins need to follow either OpenTelemetry specification or OpenTracing specification. OpenTelemetry is preferable since OpenTracing will be deprecated in the next few months.

### OpenTelemetry

All opentelemetry plugins must have a function named `InitTracer` with the following signature

func([]string) (opentelemetry.Tracer, error)

### OpenTracing

Implementations of OpenTracing API can be found in the
https://github.com/skipper-plugins/opentracing repository.

All opentracing plugins must have a function named `InitTracer` with the following signature

func([]string) (opentracing.Tracer, error)


An example plugin looks like
```go
package main
Expand Down
19 changes: 16 additions & 3 deletions docs/tutorials/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -280,13 +280,26 @@ delay between the calls.
A complete example is the [routestring implementation](https://github.com/zalando/skipper/blob/master/dataclients/routestring/string.go), which fits in
less than 50 lines of code.

## Opentracing
## Tracing

Currently Skipper supports both Opentelemetry and Opentracing as tracing options. Opentracing is soon to be deprecated in favor of Opentelemetry.

Please check the [tracing package](https://github.com/zalando/skipper/blob/master/tracing)
and ask for further guidance in our [community channels](https://github.com/zalando/skipper#community).

### Opentelemetry

Your custom Opentelemetry implementations need to satisfy the `opentelemetry.Tracer` interface from
https://github.com/open-telemetry/opentelemetry-go and need to be loaded as
a plugin, which might change in the future.

### Opentracing

Your custom Opentracing implementations need to satisfy the `opentracing.Tracer` interface from
https://github.com/opentracing/opentracing-go and need to be loaded as
a plugin, which might change in the future.
Please check the [tracing package](https://github.com/zalando/skipper/blob/master/tracing)
and ask for further guidance in our [community channels](https://github.com/zalando/skipper#community).

Currently Skipper is migrating to Opentelemetry, what this means is, when this plugin is loaded it will be wrapped in a type called [TracerWrapper](https://github.com/zalando/skipper/blob/master/tracing/wrappers.go), this type implements `opentelemetry.Tracer` interface, and under the hood converts these function calls to `opentracing.Tracer` calls. In the future skipper will only accept plugins that return `opentelemetry.Tracer`.

## Core

Expand Down
9 changes: 5 additions & 4 deletions filters/auth/authclient.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,10 @@ import (
"strings"
"time"

"github.com/opentracing/opentracing-go"
"github.com/zalando/skipper/filters"
"github.com/zalando/skipper/net"
"go.opentelemetry.io/otel/trace"
"go.opentelemetry.io/otel/trace/noop"
)

const (
Expand All @@ -36,9 +37,9 @@ type tokeninfoClient interface {

var _ tokeninfoClient = &authClient{}

func newAuthClient(baseURL, spanName string, timeout time.Duration, maxIdleConns int, tracer opentracing.Tracer) (*authClient, error) {
func newAuthClient(baseURL, spanName string, timeout time.Duration, maxIdleConns int, tracer trace.Tracer) (*authClient, error) {
if tracer == nil {
tracer = opentracing.NoopTracer{}
tracer = noop.NewTracerProvider().Tracer("Noop tracer")
}
if maxIdleConns <= 0 {
maxIdleConns = defaultMaxIdleConns
Expand All @@ -53,7 +54,7 @@ func newAuthClient(baseURL, spanName string, timeout time.Duration, maxIdleConns
ResponseHeaderTimeout: timeout,
TLSHandshakeTimeout: timeout,
MaxIdleConnsPerHost: maxIdleConns,
Tracer: tracer,
OtelTracer: tracer,
OpentracingComponentTag: "skipper",
OpentracingSpanName: spanName,
})
Expand Down
10 changes: 6 additions & 4 deletions filters/auth/grantconfig.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@ import (
"strings"
"time"

"github.com/opentracing/opentracing-go"
"github.com/zalando/skipper/filters"
snet "github.com/zalando/skipper/net"
"github.com/zalando/skipper/routing"
"github.com/zalando/skipper/secrets"
"go.opentelemetry.io/otel/trace"
"golang.org/x/oauth2"
)

Expand Down Expand Up @@ -119,8 +119,10 @@ type OAuthConfig struct {
// MaxIdleConnectionsPerHost used for tokeninfo, access-token and refresh-token endpoint.
MaxIdleConnectionsPerHost int

// Tracer used for tokeninfo, access-token and refresh-token endpoint.
Tracer opentracing.Tracer
// Tracer is an abstraction around different tracers. Currently skipper can
// operate both with OpenTracing and OpenTelemetry. This tracer traces the communication
// with access-token and refresh-token endpoints.
Tracer trace.Tracer
}

var (
Expand Down Expand Up @@ -188,7 +190,7 @@ func (c *OAuthConfig) Init() error {
ResponseHeaderTimeout: c.ConnectionTimeout,
TLSHandshakeTimeout: c.ConnectionTimeout,
MaxIdleConnsPerHost: c.MaxIdleConnectionsPerHost,
Tracer: c.Tracer,
OtelTracer: c.Tracer,
OpentracingComponentTag: "skipper",
OpentracingSpanName: "grantauth",
})
Expand Down
6 changes: 3 additions & 3 deletions filters/auth/oidc.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,12 @@ import (
"time"

"github.com/coreos/go-oidc"
"github.com/opentracing/opentracing-go"
log "github.com/sirupsen/logrus"
"github.com/tidwall/gjson"
"github.com/zalando/skipper/filters"
snet "github.com/zalando/skipper/net"
"github.com/zalando/skipper/secrets"
"go.opentelemetry.io/otel/trace"
"golang.org/x/oauth2"
)

Expand Down Expand Up @@ -91,7 +91,7 @@ type OidcOptions struct {
MaxIdleConns int
CookieValidity time.Duration
Timeout time.Duration
Tracer opentracing.Tracer
Tracer trace.Tracer
}

type (
Expand Down Expand Up @@ -1032,7 +1032,7 @@ func (f *tokenOidcFilter) initClient() *snet.Client {
ResponseHeaderTimeout: f.oidcOptions.Timeout,
TLSHandshakeTimeout: f.oidcOptions.Timeout,
MaxIdleConnsPerHost: f.oidcOptions.MaxIdleConns,
Tracer: f.oidcOptions.Tracer,
OtelTracer: f.oidcOptions.Tracer,
OpentracingComponentTag: "skipper",
OpentracingSpanName: "distributedClaims",
})
Expand Down
4 changes: 2 additions & 2 deletions filters/auth/tokeninfo.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ import (
"strings"
"time"

"github.com/opentracing/opentracing-go"
"github.com/zalando/skipper/filters"
"go.opentelemetry.io/otel/trace"
)

const (
Expand All @@ -26,7 +26,7 @@ type TokeninfoOptions struct {
URL string
Timeout time.Duration
MaxIdleConns int
Tracer opentracing.Tracer
Tracer trace.Tracer

// CacheSize configures the maximum number of cached tokens.
// The cache evicts least recently used items first.
Expand Down
10 changes: 6 additions & 4 deletions filters/auth/tokenintrospection.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import (

"github.com/opentracing/opentracing-go"
"github.com/zalando/skipper/filters"
"github.com/zalando/skipper/tracing"
"go.opentelemetry.io/otel/trace"
)

const (
Expand All @@ -37,7 +39,7 @@ const (

type TokenintrospectionOptions struct {
Timeout time.Duration
Tracer opentracing.Tracer
Tracer trace.Tracer
MaxIdleConns int
}

Expand Down Expand Up @@ -173,7 +175,7 @@ func NewSecureOAuthTokenintrospectionAllClaims(timeout time.Duration) filters.Sp
// NewOAuthTokenintrospectionAnyClaims, NewOAuthTokenintrospectionAllClaims,
// NewSecureOAuthTokenintrospectionAnyKV, NewSecureOAuthTokenintrospectionAllKV,
// NewSecureOAuthTokenintrospectionAnyClaims, NewSecureOAuthTokenintrospectionAllClaims,
// pass opentracing.Tracer and other options in TokenintrospectionOptions.
// pass open telemetry Tracer and other options in TokenintrospectionOptions.
func TokenintrospectionWithOptions(
create func(time.Duration) filters.Spec,
o TokenintrospectionOptions,
Expand All @@ -193,7 +195,7 @@ func newOAuthTokenintrospectionFilter(typ roleCheckType, timeout time.Duration)
typ: typ,
options: TokenintrospectionOptions{
Timeout: timeout,
Tracer: opentracing.NoopTracer{},
Tracer: &tracing.TracerWrapper{Ot: opentracing.NoopTracer{}},
},
secure: false,
}
Expand All @@ -204,7 +206,7 @@ func newSecureOAuthTokenintrospectionFilter(typ roleCheckType, timeout time.Dura
typ: typ,
options: TokenintrospectionOptions{
Timeout: timeout,
Tracer: opentracing.NoopTracer{},
Tracer: &tracing.TracerWrapper{Ot: opentracing.NoopTracer{}},
},
secure: true,
}
Expand Down
9 changes: 7 additions & 2 deletions filters/auth/webhook.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,11 @@ import (
"time"

"github.com/opentracing/opentracing-go"
"go.opentelemetry.io/otel/trace"
"golang.org/x/net/http/httpguts"

"github.com/zalando/skipper/filters"
"github.com/zalando/skipper/tracing"
)

const (
Expand All @@ -20,7 +22,7 @@ const (
type WebhookOptions struct {
Timeout time.Duration
MaxIdleConns int
Tracer opentracing.Tracer
Tracer trace.Tracer
}

type (
Expand All @@ -39,7 +41,10 @@ var webhookAuthClient map[string]*authClient = make(map[string]*authClient)
// to validate authorization for requests via an
// external web hook.
func NewWebhook(timeout time.Duration) filters.Spec {
return WebhookWithOptions(WebhookOptions{Timeout: timeout, Tracer: opentracing.NoopTracer{}})
return WebhookWithOptions(WebhookOptions{
Timeout: timeout,
Tracer: &tracing.TracerWrapper{Ot: opentracing.NoopTracer{}},
})
}

// WebhookWithOptions creates a new auth filter specification
Expand Down
14 changes: 9 additions & 5 deletions filters/filters.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import (

log "github.com/sirupsen/logrus"

"github.com/opentracing/opentracing-go"
"go.opentelemetry.io/otel/trace"
)

const (
Expand Down Expand Up @@ -48,6 +48,10 @@ type FilterContext interface {
// with its properties changed by the filters.
Request() *http.Request

// Set a new request object in the context. This is useful when its necessary to make small changes
// to the request, like set a new request context.
WithRequest(*http.Request)

// The response object. It is returned to the client with its
// properties changed by the filters.
Response() *http.Response
Expand Down Expand Up @@ -107,11 +111,11 @@ type FilterContext interface {
// Allow filters to collect metrics other than the default metrics (Filter Request, Filter Response methods)
Metrics() Metrics

// Allow filters to add Tags, Baggage to the trace or set the ComponentName.
Tracer() opentracing.Tracer
// Allow filters to add Tags, Baggage to open tracing trace or set the ComponentName.
Tracer() trace.Tracer

// Allow filters to create their own spans
ParentSpan() opentracing.Span
// Allow filters to create their own open tracing spans
ParentSpan() trace.Span

// Returns a clone of the FilterContext including a brand new request object.
// The stream body of the new request is shared with the original.
Expand Down
18 changes: 12 additions & 6 deletions filters/filtertest/filtertest.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@ package filtertest
import (
"net/http"

"github.com/opentracing/opentracing-go"
"github.com/zalando/skipper/filters"
"go.opentelemetry.io/otel/trace"
"go.opentelemetry.io/otel/trace/noop"

log "github.com/sirupsen/logrus"
)
Expand All @@ -33,7 +34,7 @@ type Context struct {
FBackendUrl string
FOutgoingHost string
FMetrics filters.Metrics
FTracer opentracing.Tracer
FTracer trace.Tracer
}

func (spec *Filter) Name() string { return spec.FilterName }
Expand All @@ -58,15 +59,16 @@ func (fc *Context) ResponseController() *http.ResponseController {
return http.NewResponseController(fc.FResponseWriter)
}

func (fc *Context) Tracer() opentracing.Tracer {
func (fc *Context) Tracer() trace.Tracer {
if fc.FTracer != nil {
return fc.FTracer
}
return &opentracing.NoopTracer{}
return noop.NewTracerProvider().Tracer("Noop tracer - filtertest")
}

func (fc *Context) ParentSpan() opentracing.Span {
return opentracing.StartSpan("test_span")
func (fc *Context) ParentSpan() trace.Span {
_, span := fc.FTracer.Start(fc.FRequest.Context(), "test_span")
return span
}

func (fc *Context) Logger() filters.FilterContextLogger {
Expand All @@ -79,6 +81,10 @@ func (fc *Context) Serve(resp *http.Response) {
fc.FServed = true
}

func (fc *Context) WithRequest(r *http.Request) {
fc.FRequest = r
}

//lint:ignore ST1016 ignore receiver name, because of type reuse
func (spec *Filter) CreateFilter(config []interface{}) (filters.Filter, error) {
return &Filter{spec.FilterName, config}, nil
Expand Down
8 changes: 4 additions & 4 deletions filters/openpolicyagent/evaluation.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,11 @@ import (
"github.com/open-policy-agent/opa/rego"
"github.com/open-policy-agent/opa/server"
"github.com/open-policy-agent/opa/tracing"
"github.com/opentracing/opentracing-go"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/trace"
)

func (opa *OpenPolicyAgentInstance) Eval(ctx context.Context, req *ext_authz_v3.CheckRequest) (*envoyauth.EvalResult, error) {
func (opa *OpenPolicyAgentInstance) Eval(ctx context.Context, span trace.Span, req *ext_authz_v3.CheckRequest) (*envoyauth.EvalResult, error) {

decisionId, err := opa.idGenerator.Generate()
if err != nil {
Expand All @@ -29,9 +30,8 @@ func (opa *OpenPolicyAgentInstance) Eval(ctx context.Context, req *ext_authz_v3.
return nil, err
}

span := opentracing.SpanFromContext(ctx)
if span != nil {
span.SetTag("opa.decision_id", result.DecisionID)
span.SetAttributes(attribute.String("opa.decision_id", result.DecisionID))
}

var input map[string]interface{}
Expand Down