API proxy connection to backend cannot be secured

[rhuss]

We have the use case that we have monitoring agents installed on our Java pods (Jolokia )which are accessed via the OpenShift API Proxy. This agent is used by the fabric8 console for a deep inspection of the managed pods (e.g. you can dive into the pod for debugging, visualisation of Camel routes, JMX operations, ...).

This agent can be secured via OAuth in that it can validate a given token against the OpenShift OAuth server.

However, the OS API Server seems to filter out the Authorization: header before accessing the backend. I guess this is for security concerns to prevent a malicious agent to get access to the admin token.

My question is whether there is an alternative solution for this use case ? It would possibly help to copy over another, custom header (like X-Authorization:) over to Authorization: before contacting the backend. On the other hand, if the client and the backend agree, the client can always push the token to the other side (eg. by using any non-filtered header or even via the payload). So maybe filtering the Authorization: header doesn't make so much sense ? Ok, this filter prevents unconditionally passing of the header to the backend when the client doesn't want to.

tl;dr - Is there a canonical way how a proxied service can participate in OAuth SSO ?

[liggitt]

Authentication to the OS API is distinct from accessing the applications... it is very intentional not to leak API credentials to applications running on the infrastructure.

[rhuss]

@liggitt completely agreed, but I wonder how the application behind the proxy could then be authenticated when the Authorization: slot is already occupied for securing the API access.

[liggitt]

if the API was ONLY acting as a proxy, I would expect to use Proxy-Authorization for the API, and Authorization for the back-end app, but since the same auth handler services the API and proxy endpoints, and the same client code passes authentication tokens to both, it's going to be tough to special case auth to the proxy endpoints

[liggitt]

if the app already has its own authorization, can't you just access it directly using a route->service that exposes it? I thought the reason we were going through the API was to get the benefit of the API's authorization

[rhuss]

Its also about the principal used for authorization in addition to authentication. This information gets also lost. For authentication alone IMO it would be sufficient to let proxy do the authentication alone. Maybe we could copy over the principal information into an header when proxying ?

[liggitt]

that's interesting... a REMOTE_USER header of sorts... the proxy could strip that header from requests and replace it with the authenticated principal, but I'm not sure who else has network visibility to the pods who could spoof that header

[rhuss]

if the app already has its own authorization, can't you just access it directly using a route->service that exposes it?

We could do, but since its about an agent installed for managing/monitoring every (java based) service an own route feels a bit heavy weight. Also since it's something which we don't want to expose to everyone, too, the proxy based access for this monitoring/management feature feels just right.

[jimmidyson]

The main use case right now is the jmx console in the openshift console.
This connects to pods via the api server proxy & at the moment the jolokia
endpoint in each pods is unauthenticated. We want to enable auth &
preferably reuse openshift auth token.

On 18:35, Thu, 3 Sep 2015 Roland Huß notifications@github.com wrote:

if the app already has its own authorization, can't you just access it
directly using a route->service that exposes it?

We could do, but since its about an agent installed for
managing/monitoring every (java based) service an own route feels a bit
heavy weight. Also since it's something which we don't want to expose to
everyone, too, the proxy based access for this monitoring/management
feature feels just right.

—
Reply to this email directly or view it on GitHub
#4497 (comment).

[rhuss]

yes, something like a REMOTE_USER would be perfect (e.g. the OAuth user). /wrt the network visibility this could be indeed an issue if not going over SSL. The only 'strong' solution beyond using SSL I see would be signing a la JSON web tokens but thats probably over the top ....

[rhuss]

The main use case right now is the jmx console in the openshift console.
This connects to pods via the api server proxy & at the moment the jolokia
endpoint in each pods is unauthenticated. We want to enable auth &
preferably reuse openshift auth token.

@jimmidyson but when we allow only access via the proxy, couldn't we trust the proxy that he did the authentication ?

[liggitt]

SSL only helps you if the proxy uses a client cert to make the request and the app verifies it

[rhuss]

SSL only helps you if the proxy uses a client cert to make the request and the app verifies it

That's true and of course also for the authentication trust relationship between proxy and proxied service.

[jimmidyson]

We could but that would mean changing the trusted proxy for every
installation, updating it masters are added, etc.

On 18:41, Thu, 3 Sep 2015 Roland Huß notifications@github.com wrote:

The main use case right now is the jmx console in the openshift console.
This connects to pods via the api server proxy & at the moment the jolokia
endpoint in each pods is unauthenticated. We want to enable auth &
preferably reuse openshift auth token.

@jimmidyson https://github.com/jimmidyson but when we allow only access
via the proxy, couldn't we trust the proxy that he did the authentication
?

—
Reply to this email directly or view it on GitHub
#4497 (comment).

[jimmidyson]

A jwt would do the same job as a client cert wouldn't it? Still needs validation of course. But we've discussed this before. This is another use case I guess.

[liggitt]

it allows validation, but also hands credentials to the app, which I'm not happy about

[liggitt]

we'd want separate id/access tokens

[dhirajsb]

How about some kind of temporary token with user id, which is blacklisted for api access? Doesn't quite solve your problem of hiding id, but would block api attacks.

[rhuss]

@dhirajsb the token is provided and created by the OAuth server and doesn't contain user id information itself. You receive the user info from the auth server only as response to the token verification request. So its not so easy for the API server to create such temporary tokens. Also it would involve holding state in the API server (the black list) which I don't know if this is feasible.

[liggitt]

the proxy code is all upstream, which has no dynamic token system for normal users at the moment

[rhuss]

it allows validation, but also hands credentials to the app, which I'm not happy about

You can use jwt with asymmetric, public key validation, too. The public key could be injected into the pods e.g. via envs. That way only the API server can validly sign the token (e.g. holding the username) and the backend service can verify it without sharing any credentials.

[jimmidyson]

OK so the token would be an ID token, rather than an access token? SGTM!

[rhuss]

Our current approach (probably more a workaround) to allow the transport of the authorization token between two trusted applications (i.e. the openshift / fabric8 console and Jolokia for deep inspection of a running Java container) by using a custom header (e.g. X-Authorization) doesn't work because this header can't be added to the CORS preflight check's response of the proxy itself. More details are described here.

Currently we are stucked for how to secure Jolokia as a backend, administrative service behind the API proxy. IMO the alternative to leave Jolokia unsecured is no alternative and much worse than passing the API token (since any other pod can reach these open ports). But I agree, allowing the admin token to be passed to any backend service is no solution either.

So, my question really boils down whether the use case

As an administrator
I want to access a backend administrative service via the OpenShift proxy in a secured fashion
so that I can perform application specific administrative tasks

is supported somehow ? (as mentioned above, exposing it as separate services for every pod is not really an option).

[smarterclayton]

@liggitt let's try to figure out what the minimal enablement we can do here is.

[liggitt]

There are several issues with plumbing authorization headers through the proxy to backend apps:

1. Conflicts with API authorization headers (as noted above)
   • Headers are automatically removed after auth to avoid leaking credentials
2. Conflicts with OpenShift<->Kube authorization headers when running OpenShift against a separate Kube instance (client -> OpenShift -> Kube -> backend)
   • Headers are removed and potentially replaced with the OpenShift credentials to proxy to Kube
3. API proxy support for https to the backend is not present (see The API proxy cannot handle https endpoints  kubernetes/kubernetes#14194)

If the backend app wants to handle authorization, is there a reason not to expose it as a service and take the proxy out of the equation?

[smarterclayton]

Although to be fair proxy is there precisely to let you deal with the 20% -
everything else is busted. Having the auth be able to come through a proxy
(not necessarily Kube auth, but a separate X-Authorization header) would at
least allow you to do that.

I agree though that pass through of the auth header is very scary.

On Sep 22, 2015, at 3:33 PM, Jordan Liggitt notifications@github.com
wrote:

There are several issues with plumbing authorization headers through the
proxy to backend apps:

1. Conflicts with API authorization headers (as noted above)

• Headers are automatically removed
  

  origin/pkg/auth/authenticator/request/bearertoken/bearertoken.go

  Line 35 in dc1a7b0

  req.Header.Del("Authorization")

  
  after auth to avoid leaking credentials
• Conflicts with OpenShift<->Kube authorization headers when running
  OpenShift against a separate Kube instance (client -> OpenShift -> Kube ->
  backend)
• Headers are removed
  

  origin/pkg/util/httpproxy/upgradeawareproxy.go

  Lines 243 to 245 in dc1a7b0

  	func removeAuthHeaders(req *http.Request) {
  	req.Header.Del("Authorization")
  	}

  
  and potentially replaced
  

  origin/pkg/util/httpproxy/upgradeawareproxy.go

  Lines 264 to 270 in dc1a7b0

  	func addAuthHeaders(req *http.Request, clientConfig *kclient.Config) {
  	if clientConfig.BearerToken != "" {
  	req.Header.Set("Authorization", "Bearer "+clientConfig.BearerToken)
  	} else if clientConfig.Username != "" || clientConfig.Password != "" {
  	req.SetBasicAuth(clientConfig.Username, clientConfig.Password)
  	}
  	}

  
  with the OpenShift credentials to proxy to Kube
• API proxy support for https to the backend is not present (see
  The API proxy cannot handle https endpoints  kubernetes/kubernetes#14194
  The API proxy cannot handle https endpoints  kubernetes/kubernetes#14194)

If the backend app wants to handle authorization, is there a reason not to
expose it as a service and take the proxy out of the equation?

—
Reply to this email directly or view it on GitHub
#4497 (comment).

[liggitt]

Opened kubernetes/kubernetes#14876 to address the issue with proxying TLS to the backend pods

[liggitt]

Created https://trello.com/c/kTX8fBNs/522-allow-api-proxy-to-securely-connect-to-backend-pods to track overall progress