Basic Auth with external authentication service

[giohappy]

I would like to discuss the opportunity to implement Basic Auth authentication through an external service.

The idea is to have something similar to the "Key provider using an external service" offered by the Key authentication module.

I don't know the details of the Basic Auth implementation in Geoserver, but I suppose that implementing a Web Service Authentication Provider would be enough, very similar to the Key Provider:

• URL of the service
• Reges to parse the response
• options to control cache and timeouts
• Api Key

The (optional) Api key will be used to authenticate the request.

User credentials (username and password) could be transmitted to the web service through a custom header, like. The standard Authorization header shouldn't be used though, because it semantics is different.

I would like to hear the opinions from @aaime and the others from the Geoserver team, since I will need to estimate the effort soon.

[simboss]

@giohappy which geoserver version are we targeting?
I would suggest at least 2.16 for a backport.

[aaime]

I have no idea what "Basic Auth authentication through an external service" means? :-D

[simboss]

@nmco can you groom this with @aaime help and then assign it to someone

[giohappy]

@aaime here it is our use case.

The AuthKey module permits to authenticate a user by parsing an authkey parameter. Then my intention is to map that user to its role by means of an AuthKey REST Roles service (which implements the roles and users endpoints)

I would like to implement something similar for Basic Auth access. From my understanding Geoserver's Basic Auth filter only extracts username and password credentials from the request, then it delegates the authentication to the Authentication Providers. The available APs are "username and password" (which uses a UserGroups service to perform the authentication), "LDAP" and "JDBC".
If we had a "Web Service Authentication Provider", able to perform the authentication of the user credentials (obtained from the Basic Auth filter) and return the username, I should have (in few workds) "Basic Auth with external authentication service"

If I've misunderstood the Geoserver security concepts please correct me.
I'm also assuming that retrieving the username from the Web Service (either through the ApiKey Module or this new AP that I'm proposing) is enough, and the Roles Service will do the rest. Am I wrong?

[aaime]

@giohappy ok now it makes sense (well some, I don't know exactly how authkey works, I wrote it originally but it went through 2 full rewrites and does not work at all like it used to).
I have looked into the code to figure out a bit how it works, here is what I've found, best if @mbarto and/or @afabiani confirm it though, I know very little about the GeoServer authentication part (the area that I know more is authorization):

• The authentication filter (how one providers the credentials and validates them): e.g., GeoServerBasicAuthenticationFilter which in turn uses a local user catalog to compare user name and password against. In authkey that can be delegated to an external service through WebServiceAuthenticationKeyMapper which talks to an external service in order to verify an authentication
• After successful authentication, a way to associate roles past successful authentication, which is done by a GeoServerRoleService, e.g. GeoServerRestRoleService

As far as I can see those end up doing two separate remote requests, and there is a caching mechanism in the middle to avoid doing too many requests.

So if I understand correctly, you'd like to extract username and password from the basic auth, and then do remote calls to verify identity and grab roles. I don't know if it can be done with a single call, authkey does it with two following 1-1 the Spring Security architecture (which GeoServer uses with some customization). And I'm guessing the remote calls won't be the same expected by authkey?

[giohappy]

So if I understand correctly, you'd like to extract username and password from the basic auth, and then do remote calls to verify identity and grab roles. I don't know if it can be done with a single call, authkey does it with two following 1-1 the Spring Security architecture (which GeoServer uses with some customization). And I'm guessing the remote calls won't be the same expected by authkey?

I don't know the internal details of AuthKey, but yes both AuthKey and geonode-oauth2 (which are the two modules we currently use) do two calls. First one authenticates the user and retrieves its rolenames and the second one binds them through their respective roles services.

Roughly this is what happens (from our tests)

AuthKey:

• AuthKey Filter -> retrieves users from the WebService Body response (through an apikey token)
• AuthKey UserGroup webservice -> retrieves user's roles from the WebService Body response
• AuthKey UserGroup webservice -> binds user's roles to GrantedAuthorities through ApiKey RESTO Roles service

geonode-oauth2:

• geonode-oauth2 Auth Filter -> retrieves user from an Oauth2 endpoint (through an access token)
• geonode-oauth2 -> retrieved and binds user's roles to GrantedAuthorities through ApiKey RESTO Roles service

In the end it seems that these modules shortcut the configuration of Filters + Providers.
I can't say what's the right / best approach, considering the underlying Spring framework and the
Geoserver's security model.

Options:

• extend the Baisc Auth Filter to authenticate through an external WebService + Roles Service, similar to what AuthKey and geonode-oauth2 do
• extend the username/password authentication provider to support external Web Service + Roles Service

[nmco]

@imranrajjad before you move on with this one wait for my initial investigation.

[imranrajjad]

@nmco understood

[nmco]

Unfortunately I will have not time this or next week to perform the initial investigation, hence I will ask your help whit this @imranrajjad.

Please have look at Giovanni and Andrea feedback and then let's have a quick chat.

[simboss]

usual thing @imranrajjad:

• check the requirements, makre sure they are clear, ask questions if they are not
• check code in GS, propose a solution that align with GS codebase
• provide an estimate and wait for the GO

[imranrajjad]

Before sharing the plan and estimate, I am a bit unclear on the type of response which should be expected by the authenticating webservice. Is it a one time call which responds with roles already known to GS through available Role provider services? Or will it simply respond with a yes/no.

If there is such a service available or atleast a specification is available, it will be really helpful.

Plan

Web Authentication Provider

Extend GeoServerAuthenticationProvider and implement a functionality similar to UsernamePasswordAuthenticationProvider. User will be able to configure this provider with a URL, including place holders and response parsing scheme (e.g JSON path, Regex or Xpath)

Configuration

A configuration UI that will allow users to define external URL with required place holder
(e.g http://auth.service.com/auth?use={USER}&pass={passowrd}

User can also configure how to handle response. The response handling will differ with the response content types.

Initial estimate 3 days

[simboss]

@imranrajjad have read the full story above? Have you reviewed how authkey works?
If you did, well, do it again :).

In any case, you can ask details on the service counterpart to @giohappy but, mind you, we are not developing something specific for DJAM but something rather generic like what we have for authkey.

Keep the discussion going here and keep @nmco in the loop.

[nmco]

Summarizing the actions points (for you) that resulted from our call @imranrajjad:

• Finish setting up AuthKey working with an external web service so you can properly debug how it works, ask @taba90 help, he just configure it last week for another issue.
• Investigate how AuthKey works, more precisely the integration points with Spring security to support the key external web service scenario. Describe your findings on this issue.
• Describe the possibilities you see to implemented the requested features along whit their pros and cons:
  • A new authentication provider based on a web service and then a new roles provider?
  • A single call for the authentication and roles requesting?
  • Should we have a cache in between?
  • ...
• The implemented solution should be as much generic as possible, see for example how AuthKey supports a regex to get the roles from the web service response, hence creating a loose coupling between the client and the service.

Once we have a green light for one of the propose approaches, let's setup a detailed plan and review it, this will require a GSIP ♠️.

[giohappy]

@imranrajjad and @nmco I don't know if a custom Authentication Filter would be ok here, as it was done for geonode-oauth2 and ApiKey, or if an Authentication Provider is the right way to go.

In any case I fin that:

• geonode-oauth2 has the most straightforward configuration (and it would be sufficient for my specific purposes): you set the username authentication endpoint and select and ApiKey Role Service to match user roles
  

• ApiKey uses a more convoluted approach: it gives you more functionalities, like mapping remote groups to a subset of local groups (I don't know if anybody uses this), and it uses the optional roles returned by the webservice to match Geoserver groups and/or roles. For roles matching it uses the AuthKey UserGroup Service where the REST Roles Service is selected. It basically decouples the filter and the REST roles service through a UserGroups Service.

AuthKey Configuration

UserGroup Service Configuration

For our use case having something simpler, similar to geonode-oauth, is fine, but I'll let @nmco express itself on this.

[imranrajjad]

@giohappy thanks for the detailed clarification,

After going through the basic user:password based authentication and how authKey works. Below are my findings and a revised plan of implementation.

Since these requirements do not involve key mechanism, hence this implementation will go in the core and not the authkey extension.

• Currently, as rightly pointed by @aaime , GeoServerBasicAuthenticationFilter.java#L48 is a wrapper around org.springframework.security.web.authentication.www.BasicAuthenticationEntryPoint to which credentials based authentication is delegated to and this is the part that this requirement wishes to carry out using an external web service.
• Since we are concerned with user/pass like credentials. It will make sense to introduce a new Authorization Provider which delegates calls to an external web service. The interaction with the external web service can be made generic and configurable by taking inspiration for Authkey, where users can define a an external end point with placeholders. The authentication process can be similar to whats happening in WebServiceAuthenticationKeyMapper.callWebService.

For processing roles, there can be two scenarios

1. Roles are returned against a successful authentication call
2. The service only returns confirmation in terms of content (which can be parsed using regex) or HTTP code.

To cater for both scenarios, the Provider GUI page can allow user to either configure a local service available in Geoserver or it can allow user to parse response to extract roles, again inspiration can be taken from how its being in authKey. Something similar to WebServiceBodyResponseUserGroupService.extractRoles but embeded inside the new Authorization Provider. This will handle both scenarios where a web service responds with authorities and where its just confirmation of credentials.

I would be glad to hear everyone`s thoughts on this.

[nmco]

@giohappy and @imranrajjad I don't understand nothing about GeoServer security authentication neither as a user, and certainly not code wise, so please read my comments taking that into consideration.

That say, I had a quick read through the GeoServer documentation and based on this description from @giohappy :

I would like to implement something similar for Basic Auth access. From my understanding Geoserver's Basic Auth filter only extracts username and password credentials from the request, then it delegates the authentication to the Authentication Providers. The available APs are "username and password" (which uses a UserGroups service to perform the authentication), "LDAP" and "JDBC".
If we had a "Web Service Authentication Provider", able to perform the authentication of the user credentials (obtained from the Basic Auth filter) and return the username, I should have (in few workds) "Basic Auth with external authentication service"

It looks to me we have two needs:

• A new authentication provider, to authenticate the user and obtain a principal.
• A new roles service, to obtain the authenticated user authorities.

Then there is this (insane) security system architecture, which seems to imply that roles can be obtained directly without have the groups service in the middle, but sometimes not.

Long story short we (@nmco, @imranrajjad and @giohappy ) need to have a call tomorrow to discuss this to understand what's actually needed in the user perspective, what can we do to implement this and make it generic.

[giohappy]

@imranrajjad here below I attach an example from a working configuration of an AuthKey REST Roles Service

Geoserver Configuration

Sample requests

Users endpoint
(I'll send you <api_key> in a private channel to test it)

Request:

curl -X GET \
  https://dev.geonode.geo-solutions.it/api/users \
  -H 'Accept: */*' \
  -H 'Accept-Encoding: gzip, deflate' \
  -H 'Authorization: ApiKey <api_key>' \
  -H 'Cache-Control: no-cache' \
  -H 'Host: dev.geonode.geo-solutions.it' \

Response (partial view):

{
    "users": [
        {
            "username": "AnonymousUser",
            "groups": [
                "anonymous"
            ]
        },
        {
            "username": "henryhansen",
            "groups": [
                "anonymous",
                "registered-members"
            ]
        }
    ]
}

Roles and adminRole endpoints
Same as above, just change the endpoint URLs as from the Geoserver configuration.

[imranrajjad]

@nmco and @giohappy thanks for the meeting and clarification. Below please find the updated plan after our call yesterday.

Litmus Test

• Configured a mock Authkey Rest Role service and AuthKEY WebService Body Response UserGroup
• Configured an Authkey filter with web service mapper as shown below

With a correct regex to parse the user name from the response, I was able to use role services associated with all of the user groups and not just the Auth Key user groups.

What this means in practice is that taking a username we can use any one of the available role services to authorize users. In case of Authkey mapper the user name is retrieved by providing the webservice with auth key. In the current scenario we will work with credentials directly instead of an key, hence this forces us to introduce an AuthorizationProvider.

Implementation Plan

Authentication

To understand the implementation plan, we can look at the example of authentication and authorization being done inside JDBCConnectAuthProvider. Where the credentials are passed to database for authentication purpose and a successful authentication is followed by Authorization using configured user group service.

In current scenario we will replace Database with a configurable Web Service. Since we will be working in AuthorizationProvider, we will not require to parse username from the response as it is already available as part of credentials.

Authorization

Since we are working with Web service here, we can provide option to directly extract roles from web service response as being done in WebServiceBodyResponseUserGroupService.extractRoles. User can configure the regex in GUI and directly parse roles without delegating username to available Role services for simplicity.

For using existing Role services, use of the API calls such as getSecurityManager().listRoleServices() and getSecurityManager().loadRoleService("role_service_name") will allow to obtain a Role service instance which can be used to instantiate a RoleCalculator. Which will be used to obtain authorities /roles.

GUI

Using radio buttons user can navigate between direct role handling in web response and delegating authorization to existing services. Other options include mandatory web service configuration with placeholders and default admin role.

[giohappy]

@imranrajjad thanks for the analysis. I think it is a good plan. I'm not sure if your proposal is to implement both alternatives: extracting of roles from the web response or configuring a roles service.

With regard to this do you confirm that extracting the roles from the web service response is enough? I supposed that a Roles Service was required in any case, and one of the reasons was to instantiate a RoleCalculator.
Can you clarify this point?

[imranrajjad]

@giohappy

1. I propose to implement both. Please forgive my mock of radio buttons.
2. Extracting roles from Web service is indeed enough but since this has to be generic, we would want to consider the case where service does not return roles, in that scenario we will have to use an available role service, for simplicity which could the be the default one as being done here : WebServiceBodyResponseUserGroupService.extractRoles. So yes when extracting roles from web service response, the default role service will come into effect

[nmco]

Thank you for the plan @imranrajjad it matches indeed what we discuss during our meeting 👍.

@giohappy the two options Parse roles from web service and Role service are mutually exclusive, for your use case @giohappy I think you only need the first one. That say as @imranrajjad stayed, the second option will make this solution generic enough to go into core, and I suspect that the effort to implement it will be quite small. We will provide an estimate for both and you can make a final decision @giohappy.

A few technical questions \ comments I have:

• The user name and passwords will be send in clear to the web authentication service if HTTPS is not used, this shoudl be stressed out in the documentation.
• Since the authentication for each request will imply a remote request to another service some caching should be used in the middle, is there already some generic cache in the security system that can be used @imranrajjad? Or is there some session mechanism that will be used? Can you have look around to what's already done in GeoServer security system a clarify this point?

I have create this task sheet for the estimates, @imranrajjad make sure you create tasks for each independent functionality so @giohappy can pick what matches his budget. Take this into account when wrting the estimates:

• Since we are targeting core a GSIP will be required, mention explicitly that we are targeting 2.18-RC, 2.17.1 and 2.16.4.
• Documentation in GeoServer will need be updated with clear instruction in how to use this new security mechanism.
• Integration tests will need to be implemented, this means testing with the GeoServer HTTP mock and caching in between.

[giohappy]

sorry for the insistence but let me see if I understood it correctly.

• in case the (alternative) option to extract users from the service response, the default Roles service will be used under the hood
• otherwise a Roles service can be configured. In this case the webservice doesn't need to return the roles inside the response, because the mapping between user and role will be done by the Roles service.

It's not clear to me why the latter case (which might also be configured to use the default Roles service) doesn't need the roles from the webservice response, while the first requires them (and still uses the default Roles service).

[taba90]

I made a new pull request with a branch that is a checkout from the one in the previous pr, and added some commits.
geoserver#4906
@afabiani and @aaime for the review, one note about this piece of code that adds a ROLE_ prefix to the roles retrieved from the external services: it seems done with full knowledge of the facts, but seems to me that will imply also that will never allow to detect and admin from the response roles if the Geoserver admin role doesn't start with "ROLE_". For the moment I left it as it was.

[giohappy]

@taba90 if I remember correctly the role services retrieve the actual admin role from a specific endpoint (see image below). This endpoint returns the role to be considered a Geoserver admin, independently from the ROLE_* convention.

The convention of the ROLE_* prefix stems from the AuthKey module. It's exaplined inside AuthKEY WebService Body Response UserGroup Service ("Additional Options" sub-paragraph)

[taba90]

thanks for the heads up @giohappy now its clear. Currently there is no possibility to specify a separate regex/path for the admin role in the new community module.
Let me know if this should be implemented. For the moment I've simply limited to make the module ready to be contributed with the functionality already in place.

[nmco]

@taba90 for the moment let's focus on the functionalities already in place (this module if I'm not mistaken was already tested by @giohappy). @giohappy if you would like that extra functionality I would recommend to create a new issue.

[giohappy]

I haven't asked anything :)
A long time has passed since the last tests. Do you want to test it again @taba90 before merging?

[simboss]

@giohappy that is a good idea.

[taba90]

the old pr is still open. If it is meant to be closed can some one do it? (I don't have permissions) geoserver#4483

[taba90]

Attaching here the 2.18.2 jar for test purpose
gs-authbasic-2.18.2.zip

[simboss]

@giohappy please, liaise with @taba90 and maybe michal for testing this before we merge it.

[giohappy]

@simboss test were already done on Friday. Michal confirms that it's working.

[simboss]

perfect.

@taba90 I have not checked the PR but I would want to see clear documentation provided as part of this PR.

If we do not have it, please, open a seprate JIRA and add it in a separate PR.
Make also sure code coverage is enough.

[simboss]

K, I see there is basic documentation but I am not sure it suffices.

We need user documentation so that someone who wants to use this feature does not have to become crazy to understand how it works.

[aaime]

I've reviewed the module, there are three significant issues.
I also agree with @simboss existing documetation is insufficient, got comments about it in my code review, but before submitting it we need to resolve the points below:

Naming

First one is naming: it should not be called basic auth, it's confusing (I see this has been discussed above, but it's worth repeating).

The security subsystem is complicated enough already, without adding extra confusion to existing terminology.

The basic authentication is one of the ways a username/password can be provided to a server, and it's already managed by a core functionality, in the form of an authentication filter.

What the module contains is an authentication provider, which takes an already populated Authentication object (populated by an auth filter) and calls onto an external service to validate the username/password. This is not different from the LDAP authentication or the JDBC connection authentications. See for reference the GeoSolutions training and the GeoServer own documentation

Since the module documentation and the UI labels call it "Web Service authentication", I would call it that way, it's less confusing than "basic auth".

Less secure than basic auth itself

Basic authentication is often critisized by being weak, but at least, it's base64 encoding the username and password to avoid accidental credential discovery by admins monitoring the network and the requests performed on it.

This module instead sticks clear text usename and passwords straigth into the URL or in the headers, with no encoding whatsoever. This is quite bad, the people monitoring the network and services can be trusted, but having usernames and password appear as-is on their screen (where every by-standers can also see them) is unacceptable. Even if the admins were locked in a room alone, they are not necessarily the same people that are managing the user credentials, they should never be presented with them in such a casual way.

Without asking to apply any actual security to it (e.g., encryption) let's at least be as good as basic auth, and base64 encode them? Please?

HTTPS check is too weak

The code logs a warning if HTTP is used instad of HTTPS. Better than nothing, but the admin is playing ona GUI when that happens, not looking at the logs, so they won't see the warning. I recommend having an explicit GUI checkbox to allow HTTP connection usage (handy for quick testing I reckon), that an admin has to manually check in order to save a HTTP link.
Mind, a network might be private, that does not make it secure though, if an intruder sniffin HTTP traffic gets to see the urls with credential going around it's gonna be "jackpot" for them.

[giohappy]

@aaime I totally agree with base64. This actually was required by myself long time ago in this comment. The comments don't collect all the discussion we had in the past on this thing, but I'm in favour of it of course. This will require some small chenages to DJAM for the decoding of a base64 encoded GET param, but it's straightforward. Just le us know how we wanto to name the parameter.

I'm also fine with the forced checkd for HTTP usage.

[taba90]

pr updated

[taba90]

@giohappy the module has been merged on master.
I've applied the fix requested by Andrea:

• the credentials are sent Base64 encoded.
• there is a new option from UI to enable authentication using an http service.
• added a check on the response code that has to be 200 to be authenticated. Before the checks was only on the response body to be not null.
• module renamed.

I've already done some manual tests, but if before doing any backport you prefer to give a quick test another time in your environment, here is a 2.18.2 version of the plug-in.
geoserver-2.18.2-web-service-auth.zip
Let me know if/when I can go ahead and do the backports.

[giohappy]

I will tell Michal inside the chat.

[simboss]

@taba90, if you do not hear from @giohappy shortly, move on with the backport up to 2.18

[nmco]

@taba please backport.

[simboss]

@taba90 we need to go as back as 2.18

Notify @giohappy once done.

[nmco]

Back-ports merged, this will be available on the nightly builds starting tomorrow.

[simboss]

cool, thx.

Let's leave it open until @giohappy decides to start using it.

[giohappy]

@simboss I've opened another issue under support, because I'm not sure if it's something with the Basic Auth module or what else but things are far from working on dev and staging server.

https://github.com/geosolutions-it/support/issues/1419

[simboss]

@nmco as per our discussion, please, ask @zvrablik to help with this

[nmco]

@simboss there is another issue created by @giohappy for the problem in question:
https://github.com/geosolutions-it/support/issues/1419

Zdenek is already working on it.

@giohappy I think this one should be closed.