token based authorization is reccomended/recommended. Why? #193
https://geonovum.github.io/KP-APIs/API-strategie-extensies/#authorisation states token based authentication is recommended, but without providing info why.
I think there should be made a distinction between external API communication and internal communication.
Internal communication is easy: The systems can trust each other, and probably no (state) changes are to be expected, so tokens created during the initiation of the process are still valid during the calling of underlying systems and so don't need to be revalidated in the meantime. Here tokens are the prefered way of distributing authentication/authorization, since they have to be retrieved only once from a authentication server, and can then be distributed across different systems.
External communication is harder: The systems can't trust each other, and you have no control over possible events between requests, like a logout action. So if you use tokens, which are provided by an authorization server and because of the signing the token itself can verified and trusted, you still have to check if the token hasn't been revoked (because the user has been logged out in the meantime). To solve this you can of course reduce the lifetime of the access-token, and use a refresh-token (with a long lifetime) to get a new access-token from the authentication-server, that in its turn can check if the user hasn't logged out in the meantime (now this sounds like a session to me though). Although it's not uncommon to use a refresh token, it does make things somewhat more complicated. Another problem with the refresh-token is that you have to make sure that it doesn't get stolen, since else the bearer can of course gets its own access-tokens from the AS. So developers have to be aware they should handle/store refresh-tokens differently than access-tokens, although they are both tokens.
The text was updated successfully, but these errors were encountered:
In modern web applications, state is managed on the client, where it doesn't matter if it's a browser/mobile/desktop or server app. A session cookie is a browser-specific approach and typically used to maintain state on the server, which is explicitly discouraged in the design rules (https://geonovum.github.io/API-Designrules/#api-02-do-not-maintain-state-information-at-the-server). On the contrary, token-based authorization is meant for stateless invocation of API's, which provides lots of benefits in terms of decoupling, scalability, performance etc. and enables features like delegated authorization and single sign-on (SSO).
Tokens are often stateful (e.g. JWT), which means they can be obtained once and contain all necessary information to verify the token for subsequent requests, without having to reach out to the authorization server every time. In that case, a user sign-out removes the access token from the client and revokes the refresh token on the authorization server. The access token will still remain valid during the remaining lifetime (therefore a short expiry lifetime for access tokens and long lifetime for refresh tokens is a best practice), but the user won't be able to use it anymore since it has been removed from the client. The refresh token cannot be used anymore to obtain new access tokens, since it has been revoked. In the experience of the user, it has been signed out instantly, despite the access token remaining valid for a relatively short time. Usually, this is perfectly acceptable.
If you really want to be able to instantly block access to the API for specific access tokens, you must either validate the token against the authorization server for every request (which increases latency) or otherwise implement some other sort of blocking mechanism in your API gateway, but in most cases this is unnecessary and introduces needless complexity.
Token-based authorization (e.g. OAuth2) really shouldn't be hard to implement or insecure, because there are plenty of client libraries which can help you with that and provide a high level of security. On the server-side I would recommend using standard products or cloud services, like AWS Cognito, Azure AD or OAuth0, which play very well with popular API gateway solutions.
Regarding internal vs external communication, I don't think there should be a large distinction. In a microservices landscape, as the number of services grows, the attack surface increases and security risks will grow. Every service should therefore protect its own boundaries and must therefore consider every client, internal or external, as a possible threat (zero-trust model). Therefore, it's a good practice to protect your internal services by using (micro-)gateways or service mesh.
Thanks for your feedback @joostfarla
I agree with you that tokens have a potential (performance) benefit, under the assumption that it's OK that services can still be accessed for a relative short amount of time after a (forced) logout. That is however exactly my point. the recommendation to use token based authorization has been made under the "security" headline, and the only thing I see is a step backwards security wize regarding tokens, vs sessions unless you check the state every request. Of course you can check the validity of a token every request, just like you have to get the session at every request when you are using session-ids, but in that case session-ids are a lot easier to implement than tokens.
You say that only checking if the session is expired when the access-token has been expired is usually good enough, but I don't think that is a very strong argument, especially under the headline "security", and certainly not without mentioning this assumption. At the dutch government they are working towards DigiD and eiDAS substantial and high, creating a very high level of reliability of identification. The fact that a stolen token can than be used to access all kind of services, without the knowledge at the service if this token has been provided by the logged in user and without the user or some operator being able to revoke the access immediately can't be a good security measure I think.
Regarding internal and external services, I still see a distinction in the fact that request from the outside still have to be validated first (check if access hasn't been revoked at the authorisation server), while internal calls are already validated once and it shouldn't be necessary to recheck this every time (when services call each other directly). An API-gateway can optimize this.
I like the discussion in order to describe/find acceptable session requirements. In the past I've already done some research together with a colleague of mine, and concluded that using tokens introduces some additional security risks compared to "good old" sessions. At OWASP many risks and mitigating solutions have been described in order to secure your (web)applications when using session-ids in cookies, and I cannot see that using JWTs should change anything about how to handle these identifiers, although in practise you see that JWTs are handled completely different (and so much more vulnerable for theft).
In my search for more best practices I found the following blog post regarding the secure use of sessions: https://hackernoon.com/the-best-way-to-securely-manage-user-sessions-91f27eeef460 (please also read part 1) It contains a good overview of different situations, risks and solutions around the aspects of secure session management. It shows how to prevent and identify misuse (based on server-sided sessions, as well as with tokens), with many considerations on how to deal with different requirements, accompanied with an opensource implementation: https://supertokens.io/
In short, prevent token theft, protect against:
Accomplish this by:
See see https://supertokens.io/benefits for a more detailed overview
FIDO 2 might introduce some nice opportunities to bind a session to a user session, but it requires users to have a compatible device with possibly additional security hardware, and a user-agent that supports it. Although the possibilities and benefits look very promising, but for now the adoption of these requirements seems to low.
Apps still have to make sure tokens are handled and stored secure, but in contradiction to web, probably don't have to deal with the complications of unsandboxed scripts having access to your session-identifiers. However the API still have to be able to deal with both web and app requests..
Finally I want to add a link to a presentation of netflix, showing that they too show a similar implementation as I mentioned, regarding a gateway that deals with handling the authentication identifiers for outside requests, before internal services are being reached. https://qconsf.com/system/files/presentation-slides/qconsf2019-satyajit-thadeshwar-user-device-identity-for-microservices-netflix-scale.pdf
I completely agree with Leon van der Ree with his critique on this section of the standard:
I second this for the same reasons given, notably:
I do disagree with a remark made IMHO too lightly:
Practically, in a high traffic API, you can't, as secure token validation has very high costs. I feel that being able to implement a performant API at reasonable cost is important for government APIs. Therefor some form of authentication (and probably authorization) caching will be necessary, practically (re)implementing session IDs.
I do not agree with any reason I've heard so far to advice against session cookies in the standard (as an alternative to session token). I also want to bring another reason to allow session cookies:
Session cookies are better standardized, documented, supported and simple to understand and implement and that makes them (since they have no security downsides, compared to cached session tokens) more secure.
I'm under the impression that some concepts get mixed up in the discussion. Let me try to define the most important ones before proceeding:
A session is a way of persisting data belonging to a given session ID (server-side). Sessions are typically short-lived and used to save temporary state throughout the user's interaction with the web application. Examples could be login state, a shopping cart or form data in a multi-step registration form. Sessions are useful for traditional web applications, where every user interaction triggers a full page (re-)load. On the contrary, modern web applications (a.k.a. Single Page Applications) do not re-load pages, but invoke API calls on the background to store or retrieve data. Sessions violate one of the most important REST principles: stateless. Quote:
Given the definitions above, I'll try to respond to the discussion.
Sessions are not RESTful, because they keep temporary state on the server.
Of course, there is a crucial responsibility of keeping tokens secure and protect the application against XSS. For browsers, that's the current reality. Other environments, like mobile devices, do have secure ways to store sensitive information. Cookies can be protected against XSS, but might still be vulnerable to CSRF attacks.
This depends on the way you look at it. By not storing session information on the server (and thus being stateless), the (resource) server has no notion of a user being "logged in" or not. It only knows how to verify a token. How the user has obtained the token is not important and can be completely decoupled (e.g. OAuth2).
Verifying a JWT is simple; there are plenty of libraries available. Besides that, JWT's do not even have to be verified by the server application itself, since they can be checked by a gateway sitting in front of your app (e.g. a sidecar container). That's a big advantage when having multiple (micro-)services.
If this really is a concern, even with ultra-short-lived tokens, you could implement some kind of revocation list, which can be checked for every request.
As a last note: there is nothing against using sessions for traditional web applications, which perform a page-render on the server-side. I'm just saying they should not be used for API's.
This is good explanation of REST theory and it would be great if that were implementable in a high traffic API. Unfortunately, it is not (due to high CPU load/latency). We need to cache the validation of the token and by your definition create state on the server. Since we do that, we might as well allow a standardized, explicit, popular and proven (arguably more secure) method to cache the validation (session cookies).
@joostfarla We agree on the "stateless" theory, but can we also agree that the theory is not practically implementable for high traffic usage?
NB: I don't only mean the technical (RSA) verification process of the token signature, I mean the actual authentication (and probably authorization) validity (including revocation lists etc).
(edits: trying to clarify while keeping it short and to the point)
I'm not sure what you mean by "caching the validation of the token" or how a session cookie would help with that. Either way, for RESTful API's, access tokens must be verified repeatedly for every invocation and should not rely on a cache. Verifying the signature and validity is not a CPU-intensive task and can easily be performed in a negligible amount of time (<1ms).
Performance and scalability are in fact great benefits of stateless communication and using JWTs. Since the token contains all data attributes (claims) necessary to perform authorization, there's no need to consult a datastore for user details or privileges. And because the token is cryptographically signed, there's no need to consult the token-issuing server to confirm its authenticity. This makes them a perfect fit for low-latency API's in distributed environments.
Having a revocation list complicates things a bit (that's why this should be avoided where possible), but this could still be implemented in a performant manner for high-traffic API's (e.g. by synchronising the list with every gateway instance).
I would recommended reading the following article, which might clarify things:
@joostfarla Thanks for your feedback and for showing that you also know where you are talking about. I am also well aware of the concepts like sessions, cookies, and tokens.
I also think we might say the same thing (or come the the same conclusion).
In this issue, I begin with mentioning that "the article states token based authentication is recommended, but without providing info why". What I mean with this, and what I often see in practice is, that web-developers are using API's in a same way as other (native) app developers, without being aware of the differences and risks their browser environment are placing them in.
I completely agree that an API of a (micro)service, should not be aware of the state/session of the user. However, when an API is providing access to a service, with only a token, than there aren't many developers that will think of introducing some sort of proxy that can mitigate the risks of their (web)environment. And therefor I think that, in a chapter about security, it would be good to mention the risks there when you are using tokens. That are things like XSS in web enviroments, and stolen tokens (because of device theft, or whatever reason there can be), that you might want to be able to revoke.
Ways you can mitigate these issues are complex: in web you should be aware of XSS, which can be mitigated with HTTP-ONLY cookies (containing tokens, or session-ids, so that web-services can act as a proxy to the API, replacing the session ID with the token that is stored server-sided), and by implementing refresh tokens, with all the means necessary to be able to revoke access as soon as necessary.
So in short, the security chapter currently contains to little detail, and I think that with all the info combined in these comments we can improve it a lot.
Thanks for the interesting discussion, @lvanderree @mevdschee! I fully agree that chapter 4.3 is incomplete and inaccurate on this subject. It should substantiate those principles and it should provide guidance and awareness of the security risks. Hopefully the security working group will improve the contents based on the suggestions in this issue. /cc @fterpstra
It looks like there are two different dimensions: the difference between tokens/sessions vs the difference between different means of storing information (e.g. HttpOnly cookies vs JS accessible storage).
I do fully agree with @joostfarla that sessions violate one of the most important REST principles, statelessness, which is an essential enabler for true scalability. It may be required to store some information about provided tokens on the server to accommodate instantaneous log-out / revocation lists, but apart from that, the client should store the session information.
Hello. I am the co-founder and CTO at SuperTokens. Thank you all for checking out our article mentioned by @lvanderree. Since this discussion is open to the public, I assume that my comments are welcome.
Due to the lack of clarity in the terminology, I will first define the various terms:
Stateful vs Stateless in the context of restful API
My view on this is a little controversial, but with good reason.
It's clear that the main purpose of wanting stateless is to scale easily. In order to understand this statement, we must define what are the bottlenecks that prevent scaling, and what does "stateless" actually mean.
What does session stateless actually mean?
There are multiple definitions for this. Some say that one request should not depend on another. Others say it means no session data is stored on the server side.
In case of Opaque tokens, we will have to store all session data on the server side. However, requests can still be "independent" to each other. What independence means is also not clear since, for example, we would want a "cart checkout" request to succeed only if an "insert item in cart" request was made before. This way, they are not independent... So perhaps, Opaque tokens are not stateless.
In case of JWT, we will have to store the JWT signing key on the server. The same argument about independent API calls holds here too. Does this mean JWTs are not "truly" stateless either?
Finally, do we really want to be stateless when it comes to sessions? This will prevent important business analytics like knowing how many devices are logged in, or allowing users to choose which device they would like to log out of (like apple and google do.)
Hence, personally, I have dropped the notion of worrying about RESTful, stateless APIs. Instead, I focus more on what prevents scalability:
What are the bottleneck for scalability?
In most web apps, the biggest latency is caused by network requests in APIs (database / cache calls for example).
An API with 0 network requests will be orders of magnitude faster than an API with 1 network request.
On the other hand, if we have an API with 100 sequential network requests, removing one of them will not really improve speeds noticeably. However, I think it's safe to assume that most APIs have 1 to 3 sequential network calls.
Given the above, the fewer network calls we can make, the better it is. Hence, in deciding which is better for scalability - Opaque tokens or JWTs, we should consider, not statelessness, but the number of network calls your APIs are likely to have.
Opaque or JWTs
There is no right answer here. This blog post that I have written will better be able to explain the difference.
In short, it's possible to scale with Opaque tokens (Facebook doesn't use JWTs) and it's also possible to minimise all the risks of using a JWT. However, it is harder to scale with Opaque tokens, and harder to secure JWTs. In general, I recommend that if your API is expecting a very high amount of traffic, then stick to JWTs since API speed and uptime is more important than security, given that we can secure the APIs extremely well even with JWTs (As mentioned in the blog link above).
One point I would like to bring about is that we talk about wanting to be able to revoke a session immediately in case some anomaly like theft is detected. But what we fail to discuss is how do we actually detect theft? Because only after proper detection (no false negatives / positives), would we even consider revoking the session.
Detecting token theft
The classic methods for this are to use IP address and device fingerprint pattern matching.
If you user's IP address changes "drastically" you revoke the session. However, this can cause issues when a user uses a VPN. Also, an attacker can bypass this by being "roughly" in the same location as the targeted victim.
If the user's device fingerprints change, then you revoke the session. If an attacker is able to steal the session tokens (Opaque or JWT), they will likely also be able to get the computed device fingerprint, rendering this useless.
That being said, both the above methods can be used, but not on their own. We should instead use the concept of rotating refresh tokens (as recommended by IETF in their RFC here) to really detect refresh token theft in a reliable manner.
Access token theft can be detected by the traditional methods, and then the access token can be revoked if necessary. This will in turn result in the users using their refresh token, in which case we can reliably detect theft if it has occurred. If no theft has occurred, then the users will not be logged out.
I am happy to go more into detail of this section
Where to store the tokens?
For browsers, it's recommended to always use httpOnly, secure cookies to store the access and refresh tokens. This will prevent token theft via XSS attack (which can be done in a variety of ways including social engineering - something you have no control over). If using this, we will have to prevent CSRF attacks, but those are completely solved for.
If the token information is to be read on the frontend for business logic, then you should use another token called Open ID Connect tokens, whose only purpose is to be read by the client. This token cannot be used to get access to APIs - it is not the access token. I'll be happy to elaborate more on this.
I would like to recommend that you use:
Some more comments
Delegated authorisation is different to first party authorisation. Delegated authorisation can be done by following the Authorisation code grant flow (which requires the use of JWTs) mentioned in the OAuth spec. However, for the purpose of first party communication, using JWTs or Opauqe is only a matter of scalability vs security.
Single sign on can also be implemented using Opaque tokens.
These products offer user management as a service. They allow you to access and manipulate your user's information via OAuth 2.0 protocol. They are not meant to be used to enable first party communication between your frontend and backend (even though people do that, which is a mistake from a security point of view).
As discussed earlier, we must also consider how we will go about detecting such a theft.
Cookies are transported via headers (
Thank you for taking the time to read the above information. I hope this helped! I'll be happy to answer any questions.
@rishabhpoddar First of all thank you for your extensive and insightful reply.
I have another question: In reality most government API's (I worked with) are provided only (some) authentication claims by some OAuth provider and no authorization grants. Thus leading to a scalability problem if you don't cache the authorization grants (those require expensive look-ups in an authorization service). How would you deal with this situation?
I don't quite understand this sentence. The term "authentication claims" is extremely overloaded in general. It would help if you clarify.
I'm not sure I understand the meaning of "authorization grants"? In the context of OAuth, do you mean access, refresh, ID Connect tokens? Or authorisation code (a token used in Authorisation code grant flow to get the access and refresh tokens)?
Also, is this discussion for the purpose of defining sessions between your system and another system (OAuth), or between your system's backend and frontend? Or both?
Okay.. so in most API systems the JWT claims are either "who you are" (authentication) or "what you are allowed" (authorization). I was using the words "authentication claim" vs "authorization grant" to make the difference between those two types of claims in the JWT. We need an LDAP tree with organizational structure and one with grouped permissions and cross connects between them, defining the "authorization grants" that we are obliged to report twice per year to the "application owner" conform the (Dutch) government laws (BIO / BBN2). Most environments thus have an authentication provider and an authorization provider, both supported by a "tree" of either of organization structure or grouped permissions.
Both, as we are trying to make an API standard for all (Dutch) government APIs, not ruling out any use case.
Thanks for the clarification. I now understand the difference between "authentication claim" and "authorization grant".
Furthermore, I understand that you defining how the Dutch govt's APIs (DGAPIs) are to be accessed, by third party services (any external app that wants to talk to the DGAPIs - for whatever reason) and by first party clients (Dutch govt's own frontend clients, like their website, where users can login and then access DGAPIs).
If the above is correct, then when you say "are provided only (some) authentication claims by some OAuth provider", who are these OAuth providers? Don't DGAPIs store information about its users in its own database?
Regardless, to answer your question, if the authorization grants are not passed in the JWT, then yes, you would have to cache that information. I would recommend having that information as a part of the JWT itself though (if that's possible).
Also to clarify, the OAuth flow that DGAPIs implement can follow the standard OAuth flow - Authorisation code grant flow with PKCE is what the new OAuth 2.1 standard will recommend. On the other hand, for first-party access, OAuth doesn't work as is - you will need to implement something like rotating refresh tokens if you want to maintain long sessions for your first party clients (regardless of the access token type).
I'm not sure if the above answers your query @mevdschee. The system architecture you are talking about is still quite vague in my head. I am happy to use some other method of communication if you'd like.
@rishabhpoddar Thank you again for your extensive reply and especially for explaining the use of OAuth in a complex architecture like that. I guess you are right and we are talking about system architecture and not about API design. Then again we should be careful not to forbid a good system design with too strict API guidelines.
I agree and I believe that that is what @joostfarla was suggesting as well. This means we cannot use authentication providers to directly provide a JWT for our API, but we need to talk to an authorization provider first to enrich the JWT with authorization information.
Sorry for my late reply, and thank you @rishabhpoddar for your insights.
I finally got some time to summarize all provided information and write a recommendation.
I think this discussion showed the following insight:
Tokens are used to provide (among others) authentication and authorization information. This can be done via:
Both alternatives have there pros and cons:
API's should be usable for:
This difference in transport is something that can be handled via the API, by supporting both mechanisms: Allowing for token transport via cookies, as well as via headers. However it is important that the client is aware of the appropriate solution he should implement (which comes down to: web use http-only cookies).
Said that, it is good to realize that there can be a difference between the outside API that your clients should implement, and the API that your (or even external) services describe, by placing something like an API gateway in between! This gateway is responsible for handling the token of your client. It can talk to a session-store, or check for the validity of a JWT and even exchange the token from the client, for another (possibly even refreshed) token which is used to communicatie with the service performing the actual action.
We are potentially looking for solution in which a nation wide Authentication Provider (like DigiD) can provide authentication confirmation about an identity, but I can't imagine this AP can provide all the authorization details for every service in the country. This means additional information has to be added to the user session/token.
Is there any way I (SuperTokens.io) can help you out in building this? We have an open-source project that facilitates exactly that. The project is unique in its ability to handle all the edge case issues when implementing rotating refresh tokens while maintaining scalability.
I'd be happy to chat more about it on a call if that works for you. Else, i'll also be happy to answer questions here - should you be interested in this.
If the above seems spammy, I apologies for it. I would not be suggesting my product unless I genuinely believed it solves the problem you are discussing about (or at least a core part of it). Thanks
Bijgewerkt zoals besproken https://github.com/Geonovum/KP-APIs/blob/master/Werkgroep%20API%20beveiliging/Verslagen/verslag%2013%20november%202020.mdown Gekeken naar https://cheatsheetseries.owasp.org/cheatsheets/REST_Security_Cheat_Sheet.html#validate-content-types en volgende overgenomen: - Security Headers = overgenomen - CORS = overgenomen - Sensitive information in HTTP requests = overgenomen - HTTP Return Code = overgenomen - Sensitive information in HTTP requests = Geonovum#219 - HTTPS = wordt al genoemd in ext, check op aanvullende info - Access Control / JWT = wordt al genoemd in ext, check op aanvullende info + Geonovum#193 - API Keys = wordt al genoemd in ext, check op aanvullende info - Input validation = buiten scope - Management endpoints = buiten scope - Audit logs = buiten scope