Feature Request: Rate limiting and Scoring #2151
Comments
dsandip
added
k/ideas
|
I'd second this. You could easily imagine malicious queries taking down APIs powered by Hasura. {
todos {
user {
todos {
id
user {
id
todos {
id
}
}
}
}
}
}For the record, there was some discussion around this in #346, #989 and #1283 but they don't seem to have lead anywhere. |
|
That's good to hear, haven't seen that before. |
I agree. Maybe starting with depth limiting would be easier and have relatively great impact on stopping malicious queries. |
|
Wish List |
|
@ptrobert Short-term, most of those could be solved using the webhook auth method. |
|
@mfdeux currently hasura is not forwarding the graphql query name/query details. If I want to do rate limit on insertions to a certain collection/table, how can we do it with webhook? |
|
Ok, I understand. A short term hack I’m using is running through a proxy
and parsing the query (using JS or Go), and then implementing rate limiting
based on that, but obviously not a long term soution.
…On Sat, Aug 3, 2019 at 21:25 Tianxiang Chen ***@***.***> wrote:
@mfdeux <https://github.com/mfdeux> currently hasura is not forwarding
the graphql query name/query details. If I want to do rate limit on
insertions to a certain collection/table, how can we do it with webhook?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2151?email_source=notifications&email_token=AEZLJ2I6FWS4YSGVCUVXGULQCYVYVA5CNFSM4HL2LKZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3PYTBI#issuecomment-517966213>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEZLJ2OFJZMKZFNSHTAE623QCYVYVANCNFSM4HL2LKZQ>
.
|
|
Yes, that can be a solution, but not very easy to use :) |
|
opened a feature request for passing hasura query/mutation into webhook auth post body #2666 |
|
I found this, it may provide some good ideas for complexity analysis: |
|
|
Has anyone tried to pass through the Postgres Even if it's off by orders of magnitude, being able to configure a |
|
This would be so valuable for us! |
|
@coco98 can you share your plans whether you will implement rate limiting in some form in the open source version of Hasura, or will it be a Hasura Pro option only (https://www.youtube.com/watch?v=JS6eMQ6H7eA)? Thanks! |
|
@beepsoft, I didn't see that until I read your message.
I hope Hasura is not becoming a "fake open source". It's understandable that offering paid services and paid support. However, splitting software features as an open source vs. pro seems to me not "open source". Worst of all, offering a fundamental feature such as rate limiting as a "pro" feature... Especially considering without rate limiting, a public facing graphql server is very vulnerable to attacks. This is just my 2𝇍 as a developer, who publishes and contributes lots of open source projects. |
|
Hi folks. We’ll always make sure you can run Hasura in production! Most of our time is and will be spent on open-source work. Hasura Pro add-ons need Hasura (open-source) to have the required building blocks for rate-limiting (for eg: how allow lists work). We’ll be putting up the spec and raising the PR in a bit. Also, I'm putting together a blogpost to answer how we will manage Hasura open-source soon too. :) |
@coco98 Does this mean that the open-source version of Hasura will definitely have some form of rate-limiting and this will be extended for the pro version? |
|
Any progress on this? |
|
Looks like they've added rate limiting as a Pro-only feature 😞 |
|
From https://hasura.io/
@coco98 said:
...and this: I agree with @tspike: 😞 |
|
@beepsoft I hoped as you thought. However this case has been open for a year and rate limiting seems already be available in "Pro" 🙄 version for months.
I'm not trying to be rude, but IMHO this sentence does not mean open source version gets rate-limiting. I think, it means we are using open source version and all PRs as a base for our closed version. |
|
@ozum I would really like this feature too and although I would not be too happy if it was only available in the Pro version, I would much appreciate a clean statement from Tanmai or someone from Hasua regarding its availability. So, @coco98, @marionschleifer will we have rate limiting in the OS version or will it be a Pro only feature? And as always: thank you very much for your efforts! |
|
I believe this issue #2269 is the only thing blocking the ability to setup rate limiting via authhook. If Hasura enabled the ability for an authhook to set and pass headers back to the client in response, we could use session cookies to manage rate limiting manually. |
|
Thanks @coco98 for the detailed explanation! Allow list seems to be a viable solution for now. My only problem with it is that it is rather difficult to collect all the queries my various modules send to Hasura. Is there some support (planned) for this? For example, a special mode of Hasura - like the migration mode now - which records all the queries sent to Hasura and at the end with a click of a button or a cli call would generate an allow list from the collected queries. I guess this could also be implemented with eg. Apollo acting as a gateway for Hasura and collecting the queries. |
|
@beepsoft, not directly what you want, but I saw this library. It may help: https://github.com/apollographql/persistgraphql |
|
@ozum analyzing .graphql files is a good idea, thanks! I also have dynamically generated Hasura queries and mutations in which case I can only identify those either when I actually send them to Hasura or when received by Hasura or something in between me and Hasura. I feel there could be existing solutions for this already. Maybe one of these? https://gitlab.com/arboric/arboric |
|
@beepsoft they seem nice and also be useful for other purposes. AFAIK, white listing requires your queries are fixed before Hasura schema is built, and only GraphQL variables can be changed at runtime. Are you sure you could use white listing for dynamically generated queries? @coco98, could better answer that, if I'm wrong. |
|
@ozum My use case is a (would be) generic form system, which can work with any Hasura schema + some other configurations and generate forms to edit the entities of that schema. So practically these will be the same queries/mutations for all entities but with their specific types and fields included in the query/mutation expressions. Then I would have my tests, which force generation of all the queries possible for the given schema and I would like to have those collected in the allow lists. Actually I start off from Java/JPA and generate Hasura customizations based on it (using https://github.com/beepsoft/hasuraconf) and now working on adding also json-schema generation (https://github.com/victools/jsonschema-generator) with some custom fields (eg. to describe the relationship between entities, add field validation information, etc.): things that cannot be derived from the Hasura graphql schema itself. Then I have mst-gql (https://github.com/mobxjs/mst-gql) to generate state store and graphql communication handling and will eventually have my components to actually display forms using all the above. And then, in production it would be great to have allow lists generated automatically. :-) |
|
@coco98 thanks for the update on this looking forward to this feature in the OSS version |
|
Hi @coco98 any update or timeline when this will be pushed into the OSS version |
|
While I'm fully agree with @mitar 's comment, I think a Query Cost Analysis (or at least a Depth Limiting) is crucial to every GraphQL implementation. It's because an attacker must not have a knowledge about the application. Just use introspection to find a cycle. Then the attacker doesn't need a bunch of resources like a botnet. It's just a matter of some copy&paste. And every such query is able to bring the database and therefore the whole application down. Just a single query issued from a mobile phone through 2G network is enough. In the GraphQL world the rate limiting doesn't make any sense, because you can fetch the whole object graph regardless you need the data or not. It's become very expensive for the service/api provider (shot in own foot). Rate limiting is a thing to bill partners for usage of RESTlike endpoints, even in a wrong way. So, it's really an enterprise feature. But it can be practicable together with the deep limiting. @arpitjacob just for rate limiting look at something like nginx. Nginx is very lightweight and powerful. BTW, I think the auth-hook can be misused for the rate limiting. Proposed allow-listing isn't practicable for most of the applications. It leads GraphQL ad absurdum. On the one side you should fetch the data you need only (sold as underfetchin/overfetching solver), but on the other side you are forced to use exact predefined queries. It's even worse: "The order of fields in a query will be strictly compared.". W00t? And think about that: how you will manage old apps which are not updated by users (=uncontrolled environments)? For a nearly every small change you are forced to add a new copy of definition to allow list and you can't just delete the old one. From the fist post you can navigate to the graphql-cost-analysis project. I like the idea to not to try some cost heuristics, but allowing developers to specify arbitrary cost values. It's powerful and makes the implementation easier. The integration examples are very easy with apollo engine and others. But it seems impossible to integrate it with hasura, because hasura has fundamentally another view on architecture. I see different ways to approach the problem. The worst one is to try to implement something like GraphQL proxy which can be instrumented with costs on queries (and mutations?). But it leads to double maintenance and potentially expose the structure by bypassing permissions on introspection. But it could work for rate limiting and depth limiting. It's possible, that there are already some usable/lightweight solutions. The better two are to implement a similar functionality like in graphql-cost-analysis and/or allows to specify and process custom directives. I think the latter one is fully compatible with community vs. enterprise versions of hasura: for the community we can build own cost limiter and enterprise version can offer one out of the box. I understand, that hasura must make money. But I hope there is a place for an open source solution to problems with a huge impact. |
|
So current business model is that you get unrestricted queries for free, but for restricted queries you have to pay. Would people prefer that Hasura has a different business model where you get restricted hard-coded queries of depth up to 2 for free, but for unrestricted queries or for other cost metrics you have to pay? |
|
They are offering the following feature on their Free Tier.
I don't want to debate or complain, this is such a key features and useful to have it in the Open Source Version. I hope they can push this to the Open Source Version. |
I'm not sure what do you mean by "restricted hard-coded queries". Personally, I very like following business model: free for free AND open source databases (postgres, mariadb, yugabytedb, cockroachdb core) with optional support subscription and billed for commercial databases (enterprisedb, yugabyte cloud/plattform, cockroachdb enterprise/cloud, MS SQL Server, oracle, db2). |
|
Does anyone already have a solution on how to add rate-limiting to the open-source version? I totally understand @mitar's point and that here maybe is not the best place to discuss Hasura's business model. But I also agree that API rate-limiting is such a basic security concept that no tool should be considered |
Hasura Container + Fail2Ban on host? |
|
Simple rate limiting may be doable with POST webhook since Hasura v2.0, where auth service can see graphql query and decide to permit or forbid it. Query scoring might be doable with such webhook + external auth service too. While I personally think query allowlist could be an option too: at least it is simpler. (I'm using React + Relay which has an option to export query texts) |
This approach is kind of a hack since the webhook is meant to be used as authentication endpoint. It would be more useful to have a pre-execute hook which is triggered after authentication but before graphql-engine execution. Several things should be possible in this webhook like rejecting the query, validating the input and calculating the cost. Even extending the query could be an option..., see https://github.com/mercurius-js/mercurius/blob/master/docs/hooks.md#preexecution for a possible call signature |
|
Hasura is an awesome project! Just provide a workaround way. For rate limiting, here is a way using Traefik as reverse proxy in a sidecar container. Hopefully save some time for the people in future! For scoring, as Traefik provides many middlewares, and also can write new middleware and plug in, technically we can write a middleware. It would be great if someone write one and publish so everyone can use : )
|
|
If people are still wondering how to secure and possibly get cdn caching with hasura, checkout the article I wrote on how to use edge functions (vercel or cloudflare workers) to create a reverse proxy to improve the security: https://ilijanl.hashnode.dev/how-to-secure-and-make-graphql-blazingly-fast Another alternative is wundergraph |
|
We only recently found out that this project is a "fake open source", I wish we found out about it sooner. Turns out it is trivial to DoS hasura using a simple "query depth attack" and there is no depth limit to prevent that, this should be a basic thing to have for GQL database, but alas. |
|
@kurevin This isn't true. You can already pin your queries and avoid many attacks, including the mentioned one. The Hasura team does a great job making it clear which features are available in the free and open-source version and which are not. Fundamentally, they need to generate revenue to pay developers on a daily basis. Why don't you work for free full-time? Therefore, some features are provided for free and some are not. Just because you want it for free doesn't mean that it's a basic function. At its core, Hasura is still an amazing product in the free version. BTW, I am a very happy free version user. |
|
Our company has also successfully built on top of the Hasura community version for the last 4 years and its been an amazing experience. If you want to prevent query depth attacks just configure your allow list to only permit the operations that your application actually needs: https://hasura.io/learn/graphql/hasura-advanced/security/3-allow-list/ |
It is not a basic function, but a critical vulnerability that was not fixed in last 4 years. Independent pen test confirmed that. I would not care if that was just an optional functionality. Its like you have forgotten what you wrote almost 4 years ago... #2151 (comment)
Like it was already mentioned in this issue - allow list is an inflexible nuclear option, not practical in most projects, and almost unusable when your product changes and grows rapidly due to massive overhead, depth limiting does the same job with zero overhead from the dev team. We are planning to explore GraphQL proxy solutions, like https://www.graphql-portal.com to add basic depth limiting and\or cost analysis and maybe extend some functionality. At least rate limiting was never a problem for us as there are enough tools to implement rate limit outside of hasura, same cannot be said for depth limiting or query cost as it is GraphQL specific. |
I haven't forgotten, and I'd love to have this functionality too, so we can eliminate workarounds. The distinction here is that I don't refer to it as a "fake open source" project. I don't criticize product management for choosing not to include it in the version available for free. I am genuinely grateful for such an amazing product. |
and others
Would like to have rate limiting (# per user per second) and scoring of query cost (score per user per minute etc.).
Below points mentioned in https://blog.apollographql.com/securing-your-graphql-api-from-malicious-queries-16130a324a6b seems to nice to have in Hasura.
Kind Regards,
The text was updated successfully, but these errors were encountered: