Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duplicate or Conflicting X-RateLimit Headers When Upstream is Another Kong Gateway Service with Rate Limit Plugin Set #14353

Open
1 task done
fekitibi opened this issue Mar 11, 2025 · 3 comments · May be fixed by #14359
Open
1 task done

Comments

@fekitibi
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Kong version ($ kong version)

3.9.0

Current Behavior

I have a gateway service which has a rate limiting plugin enabled. If I try to use that service as my upstream in a new service and I enable the rate limiting plugin on that service as well I am receiving duplicate rate limiting headers. It seems like kong appends the newly created rate limiting headers instead of overwriting them. This is confusing as it makes it unsure which headers we would need to use.

Image

Expected Behavior

Either rename the fields of the upstream's rate limit headers, or to strip them completely and only add the rate limiting headers from the plugin of the new service. Otherwise we could also just overwrite the fields specified in the new plugin. Another approach could be to merge the headers and only showcase the shortest limits.

Either of these behaviours would be better than the current one.

Steps To Reproduce

  1. Create a service
  2. Create a route for the new service with a path and add a rate-limiting plugin to it with random config like 10/hour
  3. Create another service and set the upstream to the path of the route we created in the previous step
  4. Add a route to this service with a path and add a rate-limiting plugin to it with a config different than the previously created plugin like 20/hour
  5. Call this newly created path and you will receive duplicate rate limiting headers

Anything else?

If this issue will get accepted to speed up the process I have been playing with the rate-limiting plugin code and if it would help I could open a PR with a solution either to clear the upstream's rate limit plugin headers before adding the new ones, or to just overwrite the newly set ones. If another solution would be preferred I can look into it.

fekitibi added a commit to fekitibi/kong that referenced this issue Mar 12, 2025
When a service's upstream is another service with rate-limiting enabled, enabling a rate-limiting plugin on the service results in duplicate headers. This PR fixes the issue by removing Kong's rate-limiting headers from the upstream response and applying only the headers from the plugin.

Kong#14353
@fekitibi
Copy link
Author

I opened a PR with what I believe is the simplest fix. It removes Kong's rate-limiting headers from the response and applies only the ones configured from the plugin.

fekitibi added a commit to fekitibi/kong that referenced this issue Mar 13, 2025
When a service's upstream is another service with rate-limiting enabled, enabling a rate-limiting plugin on the service results in duplicate headers. This PR fixes the issue by removing Kong's rate-limiting headers from the upstream response and applying only the headers from the plugin.

Kong#14353
@fekitibi fekitibi linked a pull request Mar 13, 2025 that will close this issue
3 tasks
@ProBrian
Copy link
Contributor

@fekitibi , That's a good point that we should handle such case better. However, I think this is not just as simple as removing duplicated headers only. We may need to consider more about how to handle cascaded rate-limitted services: should we respect all rate limiting windows, or the one put more close to downstream, or the one with the minimal limit, etc. We should first decide the behavior of rate limiting, then how to pick(or merge) the rate limiting capacity in headers.

@fekitibi
Copy link
Author

fekitibi commented Mar 17, 2025

Hi @ProBrian !

Here is the thought process behind this solution:

  • Allowing the smallest configured rate-limiting header to override the current headers could confuse users, especially if they see a lower limit than they originally set.
  • With this solution, if the upstream rate limit is lower than the downstream configuration, once we hit the limit, the kong-upstream-status header would return 429. Meanwhile, the rate limit headers would still reflect the configured limits, ensuring that API owners logically investigate upstream rate limiting issues.

I am open to modifying the PR. What would be the fastest way to move forward, do I have to post this topic somewhere else to discuss the best approach or will we just use this thread?

Here is some other ideas:

  • One alternative could be renaming the upstream rate limit headers (e.g., x-upstream-ratelimit-*).
  • Another idea is to introduce a respect-upstream-limit flag (true/false), which would determine whether to use upstream headers if they are set.
  • Similarly, we could add a config option with a boolean parameter for clearing upstream, headers.

Looking forward to your thoughts!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants