-
Notifications
You must be signed in to change notification settings - Fork 484
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
remove the forwarding feature #4876
Conversation
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for taking care of this. We should mention it in the changelog too.
This wasn't part of the experimental or deprecated features listed in about-versioning.md. Should we wait two minor releases before removing it? |
I don't have a strong opinion on this, but this feature was always meant for our internal usage only, and we never documented how it works or what expectations are around it. Original forwarding feature is not even mentioned in the changelog, although improvements to it and bugfixes are. |
All the fields in the |
In that case mentioning the removal in the changelog might make sense. There have been some issues related to forwarding and I assume some users are relying it. Some paper trail in the changelog is better than having to dig through pull requests. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Mauro!
Few things I've noticed:
- The forwarding feature was mentioned few times in the CHANGELOG in the past. Should we add a CHANGELOG entry to mention this experimental feature has been removed?
- Shouldn't we cleanup
ForwardingRules
,ForwardingEndpoint
andForwardingDropOlderThan
in the limits config too?
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
Thx for all the feedback. I added a CHANGELOG entry, as requested by multiple people.
|
Thx Marco, I missed that, cleaning it up. |
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
The type |
Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
I think I have addressed all the feedback, PTAL |
Other changes which I'd like to make are blocked by this. |
Hello, I was atually thinking of using this feature to send some of the data send to one of my mimir instance to another mimir but as this is removed this is not the way to go :D How would you configure it then as mimir does not support sending data via remote write? Use envoy mirroring as described https://grafana.com/docs/mimir/latest/configure/mirror-requests-to-a-second-cluster/ ? Or did I miss that mimir can be configured to send data to another location usign remote write? |
The documentation also mentions that the requests to the secondary cluster are "fire-and-forget", you have no guarantee that they succeeded, so if you're planning to mirror a production workload I don't think this is recommendable.
Mimir has not functionality to remote write to a second location, I'd recommend configuring the remote_write client to write to two separate endpoints instead. That way you can be sure that failed requests will be retried in each of the two cluster separately. |
@replay thank you for the feedback. I wanted to avoid having to pass the credentials to the customer's clusters to avoid secret from being accessible by cluster users but I guess there is no way other than having 2 remote_write configurations, thank you |
If the problem is the credentials, you could always set up a reverse proxy which has endpoints for both of your clusters and which sets the authentication header when connecting to the Mimir clusters. The remote_write clients which connect to the reverse proxy could use different credentials to auth themselves to the reverse proxy. I don't have a recommendation which reverse proxy to use, i think that most likely Nginx should be able to do that, but probably there are many others too. |
That is a really good point, thank you :) |
We don't need the distributor's forwarding feature anymore, this removes it.