Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enforce tenant labels on rule storage PUT request #219

Merged
merged 6 commits into from
Jan 27, 2022

Conversation

saswatamcode
Copy link
Member

Currently, we enforce tenant labels on rules by adding the labels to them when Observatorium API gets it from the rules storage client i.e, the rules will only have the tenant labels when we get it from the API rules/raw endpoint.

With the recent changes in thanos-rule-syncer and rules-objstore, this means,

  • Observatorium API doesn’t add a tenant label while writing rules to rules-objstore
  • With the listallrules endpoint, thanos-rule-syncer will try to get all rules by hitting rules-objstore directly (so enforcing labels won’t work as we’re no longer relying on the API read path, but read from rules-objstore directly)
  • So the rule file generated by thanos-rule-syncer, which is to be used by Thanos Ruler won’t have any tenant labels, so any resultant series will also not have labels

This PR fixes the above issue, by ensuring that the API adds tenant labels while making the PUT request to the rules storage client and validating the labels when a GET request is made. This way tenant labels are propagated to the rules storage client and then eventually to thanos-rule-syncer and Thanos Ruler.

For more diagrammatic context see MON-2169.

Copy link
Contributor

@jessicalins jessicalins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool! 📐 I think after we merge #198 we'd need to update the tests for the write cases too

var rawRules rules.Rules
if err := yaml.Unmarshal(body, &rawRules); err != nil {
level.Error(rh.logger).Log("msg", "could not unmarshal rules", "err", err.Error())
http.Error(w, "error unmarshaling rules", http.StatusInternalServerError)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
http.Error(w, "error unmarshaling rules", http.StatusInternalServerError)
http.Error(w, "error unmarshalling rules", http.StatusInternalServerError)


body, err = yaml.Marshal(rawRules)
if err != nil {
level.Error(rh.logger).Log("msg", "could not marshal YAML", "err", err.Error())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
level.Error(rh.logger).Log("msg", "could not marshal YAML", "err", err.Error())
level.Error(rh.logger).Log("msg", "could not marshal rules YAML", "err", err.Error())

body, err = yaml.Marshal(rawRules)
if err != nil {
level.Error(rh.logger).Log("msg", "could not marshal YAML", "err", err.Error())
http.Error(w, "error marshaling YAML", http.StatusInternalServerError)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
http.Error(w, "error marshaling YAML", http.StatusInternalServerError)
http.Error(w, "error marshaling rules YAML", http.StatusInternalServerError)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think since we have the same marshaling logic also in the get method maybe we could extract this to a separate function and reuse it in both get/put endpoints? same for the auth logic

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think we can do it for both marshaling and unmarshalling rules!

Copy link
Contributor

@matej-g matej-g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good and great PR description @saswatamcode! Few questions, mostly just curious 😁

api/metrics/v1/rules.go Outdated Show resolved Hide resolved
api/metrics/v1/rules.go Outdated Show resolved Hide resolved
matej-g
matej-g previously approved these changes Jan 26, 2022
Copy link
Contributor

@matej-g matej-g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing more to add 🥳

Copy link
Member

@squat squat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just one comment on continuing to enforce on read path

api/metrics/v1/rules.go Outdated Show resolved Hide resolved
api/metrics/v1/rules.go Outdated Show resolved Hide resolved
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
@saswatamcode
Copy link
Member Author

Let me also update this with the test cases from #198.

Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
test/e2e/configs.go Show resolved Hide resolved
test/e2e/configs.go Show resolved Hide resolved
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Signed-off-by: Saswata Mukherjee <saswataminsta@yahoo.com>
Copy link
Member

@squat squat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @saswatamcode

@matej-g matej-g merged commit efc673c into observatorium:main Jan 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants