Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a receive handler to Telemeter for remote-write #223

Merged
merged 4 commits into from Aug 27, 2019

Conversation

metalmatze
Copy link
Contributor

@metalmatze metalmatze commented Aug 20, 2019

We want to add a remote-write receive handler to Telemeter.
For this we need to add a new authorizer, as we are unable to use JWTs.

Works with a mock authorization-server.
Here's a Prometheus config to test with:

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'telemeter'
    static_configs:
    - targets: ['localhost:9004']

remote_write:
  - url: "http://localhost:9003/metrics/v1/receive"
    bearer_token: 'eyJjbHVzdGVyIjogImU1MGFiNGIwLTgwZjgtNDU3OC05NWVmLWQ0NzI5Njg5NzZhZSIsICJ0b2tlbiI6ICJhIn0K'

TODOs:

  • Add some basic metrics (compare with forward store)

/cc @squat @brancz @kakkoyun

@openshift-ci-robot openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 20, 2019
cmd/telemeter-server/main.go Outdated Show resolved Hide resolved
}

type AuthorizationPayload struct {
Cluster string `json:"cluster"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This'll work for now, but doesn't generally work. We need to rethink this a bit. Not a blocker on this though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack, this is 100% specific to Tollbooth. We’ll need to figure out a way to generalize this and maintain compat with Tollbooth auth

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized that there is a decent way we can keep this generic and not tightly coupled to tollbooth. tollbooth expects all authorization requests to be POST requests with the following body:

{"cluster_id":"<cluster_id>","authorization_token":"<pull_secret_auth_token>"}

We can ensure that the bearer token that prometheus is configured to send matches that format and then we POST that token to the specified URL. This way we don't have to do any custom marshalling/unmarshalling. The only variables then are the authorization URL and the token contents.

The one thing that is funny is that we are taking the bearer token header and using that as the body to of a POST request, but I still think that's better and more generic than what we have today.

cmd/telemeter-server/main.go Outdated Show resolved Hide resolved
res.Body.Close()
}
}()
body, err := ioutil.ReadAll(io.LimitReader(res.Body, 32*1024))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make 32*1024 a constant. :)

@squat
Copy link
Contributor

squat commented Aug 27, 2019

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 27, 2019
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: metalmatze, squat

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 27, 2019
@squat
Copy link
Contributor

squat commented Aug 27, 2019

/retest

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 1049e93 into openshift:master Aug 27, 2019
@metalmatze metalmatze deleted the receive branch August 28, 2019 08:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants