Prometheus metrics path /minio/v2/metrics/cluster returns 403 with generated bearer token

[pdf]

Expected Behavior

As documented in the README, the default authentication mode for this path is JWT, and a bearer token with config snippet can be generated via mc admin prometheus generate <alias>, to authenticate against the endpoint when JWT auth is enabled.

Current Behavior

After generating the config snippet and adding it to the prometheus configuration, the scraper fails, indicating a 403 error. Attempting to authenticate using the generated bearer token via the CLI confirms that it is not possible to authenticate with this token (sanitized):

$ curl -v -sSL -H "Authorization: Bearer ${TOKEN}" http://localhost:9000/minio/v2/metrics/cluster
*   Trying ::1:9000...
* TCP_NODELAY set
* Connected to localhost (::1) port 9000 (#0)
> GET /minio/v2/metrics/cluster HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/7.68.0
> Accept: */*
> Authorization: Bearer <token>
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Content-Security-Policy: block-all-mixed-content
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Vary: Origin
< X-Amz-Request-Id: 169BC0953F19E447
< X-Content-Type-Options: nosniff
< X-Xss-Protection: 1; mode=block
< Date: Mon, 16 Aug 2021 09:50:37 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact

Possible Solution

Setting MINIO_PROMETHEUS_AUTH_TYPE="public" allows access to the metrics by disabling authorization, but this is undesirable.

Steps to Reproduce (for bugs)

1. Install minio cluster
2. Leave MINIO_PROMETHEUS_AUTH_TYPE unset, or explicitly set to jwt
3. Generate bearer token via mc admin prometheus generate
4. Attempt to authenticate against /minio/v2/metrics/cluster using token
5. Receive 403 error

Your Environment

• Version used: RELEASE.2021-08-05T22-01-19Z
• Operating System: Ubuntu 20.04 Linux 5.4.0-77-generic SMP x86_64 x86_64 x86_64 GNU/Linux

[harshavardhana]

As documented in the README, the default authentication mode for this path is JWT, and a bearer token with config snippet can be generated via mc admin prometheus generate <alias>, to authenticate against the endpoint when JWT auth is enabled.

The user that is configured mc alias ls <alias> needs to have sufficient permissions to access Prometheus if you need a user who needs to have permissions with Actions: "admin:Prometheus" in your policy.

I have been generating and testing this locally works fine all scenarios - just use the credentials that has sufficient permissions.

[pdf]

The user that is configured mc alias ls <alias> needs to have sufficient permissions to access Prometheus if you need a user who needs to have permissions with Actions: "admin:Prometheus" in your policy.

This is not mentioned in the documentation anywhere, that should probably be fixed. Not only because this permission is required, but also because it's now clear to me that the bearer token does not have a policy applied to it to restrict permissions to only what's required for the metrics endpoint, making it a security concern.

That said, I've done further testing based on your feedback, and the problem appears to be that if you're logged in via a service account, no matter what permissions you have, the generated bearer token is unable to authenticate. It does work if you're logged in as the parent account.

[harshavardhana]

That said, I've done further testing based on your feedback, and the problem appears to be that if you're logged in via a service account, no matter what permissions you have, the generated bearer token is unable to authenticate. It does work if you're logged in as the parent account.

Service Accounts cannot be used and they are never to be used here. Service Accounts are meant for S3 API access, authenticate with a real parent user for Prometheus.

We may extend support for that in the future.

[pdf]

Service Accounts cannot be used and they are never to be used here.

Again, entirely undocumented, and very confusing. An error message on generate in this case would be welcomed.

[harshavardhana]

Again, entirely undocumented, and very confusing.

We never documented the way you are trying to use it and do not intend to until it's finalized in terms of behavior, so not sure where you found the idea to use it in this manner.

[pdf]

not sure where you found the idea to use it in this manner.

I didn't "find the idea", I just happened to be authenticating from the CLI via a service token, which does not seem that unusual. If there are requirements for a feature to work, either documenting what they are, or including error handling to prevent non-functional operation would greatly improve the user experience.

[liuxuzxx]

not sure where you found the idea to use it in this manner.

I didn't "find the idea", I just happened to be authenticating from the CLI via a service token, which does not seem that unusual. If there are requirements for a feature to work, either documenting what they are, or including error handling to prevent non-functional operation would greatly improve the user experience.

when use /minio/prometheus/metrics replace /minio/v2/metrics/cluster ,it is ok!

curl -v -sSL -H 'Authorization: Bearer ${TOKEN}' http://IP:9000/minio/prometheus/metrics

but the metrics data is not adapt to https://grafana.com/grafana/dashboards/13502

[zachfi]

I'm still confused what the fix is here. I'm trying to use the generated config, but I'm pretty new to minio and don't understand the aliases. Are the credentials specific to the alias? I was using the local alias, but trying to access it by the hostname of the system. I receive a 403 when using the curl command in the original post.

[mshanmu]

I'm still confused what the fix is here. I'm trying to use the generated config, but I'm pretty new to minio and don't understand the aliases. Are the credentials specific to the alias? I was using the local alias, but trying to access it by the hostname of the system. I receive a 403 when using the curl command in the original post.

1. Create a user with policy having support for "admin:Prometheus" action.
2. Then do, mc alias set <alias name> <username> <password>
3. Use this alias to generate the jwt token, mc admin prometheus generate <alias name>

[zachfi]

Thank you that worked. Is an alias just a credential set then?

[mshanmu]

Thank you that worked. Is an alias just a credential set then?

yes