Question: Operational Metrics and CLI Tools for Maintenance Tasks

[pulchart]

I'm evaluating RustFS as a potential alternative to MinIO for S3-compatible object storage. While RustFS shows great promise, I'm facing challenges with operational monitoring and maintenance/admin tooling/API.

---

What MinIO Provides (as Reference)

MinIO offers comprehensive operational tools through mc admin:

• Cluster information with disk details
  mc admin info minio --json

• Healing status and operations
  mc admin heal minio/ --verbose --all-drives

• Scanner metrics
  mc admin scanner status minio

• Cluster metrics endpoint
  curl http://minio:9000/minio/v2/metrics/cluster

• Configuration options
  mc admin config ...

Note: I'm using MinIO as a reference point only. I'm completely open to different approaches - whether that's REST APIs, custom CLI tools, extended OTEL metrics, or any other method that RustFS developers prefer. I just need some way to access this operational data programmatically.

RustFS

I cannot find any equivalent for RustFS.

---

Questions:

1. Does RustFS have operational monitoring/admin APIs or CLI tools?
2. If not, is this on the roadmap?
3. What's the recommended way to monitor disk health, healing operations, and cluster status in production?

Any guidance would be appreciated!

[loverustfs]

Here are the answers to your questions regarding operational monitoring and admin tools in RustFS:

1. Current Tooling vs. Console

We currently do not have a dedicated command-line tool similar to the MinIO Client (mc). At this stage, you can view the corresponding operational data and perform management functions directly through the RustFS Console.

2. mc admin Compatibility & Workaround

Regarding direct compatibility with mc admin: Since MinIO is released under AGPL and commercial licenses, we strictly avoid any secondary development or modification based on their codebase to prevent legal risks.

However, if you specifically need to utilize mc admin info functionality with RustFS, there is a workaround. You can compile the mc source code yourself and configure it to point to the RustFS admin endpoint by exporting the following environment variable

export MC_ADMIN_URL_PREFIX=/rustfs/admin
This will allow you to achieve the functionality of mc admin info.

3. Future Roadmap

We have plans to develop a dedicated rustfs-client tool. This is scheduled to be developed and released after the RustFS General Availability (GA) version is launched.

[pulchart]

Thanks for the explanation. Just to clarify: I don’t need a MinIO client or compatibility, the mc admin is only a reference for the type of operational visibility. A REST Admin API is fully sufficient for my use case (healing status, configuration) and later could be used by a Terraform provider to automate administration.

The mentioned environment variable doesn’t seem directly related to what mc admin uses. Could you please clarify its purpose and how it should be used, ideally with a small example?

[loverustfs]

Thanks for the explanation. Just to clarify: I don’t need a MinIO client or compatibility, the mc admin is only a reference for the type of operational visibility. A REST Admin API is fully sufficient for my use case (healing status, configuration) and later could be used by a Terraform provider to automate administration.

The mentioned environment variable doesn’t seem directly related to what mc admin uses. Could you please clarify its purpose and how it should be used, ideally with a small example?

You use the token to request these three API:

https://play.rustfs.com/rustfs/admin/v3/info

https://play.rustfs.com/rustfs/admin/v3/datausageinfo

https://play.rustfs.com/rustfs/admin/v3/storageinfo

[alkdrf]

how to use this endpoints? Using with bearer token/scrape-config generated by "mc admin prometheus generate rustfs-storage" on prometheus didn't work?

  - job_name: rustfs-datausage-job
    bearer_token: ey***Q
    metrics_path: /rustfs/admin/v3/datausageinfo
    scheme: http
    static_configs:
      - targets: ['192.168.1.11:9000']

[loverustfs]

how to use this endpoints? Using with bearer token/scrape-config generated by "mc admin prometheus generate rustfs-storage" on prometheus didn't work?

  - job_name: rustfs-datausage-job
    bearer_token: ey***Q
    metrics_path: /rustfs/admin/v3/datausageinfo
    scheme: http
    static_configs:
      - targets: ['192.168.1.11:9000']

A user has raised the same question.

Prometheus is another observable issue; you can create another issue to express your concerns.

https://github.com/orgs/rustfs/discussions/276

#796

[Krasobas]

@pulchart hello, have you figured out how to use rest api and how to generate prometheus token ?

[perfectra1n]

I know the docs say:

RustFS exports a wide range of fine-grained hardware and software metrics through Prometheus-compatible metrics endpoints. RustFS includes a storage monitoring dashboard that uses Grafana to visualize collected metrics.

But this isn't true from the code that I've read up and down. Currently, to obtain Prometheus metrics, you'd need to spin up an OpenTelemetry Collector.

Within the Collector you'd need to expose the Collector to return Prometheus metrics like:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"  # Scrape this
    namespace: "metrics"

You can then configure the environment variable RUSTFS_OBS_ENDPOINT to be the URL to the OTLP (e.g. http://otel-collector:4318).

Then you can set up a scrape job within your Prometheus:

scrape_configs:
  - job_name: 'rustfs-app-metrics'
    static_configs:
      - targets: ['otel-collector:8889']

Based on the docs, and reading through the GitHub Issues, it looks like specifically metrics is a WIP. I'll try to put a PR together to see if the maintainers are interested.

[loverustfs]

I know the docs say:

RustFS exports a wide range of fine-grained hardware and software metrics through Prometheus-compatible metrics endpoints. RustFS includes a storage monitoring dashboard that uses Grafana to visualize collected metrics.

But this isn't true from the code that I've read up and down. Currently, to obtain Prometheus metrics, you'd need to spin up an OpenTelemetry Collector.

Within the Collector you'd need to expose the Collector to return Prometheus metrics like:

exporters:
prometheus:
endpoint: "0.0.0.0:8889" # Scrape this
namespace: "metrics"
You can then configure the environment variable RUSTFS_OBS_ENDPOINT to be the URL to the OTLP (e.g. http://otel-collector:4318).

Then you can set up a scrape job within your Prometheus:

scrape_configs:

• job_name: 'rustfs-app-metrics'
  static_configs:
  • targets: ['otel-collector:8889']
    Based on the docs, and reading through the GitHub Issues, it looks like specifically metrics is a WIP. I'll try to put a PR together to see if the maintainers are interested.

Hello @perfectra1n ,

Of course, your PR is very welcome! Thank you for your contribution.

@houseme

[houseme]

I know the docs say:

RustFS exports a wide range of fine-grained hardware and software metrics through Prometheus-compatible metrics endpoints. RustFS includes a storage monitoring dashboard that uses Grafana to visualize collected metrics.

But this isn't true from the code that I've read up and down. Currently, to obtain Prometheus metrics, you'd need to spin up an OpenTelemetry Collector.

Within the Collector you'd need to expose the Collector to return Prometheus metrics like:

exporters:
prometheus:
endpoint: "0.0.0.0:8889" # Scrape this
namespace: "metrics"
You can then configure the environment variable RUSTFS_OBS_ENDPOINT to be the URL to the OTLP (e.g. http://otel-collector:4318).

Then you can set up a scrape job within your Prometheus:

scrape_configs:

• job_name: 'rustfs-app-metrics'
  static_configs:
  • targets: ['otel-collector:8889']
    Based on the docs, and reading through the GitHub Issues, it looks like specifically metrics is a WIP. I'll try to put a PR together to see if the maintainers are interested.

Hello @perfectra1n,

Thank you for your detailed explanation and for attempting to integrate Prometheus with RustFS using OpenTelemetry. Your findings and proposed solution are valuable contributions to the community. It's great that you've identified the current workflow involving the OpenTelemetry Collector and the necessary configuration steps.

I appreciate your proactive approach in trying to implement Prometheus metrics and your willingness to submit a Pull Request (PR) to explore this further. The documentation you've referenced and your practical insights will undoubtedly be helpful for other developers and for the RustFS team as they continue to develop and refine their monitoring capabilities.

Please proceed with your PR, and we look forward to reviewing your work. If you encounter any further challenges or have additional insights to share, feel free to update this thread or create a new issue.

Using Prometheus as your OpenTelemetry backend
https://prometheus.io/docs/guides/opentelemetry/#send-opentelemetry-metrics-to-the-prometheus-server

This document provides instructions on how to directly connect with the collection of indicators.

Best regards,
houseme

[houseme]

@perfectra1n Please refer to this document: https://prometheus.io/docs/guides/opentelemetry/#send-opentelemetry-metrics-to-the-prometheus-server, to learn how to connect directly to metric collection. This will provide you with more detailed information on how to configure Prometheus as your OpenTelemetry backend.