Apache Ambari is an open-source management tool that allows you to administer your Hadoop cluster easly. Ambari uses AMS (Ambari Metrics System) to monitor an collect logs from related Hadoop installtions in your cluster, exposing them to the user in each component dashboard. In addition, you may want to export those metrics to a cenralized tool you are using. Grafana has a built in support for that, which is great.
However, you may want to export AMS's collected metrics to a different source like Prometheus. This repo attepts to try and solve this issue, implementing a Prometheus exporter for Hadoop based on Ambari API.
Note - there are open source Prometheus exporters for some Hadoop components like YARN and HDFS, but, Ambari gives you a single point of access to multiple collected metrics for services, components and configuration. Thus our effort building an exporter based on its API.
We're running on Azure environment, using Azure HDInsight cluster 3.6 running HDP 2.6.
The exporter was deployed on our Kubernetes cluster running init-container that handles all secrets acquisition from an Azure Key Vault and injection to our containers.
You can download an image from this DockerHub.
- Clone the project and go to its root folder.
- Run
docker build . -t ambari-exporter, this will create a docker image on your local machine, rundocker images ambari-exporterto validate that the image exists.
Every service requires settings file to run, an example of full settings file can be found here. But, since we're using Kubernetes as our orchestrator, we can leverage helm to inject those values during deployment as environment variables.
Install init-container in your cluster for Azure Key Vault integration. This is used to extract secrets in a secure way.
View an example deployment helm chart here.
All of the exporters are based on Ambari's API.
Implementation dependencies:
- Prometheus-net - used to expose the API endpoint for Prometheus.
- Serilog - for formatting log messages.
- Newtonsoft - for Json parsing.
- FluentAssertions, Moq 4, xUnit - for testing.
- StyleCop - for code formatting.
For any issue, request, question please feel to create an issue, we'll be trying giving you the best support as possible.
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.