These demos are part of a webinar on Ambari.
- Slides and webinar recording available at http://hortonworks.com/partners/learn/#ambari
- Blog article on the topics presented in the webinar is available here
- A higher level blog about the direction of Ambari extensibility is available here
- Ambari roadmap is also available on the wiki
Want to suggest a custom Ambari service or view? Suggest here
Stacks wrap services of all shapes and sizes with a consistent definition and lifecycle-control layer. With this wrapper in-place, Ambari can rationalize operations over a broad set of services. To the Hadoop operator, this means that regardless of differences across services (e.g.install/start/stop/configure/status) each service can be managed and monitored with a consistent approach. This also provides a natural extension point for operators and the community to create their own custom stack definitions to “plug-in” new services that can co-exist with Hadoop Documentation on stacks/services can be found here
| Service name | Github url/author | Difficulty | Description | Comments |
|---|---|---|---|---|
| ntpd | abajwa | Beginner | Deploy/manage time daemon from Ambari | Most basic 'master' service example of how to wrap Ambari service around linux daemon service, using 4 files |
| Maven | randerzander | Beginner | Deploy the bits for this developer tool on a cluster | Most basic 'client' service example of how to use Ambari to install software bits on cluster (w/o starting a daemon service) |
| VNC | abajwa | Easy | deploy VNC service and developer tools | Remote desktop into your sandbox and start coding with Eclipse/IntelliJ on Spark/Storm |
| OpenTSDB | abajwa | Easy | Deploy/monitor timeseries DB on top of HBase | Implement your own Google Finance-like functionality on HDP |
| HDFS Vizualizer | abajwa | Medium | deploy/manage D3.js webapp from Ambari | Inspired by Twitters HDFS du project - see below for view details. Another custom service example |
| Document crawler | abajwa | Medium | deploy/manage Angular.js webapp from Ambari | search through Word, PDF etc docs stored in HDFS via Ambari - see below for view details. Another custom service example |
| Official component services | Apache | Advanced | Browse the code for the HDP services | Kafka and Knox are good examples to follow |
| Service name | Github url/author | Difficulty | Description | Comments |
|---|---|---|---|---|
| R | randerzander | Easy | deploy/manage R from Ambari | Another example of a 'client' service which is useful for data science use cases on Hadoop |
| iPython notebook | randerzander | Medium | deploy/manage iPython notebook from Ambari | Useful for data science use cases on Hadoop |
| Zeppelin notebook | abajwa | Medium | deploy/manage Zeppelin notebook from Ambari | Useful for data science use cases on Hadoop |
| Service name | Github url/author | Difficulty | Description | Comments |
|---|---|---|---|---|
| FreeIPA | abajwa | Easy | deploy/manage FreeIPA LDAP from Ambari | For deploying combined LDAP/KDC for Identity Management |
| Kerberos KDC | abajwa | Easy | deploy/manage KDC from Ambari | For kerberos ticket management |
| NSLCD/SSSD | abajwa | Easy | deploy/manage NSLCD from Ambari | to enable OS to recognize LDAP users |
| OpenLDAP | abajwa | Medium | deploy/manage OpenLDAP from Ambari | For deploying combined LDAP for Identity Management |
- See guide on how to use these services to enable security from Ambari UI, with minimal command line work
- NEW See guide on how to deploy these services via a Blueprint to easily setup a kerborized cluster with LDAP/PAM
| Service name | Github url/author | Difficulty | Description | Comments |
|---|---|---|---|---|
| Solr | abajwa | Easy | deploy/manage Solr from Ambari | Detailed example of a custom Ambari service that uses pidfiles to monitor the service. Will be productized by our partner LucidWorks (see below) |
| Elastic Search | saurabhmishra | Medium | deploy/manage ElasticSearch from Ambari |
| Service name | Github url/author | Difficulty | Description | Comments |
|---|---|---|---|---|
| Tachyon | seraphin | Medium | deploy/manage Tachyon from Ambari | Service to manage Tachyon in cluster mode |
| KPMG Analytics and Visualization Environment | KaveIO | Medium | KPMG Analytics and Visualization Environment | The KAVE combines the power of a Lambda Stack, with a development line and core analysis tools. |
| Solr | LucidWorks | Easy | deploy/manage Solr from Ambari | Will be added to HDP in future |
| Cask | Cask service | Medium | Deploy Cask | |
| MongoDB | nikunjness | Easy | Deploy Mongo | |
| Redis | nikunjness | Easy | Deploy Redis |
- In Ambari 2.1 onwards, custom services can push their metrics to Ambari Metrics service and define their own widgets. Documentation here
The Onyara team recently did this integration for Apache Nifi. Blog and code samples below:
- http://bryanbende.com/development/2015/07/31/ambari-metrics-part1-metrics-collector/
- http://bryanbende.com/development/2015/08/16/ambari-metrics-part2-service-definition/
Code samples:
- Simple metric producer
- Nifi metrics producer
- Nifi metrics/widgets
- HDFS metrics/widgets
Ambari supports services automating kerberos setup. More details available at: https://cwiki.apache.org/confluence/display/AMBARI/Automated+Kerberizaton
- In Ambari 2.1 onwards, custom services can define enhanced configurations (similar to HDFS, YARN etc) to make their configuration panel look native. More details available in wiki
Code samples:
- How to enable debug logging?
In /etc/ambari-agent/conf/ambari-agent.ini set loglevel=DEBUG and run ‘service ambari-agent restart'
- How to test your service outside of Ambari?
After enabling debugging you can see the full command that Ambari invokes for each of the INSTALL, START, STOP operations for your service which can be run from the command line e.g.
While you are installing service via wizard the below will give the full commands to invoke master.py:
ps -ef | grep <servicename>
e.g.
/usr/bin/python2.6 /var/lib/ambari-agent/cache/stacks/HDP/2.3/services/zeppelin-stack/package/scripts/master.py INSTALL /var/lib/ambari-agent/data/command-2502.json /var/lib/ambari-agent/cache/stacks/HDP/2.3/services/zeppelin-stack/package /var/lib/ambari-agent/data/structured-out-2502.json INFO /var/lib/ambari-agent/data/tmp
- In case service install fails, how to delete/unregister it from Ambari?
You will have to manually remove any artifacts created by service. To unregister the service from Ambari, see example here: https://github.com/hortonworks-gallery/ambari-zeppelin-service#remove-zeppelin-service
Views Customizing the Interaction Experience for Operators and Users Ambari Views will enable the community and operators to develop new ways to visualize operations, troubleshoot issues and interact with Hadoop. They will provide a framework to offer those experiences to specific sets of users. Via the pluggable UI framework, operators will be able to control which users get certain capabilities and to customize how those users interact with Hadoop. Documentation on Ambari views can be found here
| View name | Github url/author | Difficulty | Description | Comments |
|---|---|---|---|---|
| iFrame | abajwa | Beginner | embed any webapp within Ambari | Most basic example of how to build an HTML only view, using 3 files |
| REST API Explorer | abajwa | Easy | get started with Ambari REST APIs | Similar to above but also includes basic javascript |
| Hive query | randerzander | Medium | submit Hive SQL from Ambari | Basic example of java servlet view |
| Document Crawler | pcodding | Medium | search through Word, PDF etc docs stored in HDFS via Ambari | Example of view written in Node.js/Angular JS |
| HDFS visualizer | dp1140a | Medium | Navigate HDFS visually using D3 charts via Ambari | Node/D3/Grunt JS webapp that invokes WebHDFS APIs. Inspired by Twitters HDFS du project |
| Phoenix metrics charts | randerzander | Meduim | An example of an Ambari View with a simple UI for polling Phoenix and displaying realtime metrics. | |
| Apache sample views | Apache | Varying | Sample views from Apache Ambari page | Starts with examples of basic html views and advances to java servlet views with configs |
Contributed views below will be TP in upcoming version of Ambari. For setup instructions and screenshots click here
- Hive
- Tez
- Files
- Jobs
- Capacity scheduler
- Slider
- Pig
Ambari Blueprints deliver the below benefits: A repeatable model for cluster provisioning (for consistency); A method to automate cluster provisioning (for ad hoc cluster creation, whether bare metal or cloud); A portable and cohesive definition of a cluster (for sharing best practices on component layout and configuration).
- Links:
- Apache Ambari Wiki for Blueprints
- REST API explorer view to get started with APIs
- Sample automation to install/configure Ambari Agent, Ambari Server and run pre-req checks
- API Examples including:
- Basics, changing password, creating blueprints, deploying clusters, changing configurations
- Guide on how to deploy custom services via a Blueprint to easily setup a kerborized cluster with OpenLDAP/NSLCD/KDC
- Older tutorial