Branch: master
Find file Copy path
471 lines (404 sloc) 15 KB

Apache Griffin Deployment Guide

For Apache Griffin users, please follow the instructions below to deploy Apache Griffin in your environment. Note that there are some dependencies that should be installed firstly.


Firstly you need to install and configure following software products, here we use ubuntu-18.10 as sample OS to prepare all dependencies.

# put all download packages into /apache folder
$ mkdir /home/user/software
$ sudo ln -s /home/user/software /apache
$ sudo ln -s /apache/data /data
  • JDK (1.8 or later versions)
$ sudo apt install openjdk-8-jre-headless

$ java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-0ubuntu0.18.10.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
  • PostgreSQL(version 10.4) or MySQL(version 8.0.11)
# PostgreSQL
$ sudo apt install postgresql-10

$ sudo apt install mysql-server-5.7
  • npm (version 6.0.0+)
$ sudo apt install nodejs
$ sudo apt install npm
$ node -v
$ npm -v
  • Hadoop (2.6.0 or later), you can get some help here.

  • Hive (version 2.x), you can get some help here.

  • Spark (version 2.2.1), if you want to install Pseudo Distributed/Single Node Cluster, you can get some help here.

  • Livy, you can get some help here.

  • ElasticSearch (5.0 or later versions). ElasticSearch works as a metrics collector, Apache Griffin produces metrics into it, and our default UI gets metrics from it, you can use them by your own way as well.



Create database 'quartz' in PostgreSQL

createdb -O <username> quartz

Init quartz tables in PostgreSQL using Init_quartz_postgres.sql

psql -p <port> -h <host address> -U <username> -f Init_quartz_postgres.sql quartz


Create database 'quartz' in MySQL

mysql -u <username> -e "create database quartz" -p

Init quartz tables in MySQL using Init_quartz_mysql_innodb.sql

mysql -u <username> -p quartz < Init_quartz_mysql_innodb.sql

Set Env

export those variables below, or create and put it into .bashrc

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

export HADOOP_HOME=/apache/hadoop
export HADOOP_COMMON_HOME=/apache/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=/apache/hadoop/lib/native
export HADOOP_HDFS_HOME=/apache/hadoop
export HADOOP_INSTALL=/apache/hadoop
export HADOOP_MAPRED_HOME=/apache/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=/apache/spark
export LIVY_HOME=/apache/livy
export HIVE_HOME=/apache/hive
export YARN_HOME=/apache/hadoop
export SCALA_HOME=/apache/scala


  • update configuration

here are sample configurations for hadoop
Put site-specific property overrides in this file /apache/hadoop/etc/hadoop/core-site.xml


Put site-specific property overrides in this file /apache/hadoop/etc/hadoop/hdfs-site.xml

  • start/stop hadoop nodes
# format name node
/apache/hadoop/bin/hdfs namenode -format
# start namenode/datanode
# stop all nodes
  • start/stop hadoop ResourceManager
# manually clear the ResourceManager state store
/apache/hadoop/bin/yarn resourcemanager -format-state-store
# startup the ResourceManager
/apache/hadoop/sbin/ start resourcemanager
# stop the ResourceManager
/apache/hadoop/sbin/ stop resourcemanager
  • start/stop hadoop NodeManager
# startup the NodeManager
/apache/hadoop/sbin/ start nodemanager
# stop the NodeManager
/apache/hadoop/sbin/ stop nodemanager
  • start/stop hadoop HistoryServer
# startup the HistoryServer
/apache/hadoop/sbin/ start historyserver
# stop the HistoryServer
/apache/hadoop/sbin/ stop historyserver


You need to make sure that your spark cluster could access your HiveContext.

  • update configuration Copy hive/conf/hive-site.xml.template to hive/conf/hive-site.xml and update some fields.
+++ hive/conf/hive-site.xml	2018-12-16 11:17:51.000000000 +0800
@@ -368,7 +368,7 @@
-    <value/>
+    <value>thrift://</value>
     <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
@@ -527,7 +527,7 @@
-    <value>mine</value>
+    <value>secret</value>
     <description>password to use against metastore database</description>
@@ -542,7 +542,7 @@
-    <value>jdbc:derby:;databaseName=metastore_db;create=true</value>
+    <value>jdbc:postgresql://</value>
       JDBC connect string for a JDBC metastore.
       To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
@@ -1017,7 +1017,7 @@
-    <value>org.apache.derby.jdbc.EmbeddedDriver</value>
+    <value>org.postgresql.Driver</value>
     <description>Driver class name for a JDBC metastore</description>
@@ -1042,7 +1042,7 @@
-    <value>APP</value>
+    <value>king</value>
     <description>Username to use against metastore database</description>
  • start up hive metastore service
# start hive metastore service
/apache/hive/bin/hive --service metastore


  • start up spark nodes
cp /apache/hive/conf/hive-site.xml /apache/spark/conf/
/apache/spark/sbin/  spark://localhost:7077


Apache Griffin need to schedule spark jobs by server, we use livy to submit our jobs. For some issues of Livy for HiveContext, we need to download 3 files or get them from Spark lib $SPARK_HOME/lib/, and put them into HDFS.

  • update configuration
mkdir livy/logs

# update livy/conf/livy.conf =
livy.spark.master = yarn
livy.spark.deployMode = cluster
livy.repl.enableHiveContext = true
  • start up livy


You might want to create Elasticsearch index in advance, in order to set number of shards, replicas, and other settings to desired values:

curl -XPUT http://es:9200/griffin -d '
    "aliases": {},
    "mappings": {
        "accuracy": {
            "properties": {
                "name": {
                    "fields": {
                        "keyword": {
                            "ignore_above": 256,
                            "type": "keyword"
                    "type": "text"
                "tmst": {
                    "type": "date"
    "settings": {
        "index": {
            "number_of_replicas": "2",
            "number_of_shards": "5"

You should also modify some configurations of Apache Griffin for your environment.

  • service/src/main/resources/

    # Apache Griffin server port (default 8080)
    server.port = 8080
    # jpa
    spring.datasource.url = jdbc:postgresql://<your IP>:5432/quartz?autoReconnect=true&useSSL=false
    spring.datasource.username = <user name>
    spring.datasource.password = <password>
    spring.datasource.driverClassName = org.postgresql.Driver = true
    # hive metastore
    hive.metastore.uris = thrift://<your IP>:9083
    hive.metastore.dbname = <hive database name>    # default is "default"
    # external properties directory location, ignore it if not required
    external.config.location =
    # login strategy, default is "default"
    login.strategy = <default or ldap>
    # ldap properties, ignore them if ldap is not enabled
    ldap.url = ldap://hostname:port =
    ldap.searchBase = DC=org,DC=example
    ldap.searchPattern = (sAMAccountName={0})
    # hdfs, ignore it if you do not need predicate job
    fs.defaultFS = hdfs://<hdfs-default-name>
    # elasticsearch = <your IP>
    elasticsearch.port = <your elasticsearch rest port>
    # authentication properties, uncomment if basic authentication is enabled
    # elasticsearch.user = user
    # elasticsearch.password = password
    # livy
    # Port Livy: 8998 Livy2:8999
    # yarn url
  • service/src/main/resources/sparkProperties.json

      "file": "hdfs:///<griffin measure path>/griffin-measure.jar",
      "className": "org.apache.griffin.measure.Application",
      "name": "griffin",
      "queue": "default",
      "numExecutors": 3,
      "executorCores": 1,
      "driverMemory": "1g",
      "executorMemory": "1g",
      "conf": {
    	"spark.yarn.dist.files": "hdfs:///<path to>/hive-site.xml"
      "files": [
      "jars": [
    • <griffin measure path> is the location where you should put the jar file of measure module.
  • service/src/main/resources/env/env_batch.json

    Adjust sinks according to your requirement. At least, you will need to adjust HDFS output directory (hdfs:///griffin/persist by default), and Elasticsearch URL (http://es:9200/griffin/accuracy by default). Similar changes are required in env_streaming.json.


Griffin Service is regular Spring Boot application, so it supports all customizations from Spring Boot. To enable output compression, the following should be added to



It is possible to enable SSL encryption for api and web endpoints. To do that, you will need to prepare keystore in Spring-compatible format (for example, PKCS12), and add the following values to



The following properties are available for LDAP:

  • ldap.url: URL of LDAP server.
  • Arbitrary suffix added to user's login before search, can be empty string. Used when user's DN contains some common suffix, and there is no bindDN specified. In this case, string after concatenation is used as DN for sending initial bind request.
  • ldap.searchBase: Subtree DN to search.
  • ldap.searchPattern: Filter expression, substring {0} is replaced with user's login after is concatenated. This expression is used to find user object in LDAP. Access is denied if filter expression did not match any users.
  • ldap.sslSkipVerify: Allows to disable certificate validation for secure LDAP servers.
  • ldap.bindDN: Optional DN of service account used for user lookup. Useful if user's DN is different than attribute used as user's login, or if users' DNs are ambiguous.
  • ldap.bindPassword: Optional password of bind service account.

Build and Run

Build the whole project and deploy. (NPM should be installed)

mvn clean install

Put jar file of measure module into <griffin measure path> in HDFS

cp measure/target/measure-<version>-incubating-SNAPSHOT.jar measure/target/griffin-measure.jar
hdfs dfs -put measure/target/griffin-measure.jar <griffin measure path>/

After all environment services startup, we can start our server.

java -jar service/target/service.jar

After a few seconds, we can visit our default UI of Apache Griffin (by default the port of spring boot is 8080).

http://<your IP>:8080

You can use UI following the steps here.

Note: The UI does not support all the backend features, to experience the advanced features you can use services directly.