forked from apache/metron
/
README.vm
236 lines (182 loc) · 11.4 KB
/
README.vm
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
#[[#]]# Metron REST
This module provides a RESTful API for interacting with Metron.
#[[##]]# Prerequisites
* A running Metron cluster
* Java 8 installed
* Storm CLI and Metron topology scripts (start_parser_topology.sh, start_enrichment_topology.sh, start_elasticsearch_topology.sh) installed
* A relational database
#[[##]]# Installation
#[[###]]# From Source
1. Package the application with Maven:
```
mvn clean package
```
1. Untar the archive in the $METRON_HOME directory. The directory structure will look like:
```
config
rest_application.yml
bin
metron-rest
lib
metron-rest-$METRON_VERSION.jar
```
1. Copy the `$METRON_HOME/bin/metron-rest` script to `/etc/init.d/metron-rest`
#[[###]]# From Package Manager
1. Deploy the RPM at `/metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-rest-$METRON_VERSION-*.noarch.rpm`
1. Install the RPM with:
```
rpm -ih metron-rest-$METRON_VERSION-*.noarch.rpm
```
#[[##]]# Configuration
The REST application depends on several configuration parameters:
| Environment Variable | Description | Required | Default |
| ------------------------------------- | ----------------------------------------------------------------- | -------- | ---------------------- |
| METRON_USER | Run the application as this user | Optional | metron |
| METRON_LOG_DIR | Directory where the log file is written | Optional | /var/log/metron/ |
| METRON_PID_DIR | Directory where the pid file is written | Optional | /var/run/metron/ |
| METRON_REST_PORT | REST application port | Optional | 8082 |
| METRON_JVMFLAGS | JVM flags added to the start command | Optional | |
| METRON_SPRING_PROFILES_ACTIVE | Active Spring profiles (see [below](#spring-profiles)) | Optional | |
| METRON_JDBC_DRIVER | JDBC driver class | Required | |
| METRON_JDBC_URL | JDBC url | Required | |
| METRON_JDBC_USERNAME | JDBC username | Required | |
| METRON_JDBC_PLATFORM | JDBC platform (one of h2, mysql, postgres, oracle | Required | |
| METRON_JDBC_CLIENT_PATH | Path to JDBC client jar | Optional | H2 is bundled |
| METRON_TEMP_GROK_PATH | Temporary directory used to test grok statements | Optional | ./patterns/temp |
| METRON_DEFAULT_GROK_PATH | Defaults HDFS directory used to store grok statements | Optional | /apps/metron/patterns |
| METRON_SPRING_OPTIONS | Additional Spring input parameters | Optional | |
| ZOOKEEPER | Zookeeper quorum (ex. node1:2181,node2:2181) | Required | |
| BROKERLIST | Kafka Broker list (ex. node1:6667,node2:6667) | Required | |
| HDFS_URL | HDFS url or `fs.defaultFS` Hadoop setting (ex. hdfs://node1:8020) | Required | |
| SECURITY_ENABLED | Enables Kerberos support | Optional | false |
| METRON_PRINCIPAL_NAME | Kerberos principal for the metron user | Optional | |
| METRON_SERVICE_KEYTAB | Path to the Kerberos keytab for the metron user | Optional | |
These are set in the `/etc/sysconfig/metron` file.
#[[##]]# Database setup
The REST application persists data in a relational database and requires a dedicated database user and database (see https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html for more detail).
#[[###]]# Development
The REST application comes with embedded database support for development purposes (https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html#boot-features-embedded-database-support).
For example, edit these variables in `/etc/sysconfig/metron` before starting the application to configure H2:
```
METRON_JDBC_DRIVER="org.h2.Driver"
METRON_JDBC_URL="jdbc:h2:file:~/metrondb"
METRON_JDBC_USERNAME="root"
METRON_JDBC_PASSWORD='root"
METRON_JDBC_PLATFORM="h2"
```
#[[###]]# Production
The REST application should be configured with a production-grade database outside of development.
For example, the following configures the application for MySQL:
1. Install MySQL if not already available (this example uses version 5.7, installation instructions can be found [here](https://dev.mysql.com/doc/refman/5.7/en/linux-installation-yum-repo.html))
1. Create a metron user and REST database and permission the user for that database:
```
CREATE USER 'metron'@'node1' IDENTIFIED BY 'Myp@ssw0rd';
CREATE DATABASE IF NOT EXISTS metronrest;
GRANT ALL PRIVILEGES ON metronrest.* TO 'metron'@'node1';
```
1. Install the MySQL JDBC client onto the REST application host and configurate the METRON_JDBC_CLIENT_PATH variable:
```
cd $METRON_HOME/lib
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.41.tar.gz
tar xf mysql-connector-java-5.1.41.tar.gz
```
1. Edit these variables in `/etc/sysconfig/metron` to configure the REST application for MySQL:
```
METRON_JDBC_DRIVER="com.mysql.jdbc.Driver"
METRON_JDBC_URL="jdbc:mysql://mysql_host:3306/metronrest"
METRON_JDBC_USERNAME="metron"
METRON_JDBC_PLATFORM="mysql"
METRON_JDBC_CLIENT_PATH=$METRON_HOME/lib/mysql-connector-java-5.1.41/mysql-connector-java-5.1.41-bin.jar
```
#[[##]]# Usage
After configuration is complete, the REST application can be managed as a service:
```
service metron-rest start
```
If a production database is configured, the JDBC password should be passed in as the first argument on startup:
```
service metron-rest start Myp@ssw0rd
```
The REST application can be accessed with the Swagger UI at http://host:port/swagger-ui.html#/. The default port is 8082.
#[[##]]# Security
#[[###]]# Authentication
The metron-rest module uses [Spring Security](http://projects.spring.io/spring-security/) for authentication and stores user credentials in the relational database configured above. The required tables are created automatically the first time the application is started so that should be done first. For example (continuing the MySQL example above), users can be added by connecting to MySQL and running:
```
use metronrest;
insert into users (username, password, enabled) values ('your_username','your_password',1);
insert into authorities (username, authority) values ('your_username', 'ROLE_USER');
```
#[[###]]# Kerberos
Metron REST can be configured for a cluster with Kerberos enabled. A client JAAS file is required for Kafka and Zookeeper and a Kerberos keytab for the metron user principal is required for all other services. Configure these settings in the `/etc/sysconfig/metron` file:
```
SECURITY_ENABLED=true
METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf"
METRON_PRINCIPAL_NAME="metron@EXAMPLE.COM"
METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
```
#[[##]]# Spring Profiles
The REST application comes with a few [Spring Profiles](http://docs.spring.io/autorepo/docs/spring-boot/current/reference/html/boot-features-profiles.html) to aid in testing and development.
| Profile | Description |
| ------------------------ | --------------------------------------------- |
| test | sets variables to in-memory services, only used for integration testing |
| dev | adds a test user to the database with credentials `user/password` |
| vagrant | sets configuration variables to match the Metron vagrant environment |
| docker | sets configuration variables to match the Metron dcoker environment |
Setting active profiles is done with the METRON_SPRING_PROFILES_ACTIVE variable. For example, set this variable in `/etc/sysconfig/metron` to configure the REST application for the Vagrant environment and add a test user:
```
METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
```
#[[##]]# API
Request and Response objects are JSON formatted. The JSON schemas are available in the Swagger UI.
| |
| ---------- |
#foreach( $restControllerInfo in $endpoints )
| [ `$restControllerInfo.getMethod().toString() $restControllerInfo.getPath()`](#$restControllerInfo.getMethod().toString().toLowerCase()-$restControllerInfo.getPath().toLowerCase().replaceAll("[/\{\}]", ""))|
#end
#foreach( $restControllerInfo in $endpoints )
#[[###]]# `$restControllerInfo.getMethod().toString() $restControllerInfo.getPath()`
* Description: $restControllerInfo.getDescription()
#if($restControllerInfo.getParameterDescriptions().size() > 0)
* Input:
#end
#foreach( $parameterDescription in $restControllerInfo.getParameterDescriptions().entrySet())
* $parameterDescription.getKey() - $parameterDescription.getValue()
#end
* Returns:
#foreach( $response in $restControllerInfo.getResponses())
* $response.getCode() - $response.getMessage()
#end
#end
#[[##]]# Testing
Profiles are includes for both the metron-docker and Full Dev environments.
#[[###]]# metron-docker
Start the [metron-docker](../../metron-docker) environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:
```
mvn clean package
mvn spring-boot:run -Drun.profiles=docker,dev
```
The metron-rest application will be available at http://localhost:8080/swagger-ui.html#/.
#[[###]]# Quick Dev
Start the [Quick Dev](../../metron-deployment/vagrant/quick-dev-platform) environment. Build the metron-rest module and start it with the Spring Boot Maven plugin:
```
mvn clean package
mvn spring-boot:run -Drun.profiles=vagrant,dev
```
The metron-rest application will be available at http://localhost:8080/swagger-ui.html#/.
To run the application locally on the Quick Dev host (node1), follow the [Installation](#installation) instructions above. Then set the METRON_SPRING_PROFILES_ACTIVE variable in `/etc/sysconfig/metron`:
```
METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
```
and start the application:
```
service metron-rest start
```
In a cluster with Kerberos enabled, update the security settings in `/etc/sysconfig/metron`. Security is disabled by default in the `vagrant` Spring profile so that setting must be overriden with the METRON_SPRING_OPTIONS variable:
```
METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf"
METRON_SPRING_OPTIONS="--kerberos.enabled=true"
```
The metron-rest application will be available at http://node1:8082/swagger-ui.html#/.
#[[##]]# License
This project depends on the Java Transaction API. See https://java.net/projects/jta-spec/ for more details.