[PIO-101] Document usage of Plug-in of event server and engine server

Closes #440
apache · Oct 6, 2017 · d8ee0c8 · d8ee0c8
1 parent f92d2ac
commit d8ee0c8
Show file tree

Hide file tree

Showing 3 changed files with 247 additions and 0 deletions.
diff --git a/docs/manual/data/nav/main.yml b/docs/manual/data/nav/main.yml
@@ -70,6 +70,9 @@ root:
       -
         body: 'Deploying Multiple Engine Variants'
         url: '/deploy/enginevariants/'
+      -
+        body: 'Engine Server Plugin'
+        url: '/deploy/plugin/'
   -
     body: 'Customizing an Engine'
     url: '#'
@@ -111,6 +114,9 @@ root:
       -
         body: 'Using Analytics Tools'
         url: '/datacollection/analytics/'
+      -
+        body: 'Event Server Plugin'
+        url: '/datacollection/plugin/'
   -
     body: 'Choosing an Algorithm(s)'
     url: '#'

diff --git a/docs/manual/source/datacollection/plugin.html.md b/docs/manual/source/datacollection/plugin.html.md
@@ -0,0 +1,111 @@
+---
+title: Event Server Plugin
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+You can write event server plugins to handle input data. For example, it's able to block invalid data, log, get statics or forward to other processing systems. There are two types of event server plugin.
+
+- `Input Blocker`: When these plugins are present, events coming into event server will be passed through all loaded and active plugins before reaching the actual event store. The order of processing is not defined, so events can go through these plugins in arbitrary order. One use case is for validating input data and throw exceptions to prevent bad data from going in. These plugins cannot transform the event.
+- `Input Sniffer`: When these are present, events will be broadcasted to these plugins in parallel. They do not block the event from reaching event store. They are useful for logging, statistics, and forwarding to other processing systems.
+
+## Create an event server plugin
+
+At first, create a sbt project with following `build.sbt`:
+
+```scala
+name := "pio-plugin-example"
+version := "1.0"
+scalaVersion := "2.11.11"
+libraryDependencies += "org.apache.predictionio" %% "apache-predictionio-core" % "0.12.0-incubating"
+```
+
+Event server plug-ins must extend `EventServerPlugin`. Here is an example of event server plug-in:
+
+```scala
+package com.example
+
+import org.apache.predictionio.data.api._
+
+class MyEventServerPlugin extends EventServerPlugin {
+  val pluginName = "my-eventserver-plugin"
+  val pluginDescription = "an example of event server plug-in"
+
+  // inputBlocker or inputSniffer
+  val pluginType = EventServerPlugin.inputBlocker	
+
+  // Plug-in can handle input data in this method.
+  // If plug-in found invalid data, it's possible to block them 
+  // by throwing an exception in this method.
+  override def process(
+      eventInfo: EventInfo, 
+      context: EventServerPluginContext): Unit = {
+    println(eventInfo)
+  }
+
+  // Plug-in can handle requests to /plugins/<pluginType>/<pluginName>/* 
+  // on the event server in this method.
+  override def handleREST(
+      appId: Int, 
+      channelId: Option[Int], 
+      arguments: Seq[String]): String = {
+    """{"pluginName": "my-eventserver-plugin"}"""
+  }
+}
+```
+
+Plug-ins are loaded by `ServiceLoader`, so you must create `META-INF/services/org.apache.predictionio.data.api.EventServerPlugin` with a following content:	
+
+```
+com.example.MyEventServerPlugin
+```
+
+Finally, run `sbt package` to package plugin as a jar file. In this case, the plugin jar file is generated at `target/scala-2.11/pio-plugin-example_2.11-1.0.jar`, so copy this file to `PIO_HOME/plugins`.
+
+When you start (or restart) the event server, this plugin should be enabled.
+
+## Plugin APIs of event server
+
+The event server has some plugins related APIs:
+
+- `/plugins.json`: Show all enabled plugins.
+- `/plugins/inputblocker/<pluginName>/*`: Handled by a corresponding input blocker plugin.
+- `/plugins/inputsniffer/<pluginName>/*`: Handled by a corresponding input sniffer plugin.
+
+For example, if you send following request to the event server:
+
+```
+curl -XGET http://localhost:7070/plugins.json?accessKey=$ACCESS_KEY
+```
+
+The event server should respond following JSON response:
+
+```json
+{
+  "plugins": {
+    "inputblockers": {
+      "my-eventserver-plugin": {
+        "name": "my-eventserver-plugin",
+        "description": "an example of event server plug-in",
+        "class": "com.example.MyEventServerPlugin"
+      }
+    },
+    "inputsniffers": {}
+  }
+}
+```
diff --git a/docs/manual/source/deploy/plugin.html.md b/docs/manual/source/deploy/plugin.html.md
@@ -0,0 +1,130 @@
+---
+title: Engine Server Plugin
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+You can write engine server plugins to handle output data. For example, it's able to transform or log prediction result. There are two types of engine server plugin.
+
+- `Output Blocker`: Before predictions go out, they will be processed through all loaded and active plugins. The order of processing is not defined. They are useful for transforming prediction results (e.g. if you do not have access to engine source code).
+- `Output Sniffer`: These should have similar benefits with event server sniffers.
+
+## Create an engine server plugin
+
+At first, create a sbt project with following `build.sbt`:
+
+```scala
+name := "pio-plugin-example"
+version := "1.0"
+scalaVersion := "2.11.11"
+libraryDependencies += "org.apache.predictionio" %% "apache-predictionio-core" % "0.12.0-incubating"
+```
+
+Engine server plug-ins must extend `EngineServerPlugin`. Here is an example of engine server plug-in:
+
+```scala
+package com.example
+
+import org.apache.predictionio.data.storage.EngineInstance
+import org.apache.predictionio.workflow._
+import org.json4s.JValue
+
+class MyEngineServerPlugin extends EngineServerPlugin {
+  val pluginName = "my-engineserver-plugin"
+  val pluginDescription = "an example of engine server plug-in"
+
+  // inputBlocker or inputSniffer
+  val pluginType = EngineServerPlugin.outputBlocker	
+
+  // Plug-in can handle output data in this method.
+  override def process(
+      engineInstance: EngineInstance,
+      query: JValue,
+      prediction: JValue,
+      context: EngineServerPluginContext): JValue = {
+    println(prediction)
+    prediction
+  }
+
+  // Plug-in can handle requests to /plugins/<pluginType>/<pluginName>/* 
+  // on the engine server in this method.
+  override def handleREST(arguments: Seq[String]): String = {
+     """{"pluginName": "my-engineserver-plugin"}"""
+  }
+}
+```
+
+Plug-ins are loaded by `ServiceLoader`, so you must create `META-INF/services/org.apache.predictionio.workflow.EngineServerPlugin` with a following content:	
+
+```
+com.example.MyEngineServerPlugin
+```
+
+Then, run `sbt package` to package plugin as a jar file. In this case, the plugin jar file is generated at `target/scala-2.11/pio-plugin-example_2.11-1.0.jar`, so copy this file to `PIO_HOME/plugins`.
+
+To enable plugins, you have to modify `engine.json` in the root directory of your engine as follows. Defined plugins parameters can be accessed via `EngineServerPluginContext` in plugins.
+
+```json
+{
+  "id": "default",
+  "description": "Default settings",
+  "engineFactory": "org.example.recommendation.RecommendationEngine",
+  "plugins": {
+    "my-engineserver-plugin": {
+      "enabled": true
+    }
+  },
+  ...
+}
+```
+
+When you start (or restart) the engine server, this plugin should be enabled.
+
+## Plugin APIs of engine server
+
+The engine server has some plugins related APIs:
+
+- `/plugins.json`: Show all enabled plugins.
+- `/plugins/outputblocker/<pluginName>/*`: Handled by a corresponding output blocker plugin.
+- `/plugins/outputsniffer/<pluginName>/*`: Handled by a corresponding output sniffer plugin.
+
+For example, if you send following request to the engine server:
+
+```
+curl -XGET http://localhost:7070/plugins.json?accessKey=$ACCESS_KEY
+```
+
+The engine server should respond following JSON response:
+
+```json
+{
+  "plugins": {
+    "outputblockers": {
+      "my-engineserver-plugin": {
+        "name": "my-engineserver-plugin",
+        "description": "an example of engine server plug-in",
+        "class": "com.example.MyEngineServerPlugin",
+        "params": {
+          "enabled": true
+        }
+      }
+    },
+    "outputsniffers": {}
+  }
+}
+```