METRON-435 Create Stellar REPL

metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/README.md

@@ -0,0 +1,126 @@
+# Stellar Shell
+
+A REPL (Read Eval Print Loop) for the Stellar language that helps in debugging, troubleshooting and learning Stellar.  The Stellar DSL (domain specific language) is used to act upon streaming data within Apache Storm.  It is difficult to troubleshoot Stellar when it can only be executed within a Storm topology.  This REPL is intended to help mitigate that problem by allowing a user to replicate data encountered in production, isolate initialization errors, or understand function resolution problems.
+
+### Getting Started
+
+```
+$ /usr/metron/<version>/bin/stellar
+
+Stellar, Go!
+{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH}
+
+>>> %functions
+BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, ...
+
+>>> ?PROTOCOL_TO_NAME
+PROTOCOL_TO_NAME
+ desc: Convert the IANA protocol number to the protocol name       
+ args: IANA Number                                                 
+  ret: The protocol name associated with the IANA number.          
+
+>>> 6
+[=] 6
+[?] save as: ip.protocol
+
+>>> %vars
+ip.protocol = 6
+
+>>> PROTOCOL_TO_NAME(ip.protocol)
+[=] TCP
+```
+
+### Command Line Options
+
+```
+$ /usr/metron/<version>/bin/stellar -h
+
+usage: stellar
+ -h,--help              Print help
+ -z,--zookeeper <arg>   Zookeeper URL
+```
+
+#### -z, --zookeeper
+
+*Optional*
+
+Attempts to connect to Zookeeper and read the Metron global configuration.  Stellar functions may require the global configuration to work properly.  If found, the global configuration values are printed to the console.
+
+```
+$ /usr/metron/<version>/bin/stellar -z node1:2181
+Stellar, Go!
+{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH}
+>>> 
+```
+
+### Variable Assignment
+
+Stellar has no concept of variable assignment.  For testing and debugging purposes, it is important to be able to create variables that simulate data contained within incoming messages.  The REPL has created a means for a user to perform variable assignment outside of the core Stellar language.  
+
+After execution of a Stellar expression, the REPL will prompt for the name of a variable to save the expression result to.  If no name is provided, the result is not saved.
+
+```
+>>> 2+2
+[=] 4.0
+[?] save as: foo
+>>> %vars
+foo = 4.0
+>>> 
+```
+
+### Magic Commands
+
+The REPL has a set of magic commands that provide the REPL user with information about the Stellar execution environment.  The following magic commands are supported.
+
+#### `%functions`
+
+This command lists all functions resolvable in the Stellar environment.  Stellar searches the classpath for Stellar functions.  This can make it difficult in some cases to understand which functions are resolvable.  
+
+```
+>>> %functions
+BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, 
+DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, GET, GET_FIRST, 
+GET_LAST, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, 
+JOIN, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP_EXISTS, MAP_GET, MONTH, PROTOCOL_TO_NAME, 
+REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, 
+STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, 
+STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, 
+STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, TO_DOUBLE, TO_EPOCH_TIMESTAMP, 
+TO_INTEGER, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, 
+URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
+>>> 
+```
+
+#### `%vars` 
+
+Lists all variables in the Stellar environment.
+
+```
+Stellar, Go!
+{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH}
+>>> %vars
+>>> 2+2
+[=] 4.0
+[?] save as: foo
+>>> %vars
+foo = 4.0
+>>> 
+```
+
+#### `?<function>`
+
+Returns formatted documentation of the Stellar function.  Provides the description of the function along with the expected arguments.
+
+```
+>>> ?BLOOM_ADD
+BLOOM_ADD
+ desc: Adds an element to the bloom filter passed in               
+ args: bloom - The bloom filter, value* - The values to add        
+  ret: Bloom Filter                                                
+>>> ?IS_EMAIL
+IS_EMAIL
+ desc: Tests if a string is a valid email address                  
+ args: address - The String to test                                
+  ret: True if the string is a valid email address and false otherwise.
+>>> 
+```

metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarExecutor.java

@@ -0,0 +1,152 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.common.stellar.shell;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.commons.lang.StringUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.ConfigurationsUtils;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.FunctionResolver;
+import org.apache.metron.common.dsl.MapVariableResolver;
+import org.apache.metron.common.dsl.StellarFunctions;
+import org.apache.metron.common.dsl.VariableResolver;
+import org.apache.metron.common.stellar.StellarProcessor;
+import org.apache.metron.common.utils.JSONUtils;
+
+import java.io.ByteArrayInputStream;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+import static org.apache.metron.common.configuration.ConfigurationsUtils.readGlobalConfigBytesFromZookeeper;
+
+/**
+ * Executes Stellar expressions and maintains state across multiple invocations.
+ */
+public class StellarExecutor {
+
+  /**
+   * The variables known by Stellar.
+   */
+  private Map<String, Object> variables;
+
+  /**
+   * The function resolver.
+   */
+  private FunctionResolver functionResolver ;
+
+  /**
+   * A Zookeeper client. Only defined if given a valid Zookeeper URL.
+   */
+  private Optional<CuratorFramework> client;
+
+  /**
+   * The Stellar execution context.
+   */
+  private Context context;
+
+  public StellarExecutor() throws Exception {
+    this(null);
+  }
+
+  public StellarExecutor(String zookeeperUrl) throws Exception {
+    this.variables = new HashMap<>();
+    this.functionResolver = new StellarFunctions().FUNCTION_RESOLVER();
+    this.client = createClient(zookeeperUrl);
+    this.context = createContext();
+  }
+
+  /**
+   * Creates a Zookeeper client.
+   * @param zookeeperUrl The Zookeeper URL.
+   */
+  private Optional<CuratorFramework> createClient(String zookeeperUrl) {
+
+    // can only create client, if have valid zookeeper URL
+    if(StringUtils.isNotBlank(zookeeperUrl)) {
+      CuratorFramework client = ConfigurationsUtils.getClient(zookeeperUrl);
+      client.start();
+      return Optional.of(client);
+
+    } else {
+      return Optional.empty();
+    }
+  }
+
+  /**
+   * Creates a Context initialized with configuration stored in Zookeeper.
+   */
+  private Context createContext() throws Exception {
+    Context context = Context.EMPTY_CONTEXT();
+
+    // load global configuration from zookeeper
+    if (client.isPresent()) {
+
+      // fetch the global configuration
+      Map<String, Object> global = JSONUtils.INSTANCE.load(
+              new ByteArrayInputStream(readGlobalConfigBytesFromZookeeper(client.get())),
+              new TypeReference<Map<String, Object>>() {
+              });
+
+      context = new Context.Builder()
+              .with(Context.Capabilities.GLOBAL_CONFIG, () -> global)
+              .with(Context.Capabilities.ZOOKEEPER_CLIENT, () -> client.get())
+              .build();
+    }
+
+    return context;
+  }
+
+  /**
+   * Executes the Stellar expression and returns the result.
+   * @param expression The Stellar expression to execute.
+   * @return The result of the expression.
+   */
+  public Object execute(String expression) {
+    VariableResolver variableResolver = new MapVariableResolver(variables, Collections.emptyMap());
+    StellarProcessor processor = new StellarProcessor();
+    return processor.parse(expression, variableResolver, functionResolver, context);
+  }
+
+  /**
+   * Assigns a value to a variable.
+   * @param variable The name of the variable.
+   * @param value The value of the variable
+   */
+  public void assign(String variable, Object value) {
+    this.variables.put(variable, value);
+  }
+
+  public Map<String, Object> getVariables() {
+    return this.variables;
+  }
+
+  public FunctionResolver getFunctionResolver() {
+    return functionResolver;
+  }
+
+  public Context getContext() {
+    return context;
+  }
+}
+