METRON-690: Create a DSL-based timestamp lookup for profiler to enable sparse windows #450

cestella · 2017-02-10T02:28:31Z

Creating a small DSL to allow specifying profiles from windows of time that may:

repeat non-contiguously (e.g. the same hour every day for a week)
have inclusions and exclusions (e.g. the same hour every day for a week excluding holidays and weekends)

This also provides a PROFILE_WINDOW Stellar function which accepts this DSL as a string and can return the set of profiler periods based on the times selected and suitable for passing to PROFILE_GET. To be clear, you can compose the two functions.

i.e. PROFILE_GET('profile', 'entity', PROFILE_WINDOW('1 hour window every 24 hours starting from 14 days ago including the current day of the week excluding weekends, holidays:us')) would retrieve all the profile measurements written for the profile "profile" and entity "entity" for the last hour on the same weekday excluding weekends and US holidays across the last 14 days

For a complete description with examples, see here.

Acceptance testing plan will be submitted as a follow-on comment.

…dows

cestella · 2017-02-10T02:32:53Z

For reviewers, please note that as this is an antlr DSL, you may skip the generated code (the java files in the org.apache.metron.profiler.client.window.generated package , which is the bulk of this PR.

cestella · 2017-02-10T14:32:17Z

I wanted to point out one thing; while this is a DSL called from a DSL (stellar), the way the parser is written, we cache the output of the parser and can reuse them for evaluation, thus saving us much of the impact of the parsing of this DSL.

…line...and it's the right thing to do.

justinleet

This looks like a great addition, thanks a lot. Couple minor comments to look over.

I'll want to run this up, so I look forward to the acceptance testing plan

justinleet · 2017-02-10T15:25:06Z

...er-client/src/test/java/org/apache/metron/profiler/client/stellar/IntervalPredicateTest.java

+      add(Range.between(20L, 30L));
+      add(Range.between(40L, 50L));
+    }};
+    IntervalPredicate predicate = new IntervalPredicate.Identity(intervals);


Can you make this (and the similar instances) IntervalPredicate so my compiler likes me again?

Sure, but your compiler will never like you.

justinleet · 2017-02-10T15:29:54Z

...-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java

+    LinkedList<Function<Long, Predicate<Long>>> predicates = new LinkedList<>();
+    while (true) {
+      Token<?> token = stack.pop();
+      if (token == LIST_MARKER) {


Should this be .equals()?

Nope, it's reference equals. LIST_MARKER is a specific instance of Token.

Sounds good, thanks.

I should say, it's a specific static instance used for the parser to denote the end of the list of values in the stack.

justinleet · 2017-02-10T15:30:04Z

...-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java

+
+    while (true) {
+      Token<?> token = stack.pop();
+      if (token == DAY_SPECIFIER_MARKER) {


Same deal, .equals()

james-sirota · 2017-02-13T02:52:45Z

I think the API looks great. Can we provide a grammar in the README for constructing the PROFILE_WINDOW function? I think the API is so flexible that it may be hard to wrap your head around how to construct the profile retrieve string

cestella · 2017-02-13T02:57:18Z

The Readme additions were intended to break the expressions down into the possible phrases. Do you think those sections need to be structured differently?

james-sirota · 2017-02-13T03:15:33Z

I think taking the string as an argument is really powerful, but it's also really flexible. "1 hour window every 24 hours starting from 14 days ago including the current day of the week excluding weekends, holidays:us" is a really long string. I was thinking maybe define a grammar to look something like:

cmd = <window_duration> <window_period> <window_limit> <window_inclusions> <window_exlcusions> <holidays_spec>
<window_duration> = number <unit_spec>
<unit_spec> = MINUTS|HOURS|DAYS

...or something like that...a grammar for constructing this string

cestella · 2017-02-17T15:54:39Z

@james-sirota Yep, agreed. I'll modify the docs.

cestella · 2017-02-17T18:02:42Z

Testing Instructions beyond the normal smoke test (i.e. letting data
flow through to the indices and checking them).

Free Up Space on the virtual machine

First, let's free up some headroom on the virtual machine. If you are running this on a
multinode cluster, you would not have to do this.

Kill monit via service monit stop
Kill tcpreplay via for i in $(ps -ef | grep tcpreplay | awk '{print $2}');do kill -9 $i;done
Kill existing parser topologies via
- storm kill snort
- storm kill bro
Kill yaf via for i in $(ps -ef | grep yaf | awk '{print $2}');do kill -9 $i;done
Kill bro via for i in $(ps -ef | grep bro | awk '{print $2}');do kill -9 $i;done

Preliminaries

Set an environment variable to indicate METRON_HOME:
export METRON_HOME=/usr/metron/0.3.1
Create the profiler hbase table
echo "create 'profiler', 'P'" | hbase shell
Open ~/rand_gen.py and paste the following:

#!/usr/bin/python
import random
import sys
import time
def main():
  mu = float(sys.argv[1])
  sigma = float(sys.argv[2])
  freq_s = int(sys.argv[3])
  while True:
    out = '{ "value" : ' + str(random.gauss(mu, sigma)) + ' }'
    print out
    sys.stdout.flush()
    time.sleep(freq_s)

if __name__ == '__main__':
  main()

This will generate random JSON maps with a numeric field called value

Set the profiler to use 1 minute tick durations:
- Edit $METRON_HOME/config/profiler.properties to adjust the capture duration by changing profiler.period.duration=15 to profiler.period.duration=1
- Edit $METRON_HOME/config/zookeeper/global.json and add the following properties:

"profiler.client.period.duration" : "1",
"profiler.client.period.duration.units" : "MINUTES"

Deploy the custom parser

Edit the value parser config at $METRON_HOME/config/zookeeper/parsers/value.json:

{
  "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
  "sensorTopic":"value",
  "fieldTransformations" : [
    {
    "transformation" : "STELLAR"
   ,"output" : [ "num_profiles_parser", "mean_parser" ]
   ,"config" : {
      "num_profiles_parser" : "LENGTH(PROFILE_GET('stat', 'global', PROFILE_WINDOW('5 minute window every 10 minutes starting from 2 minutes ago until 32 minutes ago excluding holidays:us')))",
      "mean_parser" : "STATS_MEAN(STATS_MERGE(PROFILE_GET('stat', 'global', PROFILE_WINDOW('5 minute window every 10 minutes starting from 2 minutes ago until 32 minutes ago excluding holidays:us'))))"
               }
    }
                           ]
}

Edit the value enrichment config at $METRON_HOME/config/zookeeper/enrichments/value.json:

{
  "enrichment" : {
   "fieldMap": {
      "stellar" : {
        "config" : {
        "num_profiles_enrichment" : "LENGTH(PROFILE_GET('stat', 'global', PROFILE_WINDOW('5 minute window every 10 minutes starting from 2 minutes ago until 32 minutes ago excluding holidays:us')))",
        "mean_enrichment" : "STATS_MEAN(STATS_MERGE(PROFILE_GET('stat', 'global', PROFILE_WINDOW('5 minute window every 10 minutes starting from 2 minutes ago until 32 minutes ago excluding holidays:us'))))"
                  }
      }
    }
  }
}

Create the value kafka topic:
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic value --partitions 1 --replication-factor 1
Push the configs via $METRON_HOME/bin/zk_load_configs.sh -m PUSH -i $METRON_HOME/config/zookeeper -z node1:2181
Start via $METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s value

Start the profiler

Edit $METRON_HOME/config/zookeeper/profiler.json and paste in the following:

{
  "profiles": [
    {
      "profile": "stat",
      "foreach": "'global'",
      "onlyif": "true",
      "init" : {
               },
      "update": {
        "s": "STATS_ADD(s, value)"
                },
      "result": "s"
    }
  ]
}

$METRON_HOME/bin/start_profiler_topology.sh

Test Case

Set up a profile to accept some synthetic data with a numeric value field and persist a stats summary of the data
Send some synthetic data directly to the profiler:
python ~/rand_gen.py 0 1 1 | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic value
Wait for at least 32 minutes and execute the following via the Stellar REPL:

# Grab the profiles from 1 minute ago to 8 minutes ago
LENGTH(PROFILE_GET('stat', 'global', PROFILE_WINDOW('from 1 minute ago to 8 minutes ago')))
# Looks like 7 were returned, great.  Now try something more complex
# Grab the profiles in 5 minute windows every 10 minutes from 2 minutes ago to 32 minutes ago:
#  32 minutes ago til 27 minutes ago should be 5 profiles
#  22 minutes ago til 17 minutes ago should be 5 profiles
#  12 minutes ago til 7 minutes ago should be 5 profiles
# for a total of 15 profiles
LENGTH(PROFILE_GET('stat', 'global', PROFILE_WINDOW('5 minute window every 10 minutes starting from 2 minutes ago until 32 minutes ago excluding holidays:us')))
STATS_MEAN(STATS_MERGE(PROFILE_GET('stat', 'global', PROFILE_WINDOW('5 minute window every 10 minutes starting from 2 minutes ago until 32 minutes ago excluding holidays:us'))))

For me, the following was the result:

Stellar, Go!
Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH, profiler.client.period.duration=1, profiler.client.period.duration.units=MINUTES}
[Stellar]>>> # Grab the profiles from 1 minute ago to 8 minutes ago
[Stellar]>>> LENGTH(PROFILE_GET('stat', 'global', PROFILE_WINDOW('from 1 minute ago to 8 minutes ago')))
Functions loaded, you may refer to functions now...
7
[Stellar]>>> # Looks like 7 were returned, great.
[Stellar]>>> # Grab the profiles from 2 minutes ago to 32 minutes ago
[Stellar]>>> LENGTH(PROFILE_GET('stat', 'global', PROFILE_WINDOW('from 2 minutes ago to 32 minutes ago')))
30
[Stellar]>>> # Looks like 30 were returned, great.
[Stellar]>>> # Now try something more complex
[Stellar]>>> # Grab the profiles in 5 minute windows every 10 minutes from 2 minutes ago to 32 minutes ago:
[Stellar]>>> #  32 minutes ago til 27 minutes ago should be 5 profiles
[Stellar]>>> #  22 minutes ago til 17 minutes ago should be 5 profiles
[Stellar]>>> #  12 minutes ago til 7 minutes ago should be 5 profiles
[Stellar]>>> # for a total of 15 profiles
[Stellar]>>> LENGTH(PROFILE_GET('stat', 'global', PROFILE_WINDOW('5 minute window every 10 minutes starting from 2 minutes ago until 32 minutes ago excluding holidays:us')))
15
[Stellar]>>> STATS_MEAN(STATS_MERGE(PROFILE_GET('stat', 'global', PROFILE_WINDOW('5 minute window every 10 minutes starting from 2 minutes ago until 32 minutes ago excluding holidays:us'))))
0.028019658637877063
[Stellar]>>>

Delete any value index that currently exists (if any do) via curl -XDELETE "http://localhost:9200/value*"
Wait for a couple of seconds and run
- curl "http://localhost:9200/value*/_search?pretty=true&q=*:*" 2> /dev/null
- You should see values in the index with non-zero fields:
  - num_profiles_enrichment should be 15
  - num_profiles_parser should be 15
  - mean_enrichment should be a non-zero double
  - mean_parser should be a non-zero double
    For reference, a sample message for me is:

 "_index" : "value_index_2017.02.17.18",
      "_type" : "value_doc",
      "_id" : "AVpNPI8JQV00TRR_I4zn",
      "_score" : 1.0,
      "_source" : {
        "adapter:stellaradapter:end:ts" : "1487354498620",
        "threatinteljoinbolt:joiner:ts" : "1487354498628",
        "enrichmentsplitterbolt:splitter:end:ts" : "1487354498576",
        "num_profiles_parser" : 15,
        "enrichmentsplitterbolt:splitter:begin:ts" : "1487354498571",
        "enrichmentjoinbolt:joiner:ts" : "1487354498622",
        "mean_enrichment" : 0.025770908095283665,
        "adapter:stellaradapter:begin:ts" : "1487354498578",
        "source:type" : "value",
        "original_string" : "{ \"value\" : -0.274471660322 }",
        "threatintelsplitterbolt:splitter:begin:ts" : "1487354498625",
        "num_profiles_enrichment" : 15,
        "threatintelsplitterbolt:splitter:end:ts" : "1487354498625",
        "value" : -0.274471660322,
        "mean_parser" : 0.025770908095283665,
        "timestamp" : 1487354498529
      }

Here we've validated that the new window function can be called from the relevant topologies as well as the REPL and give consistent results that make sense.

ottobackwards · 2017-02-17T18:39:35Z

Not for nothing @cestella but those steps should just be a script if we are going to need them all the time

justinleet · 2017-02-21T20:37:34Z

...profiler-client/src/main/antlr4/org/apache/metron/profiler/client/window/generated/Window.g4

+INCLUDE : 'include' | 'INCLUDE' | 'includes' | 'INCLUDES' | 'including' | 'INCLUDING';
+EXCLUDE : 'exclude' | 'EXCLUDE' | 'excludes' | 'EXCLUDES' | 'excluding' | 'EXCLUDING';
+
+NOW : 'NOW' | 'now';


It doesn't look like "NOW" is used. Can we drop this token?

justinleet · 2017-02-21T20:47:10Z

Ran this up and tested per @cestella instructions, along with testing out the language a bit more. Everything worked well for me.

My only gotcha, and this is more a general Stellar issue, rather than an issue here is that if you make a mistake with a syntax error, e.g. STATS_MEAN(STATS_MERGE(PROFILE_GET('stat', 'global', PROFILE_WINDOW('5 minute window starting ago')))), results in org.apache.metron.common.dsl.ParseException: Unable to execute: expected at least 3 argument(s), found 2 . While technically true (because the PROFILE_WINDOW() threw an error, resulting in PROFILE_GET() only having two args), it's not a particularly helpful error. I'm curious if you have any thought on bubbling up the most specific error @cestella

Otherwise, I'm +1 on this. It was very flexible, easy to use, and a great contribution. Thanks a lot!

cestella · 2017-02-21T20:53:40Z

@justinleet yeah, there's a bug right now with stellar not bubbling up the inner exception. I created https://issues.apache.org/jira/browse/METRON-732 to track it.

cestella · 2017-02-21T20:56:03Z

@ottobackwards Yes, your observation is a good one. I wanted to toe the line between "obvious step-by-step" and "automated" and ended up on the side of "step-by-step obvious" because

I wanted people to be able to tinker and play with the steps to find things that I didn't
I didn't want to have to deal with weird one-off timing errors around some of the things specified here

It's probably worth discussing acceptance testing scripts as a general category of testing that we might be able to do with some of these scripts forming the beginnings of them.

ottobackwards · 2017-02-21T21:12:01Z

Honestly, I was thinking more about the 'kill everything' part, which seems to be repeated a lot

cestella · 2017-02-21T21:14:10Z

Oh, well, I was honestly hoping that'd go away naturally if we can trim down vagrant to give us enough headroom to not have to kill things to have it run smoothly, but your point is well taken.

nickwallen

Really nice grammar. Seems like it was a lot of work. Nice job.

nickwallen · 2017-02-21T20:45:54Z

metron-analytics/metron-profiler-client/src/main/java/Window.tokens

@@ -0,0 +1,17 @@
+COMMA=1


Is there no way to put Window.tokens and WindowLexer.tokens in the package that you created (org/apache/metron/profiler/client/window)?

I haven't found a way. I looked into it with Stellar and it didn't seem possible at the time.

nickwallen · 2017-02-21T20:47:34Z

...ofiler-client/src/main/java/org/apache/metron/profiler/client/stellar/IntervalPredicate.java

+import java.util.function.Function;
+import java.util.function.Predicate;
+
+public class IntervalPredicate<T> implements Predicate<T> {


This class seems important. Can you add a brief javadoc describing what this class does? Same for Window, WindowProcessor, HolidaysPredicate, etc. Would go along way helping someone figure out how all this logic fits together without having to read through the entirety of the code to understand it.

That's a fair ask. I'll rethink the docs and add javadocs tomorrow AM.

nickwallen · 2017-02-21T20:48:58Z

...-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/WindowLookback.java

+@Stellar(
+      namespace="PROFILE",
+      name="WINDOW",
+      description="The profiler periods associated with a fixed lookback starting from now.",


Description needs updated. Seems like copy-paste from PROFILE_LOOKBACK.

nickwallen · 2017-02-21T21:28:44Z

...-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java

+    return ret;
+  }
+
+  public static Window parse(String statement) throws ParseException {


Why do you call this parse? If I am reading the code correctly it seems like this tokenizes, parses, executes the expression and returns a result.

What would you like it to be called? Maybe process?

Sure process or execute sound good to me.

nickwallen · 2017-02-21T21:35:09Z

...-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java

+    return treeBuilder.getWindow();
+  }
+
+  public static String syntaxTree(String statement) {


Maybe my in-browser search function isn't working, but I don't see who callssyntaxTree. I also don't understand how it is different from parse. When do I use this versus parse? Comments explaining this would be super awesome.

There also seems to be some common code between parse and syntaxTree that could be factored out somehow.

Nobody is using syntaxTree, that's currently only for debugging issues with the grammar going forward for people adapting the language for new use-cases. Also, @mattf-horton asked for it during the dev discussion. It was marginally useful during the construction of the grammar to understand how things were being parsed. I'll document it as such.

nickwallen · 2017-02-21T21:55:41Z

metron-analytics/metron-profiler-client/README.md

+
+* <span style="color:red">`time_interval WINDOW?`</span><span style="color:purple">`(INCLUDING specifier_list)? (EXCLUDING specifier_list)?`</span>
+* <span style="color:red">`time_interval WINDOW?`</span><span style="color:green">`EVERY time_interval`</span><span style="color:blue">`FROM time_interval (TO time_interval)?`</span><span style="color:purple">`(INCLUDING specifier_list)? (EXCLUDING specifier_list)?`</span>
+* <span style="color:blue">`FROM time_interval (TO time_interval)?`</span>


Heck of a job trying to explain the grammar. I unfortunately still find it a little hard to grasp through the README. Unfortunately, this is probably part of the burden of defining a DSL.

I had a few random thoughts on what might help.

(1) You defined the 4 major components; Total Temporal Duration, Temporal Window Width, etc. I don't see how those relate to where you are describing "the language fits the following three forms". Would it help to define a high-level view of the DSL using those exact terms?

(2) Adding self-referencing links within the document itself might go a long way. For example, when describing the "language fits the following three forms" include links to where each section is described.

(3) It seems like time_interval and specifier_list are sub-components of your 4 major components. Maybe don't use those terms until describing each major component specifically?

Regarding 2, that is the intention of the color coding of the examples, which is lost in github but available in the site-book.

I'm going to think on these good suggestions and tackle them tomorrow am.

I'll have to checkout the site book. Its too bad, Github doesn't render that.

Yeah, I was bummed.

Actually, the color coding should link the three major forms with the types of clauses used to construct the major forms, so for 1 it becomes more clear if you look at it from the site-book as well.

I went ahead and made it color coded and made links. I also verified that it renders properly in the site-book.

nickwallen · 2017-02-21T22:15:15Z

metron-analytics/metron-profiler-client/README.md

+   * <span style="color:blue">`from 1 hour ago until 30 minutes ago`</span>
+   * <span style="color:blue">`from 30 minutes ago until 1 hour ago`</span>
+   * <span style="color:blue">`starting from 1 hour ago to 30 minutes ago`</span>
+   * <span style="color:blue">`starting from 1 hour to 30 minutes`</span>


Should I be able to do this? It does not seem to like floats and throws an exception.

from 1.5 hours ago

We do not support floats just yet as we utilize TimeUnit for our time conversion. If you want more granular than hours, then it was intended that you go down to minutes: 90 minutes ago

I made this more explicit in the docs.

nickwallen · 2017-02-21T22:18:23Z

metron-analytics/metron-profiler-client/README.md

+Due to the vagaries of the english language, the from and the to portions, if both specified, are interchangeable
+with regard to which one specifies the start and which specifies the end.  
+
+In other words <span style="color:blue">`starting from 1 hour ago to 30 minutes ago`</span> and


Are the time intervals exclusive or inclusive? Might be good to call that out specifically, if you haven't.

All time intervals are inclusive, but it's worth mentioning in the docs.

nickwallen · 2017-02-21T22:43:25Z

metron-analytics/metron-profiler-client/README.md

+* `TO` - Can be the words "until" or "to"
+* `AGO` - Optionally the word "ago"
+
+The `TO time_interval AGO` portion is optional.  If unspecified then it is expected that the time interval ends now.


One thing to consider is that we seem to be assuming "processing time" with the grammar. For example, in the testing notes that you provided, we are enriching the message with an expression like this.

STATS_MEAN(STATS_MERGE(PROFILE_GET('stat', 'global', PROFILE_WINDOW('5 minute window every 10 minutes starting from 2 minutes ago until 32 minutes ago excluding holidays:us'))))

When we grab the profile from "2 minutes ago" this will be 2 minutes ago from processing time (aka system time) rather than the event time of the message.

Is there any way to support event time now? I'm not sure that this is needed right now, but thought we should have a discussion around this at the very least.

Yes, you can pass the event time as the second argument to PROFILE_WINDOW.

That's great.

nickwallen · 2017-02-21T22:46:50Z

metron-analytics/metron-profiler-client/README.md

+
+**Examples**
+
+* A repeating 30 minute window starting 2 hours ago and repeating every hour until now.


How do I select the last 1 hour window for the past 8 Tuesdays (assuming today is Tuesday)?

How do I select the last 1 hour window for the past 8 'current day of the week'?

1 hour window every 24 hours starting from 56 days ago including tuesdays

1 hour window every 24 hours starting from 56 days ago including this day of the week

This was by far my favorite question so far. A subtle overflow bug was caught during its answer. There's now a unit test for this one. ;)

Very nice. If there is not an example like this in the README (I probably missed it) it would be good to add. This is closer to the type of use case that you were trying to solve initially with this PR.

good feedback, I'll add that.

cestella · 2017-02-21T23:56:17Z

Ok, good catch, I've made the functionality and some of the doc changes requested @nickwallen and I'll finish the rest in the morning.

cestella · 2017-02-22T02:02:34Z

Ok @nickwallen I think I have adjusted the docs and functionality sufficiently well that I have addressed your issues. Would you please give it another look and let me know if I missed anything?

nickwallen

+1 Very nice. Love it.

cestella added 5 commits February 9, 2017 17:43

Create a DSL-based timestamp lookup for profiler to enable sparse win…

04474a6

…dows

Merge branch 'master' into METRON-690-squashed

eb729a8

Removed jodatime dependency as unnecessary

51381e7

Updating docs and dependencies.

877ba3d

Finished readme

8c36be9

cestella added 3 commits February 9, 2017 21:36

Fixed weird spacing on AST class.

9c0bc91

Missed a few.

b5cc6d4

Making exceptions throw properly

c1398d5

Moved from a stack to a deque, becuase @justinleet made fun ot of off…

fe6c803

…line...and it's the right thing to do.

justinleet reviewed Feb 10, 2017

View reviewed changes

Updating test.

8d3d298

cestella added 3 commits February 17, 2017 13:03

updating.

51faaa2

Merge branch 'master' into METRON-690-squashed

1fe5256

Updating readme to react to james

928d59e

cestella closed this Feb 21, 2017

cestella reopened this Feb 21, 2017

justinleet reviewed Feb 21, 2017

View reviewed changes

Removing NOW token.

869f937

nickwallen reviewed Feb 21, 2017

View reviewed changes

cestella added 3 commits February 21, 2017 20:02

Javadoc added.

5f00ec0

Updated docs.

634ae2c

Updating docs again.

aacc201

cestella closed this Feb 22, 2017

cestella reopened this Feb 22, 2017

nickwallen mentioned this pull request Feb 22, 2017

METRON-733: Remove Geo database from ParserBolt #461

Closed

nickwallen approved these changes Feb 22, 2017

View reviewed changes

Merge branch 'master' into METRON-690-squashed

9288819

asfgit closed this in 84d3471 Feb 24, 2017


		Examples

		* A repeating 30 minute window starting 2 hours ago and repeating every hour until now.

METRON-690: Create a DSL-based timestamp lookup for profiler to enable sparse windows #450

METRON-690: Create a DSL-based timestamp lookup for profiler to enable sparse windows #450

Conversation

cestella commented Feb 10, 2017

cestella commented Feb 10, 2017

cestella commented Feb 10, 2017

justinleet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

james-sirota commented Feb 13, 2017 • edited

cestella commented Feb 13, 2017

james-sirota commented Feb 13, 2017 • edited

cestella commented Feb 17, 2017

cestella commented Feb 17, 2017 • edited

Free Up Space on the virtual machine

Preliminaries

Deploy the custom parser

Start the profiler

Test Case

ottobackwards commented Feb 17, 2017

Choose a reason for hiding this comment

justinleet commented Feb 21, 2017

cestella commented Feb 21, 2017

cestella commented Feb 21, 2017

ottobackwards commented Feb 21, 2017

cestella commented Feb 21, 2017

nickwallen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cestella commented Feb 21, 2017

cestella commented Feb 22, 2017

nickwallen left a comment • edited

Choose a reason for hiding this comment

james-sirota commented Feb 13, 2017 •

edited

james-sirota commented Feb 13, 2017 •

edited

cestella commented Feb 17, 2017 •

edited

nickwallen left a comment •

edited