Reduce makers #127

michael-mclawhorn · 2016-12-19T17:28:20Z

Big simplification of makers

archolewa

I love the use of a Function to cut down on the boilerplate. Had a few suggestsions here and there, but overall this looks great.

archolewa · 2016-12-19T17:33:04Z

...c/main/java/com/yahoo/bard/webservice/data/config/metric/makers/AggregationAverageMaker.java

@@ -35,6 +38,10 @@
 * takes aggregated data from a finer time grain (i.e. DefaultTimeGrain.DAY) and computes an average across a coarser
 * time grain (i.e. DefaultTimeGrain.WEEK). For example, given the total number of visitors to www.example.com for each
 * day of the week, we can compute the average number of daily visitors to example.com for the entire week.
+ * <p>
+ * A nested average requires the following columns, an inner query constant, and outer query sum of that constant,


Should there be a colon after following columns instead of a comma?

Also this paragraph is extremely dense, and I couldn't keep my eyes from glazing as I looked at it. I would suggest something like:

An inner query constant c to sum, and the outer query sum of c: cs.

An inner query (post) aggregation a to sum, and an outer query sum of a: as

An outer query post aggregation that computes as / cs.

archolewa · 2016-12-19T17:47:07Z

...c/main/java/com/yahoo/bard/webservice/data/config/metric/makers/AggregationAverageMaker.java

-            newInnerPostAggregations.add((PostAggregation) sourceMetric);
-        }
+        Set<PostAggregation> newInnerPostAggregations = !(sourceMetric instanceof PostAggregation) ?
+                Collections.emptySet() : ImmutableSet.of((PostAggregation) sourceMetric);


I would flip these conditions, and make the check a positive check rather than a negative one.

archolewa · 2016-12-19T17:48:31Z

...c/main/java/com/yahoo/bard/webservice/data/config/metric/makers/AggregationAverageMaker.java

-        return originalSourceMetric;
+        return !(originalSourceMetric instanceof SketchAggregation) ?
+                originalSourceMetric :
+                sketchConverter.asSketchEstimate((SketchAggregation) originalSourceMetric);


I would flip these conditions, and make the check a positive check rather than a negative one.

archolewa · 2016-12-19T17:49:00Z

...c/main/java/com/yahoo/bard/webservice/data/config/metric/makers/AggregationAverageMaker.java

-        }
-        return result;
+        return originalAggregations.stream()
+                .map(agg -> ! agg.isSketch() ? agg : sketchConverter.asInnerSketch((SketchAggregation) agg))


I would flip these conditions, and make the check a positive check rather than a negative one.

archolewa · 2016-12-19T17:49:35Z

...c/main/java/com/yahoo/bard/webservice/data/config/metric/makers/AggregationAverageMaker.java

-        return result;
+        return originalAggregations.stream()
+                .map(agg -> ! agg.isSketch() ? agg : sketchConverter.asInnerSketch((SketchAggregation) agg))
+                .collect(Collectors.toCollection(LinkedHashSet::new));
    }

    /**
     * Create an Aggregation for summing on a metric from an inner query.


Should probably toss in a blank line and a <p> tag here, to separate the summary from the more detailed description.

That hasn't been our rule. The summary ends with a period and newline. Further commentary may be offset, but need not be. The sentence end will be caught by the javadoc tool and pulled out to a summary.

When one is adding an extra line or two to the summary, I'm fine with that. But since this is a rather detailed paragraph, I was just thinking that it would look better if it was offset into its own paragraph from the summary, in the method's more detailed view.

Fair enough

archolewa · 2016-12-19T17:55:03Z

fili-core/src/main/java/com/yahoo/bard/webservice/data/config/metric/makers/CountMaker.java

@@ -2,22 +2,20 @@
 // Licensed under the terms of the Apache license. Please see LICENSE.md file distributed with this work for terms.
 package com.yahoo.bard.webservice.data.config.metric.makers;

+import static java.util.Collections.emptySet;


Why the static import? A static import makes sense to me if the class name is really long and we're using a function/constant a lot, but Collections isn't that long, and emptySet is only used once. So I would argue that the readability benefits of Collections.emptySet() outweighs the benefits from reducing verbosity.

archolewa · 2016-12-19T17:55:17Z

fili-core/src/main/java/com/yahoo/bard/webservice/data/config/metric/makers/CountMaker.java

+                query,
+                NO_OP_MAPPER,
+                metricName
+        );


Does this need to be wrapped?

archolewa · 2016-12-19T17:56:41Z

.../main/java/com/yahoo/bard/webservice/data/config/metric/makers/FilteredAggregationMaker.java

+     *
+     * @return The aggregation used by this metric
+     */
+    private Aggregation assertDependentIsAggregationMetric(final LogicalMetric sourceMetric) {


final not necessary on final LogicalMetric sourceMetric.

archolewa · 2016-12-19T17:57:42Z

fili-core/src/main/java/com/yahoo/bard/webservice/data/config/metric/makers/MetricMaker.java

+                .map(metrics::get)
+                .map(LogicalMetric::getTemplateDruidQuery)
+                .reduce(TemplateDruidQuery::merge)
+                .orElseGet(() -> {


Should use orElseThrow here instead.

archolewa · 2016-12-19T17:58:03Z

...main/java/com/yahoo/bard/webservice/data/config/metric/makers/RawAggregationMetricMaker.java


 /**
 * Metric maker which only creates aggregations.  They do not depend on other metrics and cannot be
 * validated against the metric dictionary
 */
 public abstract class RawAggregationMetricMaker extends MetricMaker {

+    private static final int DEPENDENT_METRICS_REQUIRED = 1;
+
+    private final BiFunction<String, String, Aggregation> aggregationFactory;


Awww yeah...

archolewa · 2016-12-19T23:02:32Z

👍

archolewa · 2016-12-20T15:55:15Z

👍 Reafirming my plus one.

cdeszaq · 2016-12-20T21:05:07Z

...c/main/java/com/yahoo/bard/webservice/data/config/metric/makers/AggregationAverageMaker.java

    }

    /**
     * Copy a set of aggregations, replacing any sketch aggregations with sketchMerge aggregations.
+     * This is an artifact of earlier sketch code which didn't have a single sketch type that could be used without
+     * finalization in inner queries.


If this is an artifact and is no longer needed, should we deprecate it?

Not until we remove that deprecated code, I believe.

If that's the case, can we @Deprecate this method as well, so that when we do the next sweep to remove deprecated code, this will be flagged and we can remove it?

cdeszaq · 2016-12-20T21:11:52Z

...-core/src/main/java/com/yahoo/bard/webservice/data/config/metric/makers/ArithmeticMaker.java

+     * @param resultSetMapper  The mapping function to be applied to the result that is returned by the query that is
+     * built from the LogicalMetric which is built by this maker.
+     *
+     * @deprecated to override default mapping, use the Function constructor


If we're deprecating this, we should also:

Convert the test using this constructor to use the non-deprecated version

Also deprecate the ColumnMapper interface

Alternatively, if we think there's value in the existence of the ColumnMapper interface, and the JavaDoc on it's withColumName method, we should keep it...

Deprecating

cdeszaq · 2016-12-20T21:42:44Z

...-core/src/main/java/com/yahoo/bard/webservice/data/config/metric/makers/ArithmeticMaker.java

     *
     * @param metricDictionary  The dictionary used to resolve dependent metrics when building the LogicalMetric
     * @param function  The arithmetic operation performed by the LogicalMetrics constructed by this maker
-     * @param resultSetMapper  The function to be applied to the result that is returned by the query
+     * @param resultSetMapperBuilder  A builder for a function to be applied to the result that is returned by the query


If we're making this change, we should take the opportunity to make the param comment more clear / easier to understand. As it is, this doc doesn't really say anything about what this param is for / expected to do.

cdeszaq · 2016-12-20T21:44:52Z

...-core/src/main/java/com/yahoo/bard/webservice/data/config/metric/makers/ArithmeticMaker.java

@@ -58,32 +83,32 @@ public ArithmeticMaker(
     */
    public ArithmeticMaker(MetricDictionary metricDictionary, ArithmeticPostAggregationFunction function) {
        // TODO: Deprecate me, mappers should always be specified at creation time, not implicitly


Since we're changing a bunch of things anyways, should we deprecate this method as well?

I don't know if I actually agree with this TODO. Does anyone know why this is asked for?

The idea behind this TODO is that we shouldn't apply a SketchRoundUpMapper to everything that uses the ArithmeticMaker, and instead require those who want that to add the mapper they want explicitly.

For example, someone might want to add 2 estimated sketches and have the result be a floating-point, rather than an integer. The Maker should not be opinionated.

cdeszaq · 2016-12-20T21:45:53Z

...-core/src/main/java/com/yahoo/bard/webservice/data/config/metric/makers/ArithmeticMaker.java

+        this(
+                metricDictionary,
+                function,
+                (Function<String, ResultSetMapper>) (String name) -> new SketchRoundUpMapper(name)


Can replace lambda with method reference

cdeszaq · 2016-12-20T22:49:22Z

...oovy/com/yahoo/bard/webservice/data/config/metric/makers/FilteredAggregationMakerSpec.groovy

        given: "The name of the metric the maker depends on, and the maker itself"

-        Aggregation sumAggregation = new LongSumAggregation("tmp_metric", DEPENDENT_METRIC_NAME)
+        LogicalMetric metric = longSumMaker.make("longSum", DEPENDENT_METRIC_NAME)
        Dimension dim = new KeyValueStoreDimension("d", "des", new LinkedHashSet<DimensionField>(), Mock(KeyValueStore), Mock(SearchProvider))


This can be simplified using a set literal, rather than constructing a new LHS:

Dimension dim = new KeyValueStoreDimension("d", "des", [], Mock(KeyValueStore), Mock(SearchProvider))

cdeszaq · 2016-12-20T22:50:29Z

...om/yahoo/bard/webservice/data/config/metric/makers/RawAggregationMetricMakerImplsSpec.groovy

+/**
+ *
+ * Created by mclawhor on 12/19/16.
+ */


cdeszaq · 2016-12-20T23:06:29Z

...om/yahoo/bard/webservice/data/config/metric/makers/RawAggregationMetricMakerImplsSpec.groovy

+    def "Expected numeric aggregation is produced for #makerClass.getSimpleName()"() {
+        setup:
+        RawAggregationMetricMaker maker = makerClass.newInstance()
+        expect:


Blank line between blocks

cdeszaq · 2016-12-20T23:06:33Z

...om/yahoo/bard/webservice/data/config/metric/makers/RawAggregationMetricMakerImplsSpec.groovy

+    def "Expected sketch aggregation is produced for #makerClass.getSimpleName()"() {
+        setup:
+        RawAggregationMetricMaker maker = makerClass.newInstance((MetricDictionary) null, 5)
+        expect:


Blank line between blocks

cdeszaq · 2016-12-20T23:08:02Z

...om/yahoo/bard/webservice/data/config/metric/makers/RawAggregationMetricMakerImplsSpec.groovy

+    public static String FIELD_NAME = "BAR"
+
+    @Unroll
+    def "Expected numeric aggregation is produced for #makerClass.getSimpleName()"() {


I don't think the call to getSimpleName() is needed, since I believe Groovy defaults to that. That said, I also don't think method calls will work correctly with unrolling, only property-style accessors, so this should be #makerClass.simpleName

(Applies to other test as well)

cdeszaq · 2016-12-21T16:50:31Z

CHANGELOG.md


 - [Deprecated MetricMaker.getDependentQuery lookup method in favor of simpler direct access](https://github.com/yahoo/fili/pull/124)

 - [Default DimensionColumn name to use apiName instead of physicalName](https://github.com/yahoo/fili/pull/115)
-    * Deprecated `TableUtils::getColumnNames(DataApiRequest, DruidAggregationQuery, PhysicalTable)` returning dimension physical name,
+    * Deprecated `TableUtils::getColumnNames(DataApiRequest, DruidAggregationQuery, PhysicalTable)` returning dimension physical ltername,


cdeszaq · 2016-12-21T19:43:58Z

...-core/src/main/java/com/yahoo/bard/webservice/data/config/metric/makers/ArithmeticMaker.java

@@ -58,32 +83,32 @@ public ArithmeticMaker(
     */
    public ArithmeticMaker(MetricDictionary metricDictionary, ArithmeticPostAggregationFunction function) {
        // TODO: Deprecate me, mappers should always be specified at creation time, not implicitly


The idea behind this TODO is that we shouldn't apply a SketchRoundUpMapper to everything that uses the ArithmeticMaker, and instead require those who want that to add the mapper they want explicitly.

For example, someone might want to add 2 estimated sketches and have the result be a floating-point, rather than an integer. The Maker should not be opinionated.

cdeszaq · 2016-12-21T19:45:53Z

.../main/java/com/yahoo/bard/webservice/data/config/metric/makers/FilteredAggregationMaker.java

 */
 public class FilteredAggregationMaker extends MetricMaker {

+    protected static final Logger LOG = LoggerFactory.getLogger(ConstantMaker.class);


Wrong class

…tion.

… to create the mapper.

… the original NumberFormatException (Config Behavior Change)

…ssing in a metric instead of binding to a single metric aggregation. Made the FilteredAggregation constructor use the column name as as the metric, as per convention elsewhere.

cdeszaq · 2017-01-03T15:24:24Z

...-core/src/main/java/com/yahoo/bard/webservice/data/config/metric/makers/ArithmeticMaker.java

+        this(
+                metricDictionary,
+                function,
+                (Function<String, ResultSetMapper>) (ignore) -> NO_OP_MAPPER


Parens around ignore (ie. the lambda input variable) are not technically needed, since there's no type on it and there's only 1 variable.

/NotABlocker

cdeszaq · 2017-01-03T15:33:05Z

.../main/java/com/yahoo/bard/webservice/data/config/metric/makers/FilteredAggregationMaker.java

+
+    /**
+     * The metric dictionary used to resolve dependent metrics.
+     */
    protected final MetricDictionary metricDictionary;


Can/should we kick these over to be private instead of protected? My gut says we should generally not encourage extension, since it makes it harder to change internal variables like these...

/NotABlocker

cdeszaq · 2017-01-03T15:39:03Z

...c/main/java/com/yahoo/bard/webservice/data/config/metric/makers/SketchSetOperationMaker.java

@@ -27,7 +27,7 @@

    private static final int DEPENDENT_METRICS_REQUIRED = 2;

-    final SketchSetOperationPostAggFunction function;
+    private final SketchSetOperationPostAggFunction function;


Probably should call this change out in the change log. It's unlikely this was extended, but just in case...

cdeszaq

A few small things that could be tightened up, but nothing major. 👍

michael-mclawhorn force-pushed the ReduceMakers branch from 202368e to 0916584 Compare December 19, 2016 17:33

archolewa requested changes Dec 19, 2016

View reviewed changes

archolewa approved these changes Dec 19, 2016

View reviewed changes

michael-mclawhorn force-pushed the ReduceMakers branch from 079e022 to dd6875d Compare December 20, 2016 21:53

michael-mclawhorn requested a review from cdeszaq December 20, 2016 21:55

cdeszaq requested changes Dec 20, 2016

View reviewed changes

michael-mclawhorn force-pushed the ReduceMakers branch from 4f2e364 to dbe1a4c Compare December 20, 2016 23:58

cdeszaq reviewed Dec 21, 2016

View reviewed changes

cdeszaq requested changes Dec 21, 2016

View reviewed changes

michael-mclawhorn force-pushed the ReduceMakers branch from f3896d8 to 53175b6 Compare December 21, 2016 21:34

michael-mclawhorn added 6 commits December 22, 2016 13:43

Streamy refactoring to simplify ideas and clarifications to documenta…

57a9ff3

…tion.

Removed the need for a partially specified mapper by using a delegate…

0dd0ead

… to create the mapper.

Made unparseable numbers return as IllegalArgumentException, wrapping…

7e9a453

… the original NumberFormatException (Config Behavior Change)

Simple cleanups

2a7cb5f

Simplified and centralized RawAggregationMakers and added tests.

c9eecc3

Made filtered aggregation maker follow the pattern of other makers pa…

43d85da

…ssing in a metric instead of binding to a single metric aggregation. Made the FilteredAggregation constructor use the column name as as the metric, as per convention elsewhere.

michael-mclawhorn force-pushed the ReduceMakers branch from 53175b6 to 0ac197c Compare December 22, 2016 19:43

cdeszaq reviewed Jan 3, 2017

View reviewed changes

cdeszaq approved these changes Jan 3, 2017

View reviewed changes

michael-mclawhorn added 4 commits January 3, 2017 11:13

Documentation, whitespace, logger fixes

7ba6157

Deprecation and privacy changes

1c3a9bf

Functional dependency refactor

af2f6be

CHANGELOG updated

7f88179

michael-mclawhorn force-pushed the ReduceMakers branch from 8ffc0fe to 7f88179 Compare January 3, 2017 17:16

michael-mclawhorn added the MERGEABLE label Jan 3, 2017

michael-mclawhorn merged commit 7f88179 into master Jan 3, 2017

michael-mclawhorn deleted the ReduceMakers branch January 3, 2017 17:50

Reduce makers #127

Reduce makers #127

Conversation

michael-mclawhorn commented Dec 19, 2016

archolewa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

archolewa commented Dec 19, 2016

archolewa commented Dec 20, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdeszaq left a comment

Choose a reason for hiding this comment