[HYDRATOR-552] Adding Sampling plugin #1

romy-khetan · 2017-01-30T14:15:31Z

No description provided.

russorat · 2017-02-02T09:20:53Z

docs/Sampling-batchaggregator.md

+i.e, Systematic Sampling and Reservoir Sampling.
+
+
+Use Case


remove this section

russorat · 2017-02-02T09:21:54Z

docs/Sampling-batchaggregator.md

+**sampleSize:** The number of records that needs to be sampled from the input records.
+
+**samplePercentage:** The percentage of records that needs to be sampled from the input records. Either of
+'samplePercentage' or 'sampleSize' needs to be mentioned.


needs to be mentioned -> should be specified for this plugin.

russorat · 2017-02-02T09:22:30Z

docs/Sampling-batchaggregator.md

+'samplePercentage' or 'sampleSize' needs to be mentioned.
+
+**samplingType:** Type of the Sampling algorithm that needs to be used to sample the data.
+For example: Systematic or Reservoir


Type of the Sampling algorithm that should to be used to sample the data. This can be either Systematic or Reservoir.

russorat · 2017-02-02T09:23:21Z

docs/Sampling-batchaggregator.md

+For example: Systematic or Reservoir
+
+**overSamplingPercentage:** The percentage of additional records that needs to be included in addition to the input
+sample size to account for oversampling to be used in Systematic Sampling.


sample size to account for oversampling. Required for Systematic Sampling.

russorat · 2017-02-02T09:23:46Z

docs/Sampling-batchaggregator.md

+**samplingType:** Type of the Sampling algorithm that needs to be used to sample the data.
+For example: Systematic or Reservoir
+
+**overSamplingPercentage:** The percentage of additional records that needs to be included in addition to the input


that needs to be included -> that should be included

russorat · 2017-02-02T09:24:35Z

docs/Sampling-batchaggregator.md

+**random:** Random float value between 0 and 1 to be used in Systematic Sampling. If not provided, plugin will
+internally generate random value.
+
+**totalRecords:** Total number of input records to be used in Systematic Sampling.


Total number of input records for Systematic Sampling.

russorat · 2017-02-02T09:26:01Z

src/main/java/co/cask/hydrator/plugin/batch/aggreagtor/aggregator/Sampling.java

+ */
+@Plugin(type = BatchAggregator.PLUGIN_TYPE)
+@Name("Sampling")
+@Description("Sampling a large dataset flowing through this plugin to pull random records.")


Sample a large dataset allowing only a subset of records through to the next stage.

russorat · 2017-02-02T09:26:25Z

src/main/java/co/cask/hydrator/plugin/batch/aggreagtor/aggregator/Sampling.java

+
+    @Override
+    public void prepareRun(BatchAggregatorContext context) throws Exception {
+        context.setNumPartitions(1);


Why is this needed?

russorat · 2017-02-02T09:26:53Z

src/main/java/co/cask/hydrator/plugin/batch/aggreagtor/aggregator/Sampling.java

+    }
+
+    @Override
+    public void groupBy(StructuredRecord record, Emitter<String> emitter) throws Exception {


is this needed?

To group the data for sampling algorithms defined in aggregate(), this function is needed. Please suggest, if i am missing anything.

russorat · 2017-02-02T09:34:15Z

src/main/java/co/cask/hydrator/plugin/batch/aggreagtor/aggregator/Sampling.java

+
+        @Nullable
+        @Description("The number of records that needs to be sampled from the input records.")
+        private Integer sampleSize;


can this be macro enabled?

russorat · 2017-02-02T09:34:19Z

src/main/java/co/cask/hydrator/plugin/batch/aggreagtor/aggregator/Sampling.java

+
+        @Nullable
+        @Description("The percenatage of records that needs to be sampled from the input records.")
+        private Float samplePercentage;


can this be macro enabled?

russorat · 2017-02-02T09:34:27Z

src/main/java/co/cask/hydrator/plugin/batch/aggreagtor/aggregator/Sampling.java

+        @Nullable
+        @Description("The percenatage of additional records that needs to be included in addition to the input " +
+                "sample size to account for oversampling to be used in Systematic Sampling.")
+        private Float overSamplingPercentage;


can this be macro enabled?

russorat · 2017-02-02T09:34:32Z

src/main/java/co/cask/hydrator/plugin/batch/aggreagtor/aggregator/Sampling.java

+
+        @Nullable
+        @Description("Random float value between 0 and 1 to be used in Systematic Sampling.")
+        private Float random;


can this be macro enabled?

russorat · 2017-02-02T09:34:37Z

src/main/java/co/cask/hydrator/plugin/batch/aggreagtor/aggregator/Sampling.java

+
+        @Nullable
+        @Description("Total number od input records.")
+        private Integer totalRecords;


can this be macro enabled?

russorat · 2017-02-02T09:35:56Z

src/main/java/co/cask/hydrator/plugin/batch/aggreagtor/aggregator/Sampling.java

+            if (samplingType.equalsIgnoreCase(TYPE.SYSTEMATIC.toString()) && totalRecords == null) {
+                throw new IllegalArgumentException("Please provide value for 'Total Records' when selecting sampling " +
+                        "type as 'Systematic'.");
+            }


do you want to check for things like ranges here? like, if i enter a random number of 47?

ranges added

…o develop

russorat · 2017-02-07T08:30:06Z

pom.xml

+  <parent>
+    <artifactId>hydrator-plugins</artifactId>
+    <groupId>co.cask.hydrator</groupId>
+    <version>1.6.0-SNAPSHOT</version>


1.6.0 isn't out yet. this should be 1.5.1

russorat · 2017-02-07T08:31:02Z

fails checkstyle:
{code}
[ERROR] /Users/russellsavage/workspace/sampling-aggregator/src/main/java/co/cask/hydrator/plugin/batch/aggreagtor/aggregator/Sampling.java:239:15: 'if' is not followed by whitespace. [WhitespaceAround]
[ERROR] /Users/russellsavage/workspace/sampling-aggregator/src/main/java/co/cask/hydrator/plugin/batch/aggreagtor/aggregator/Sampling.java:244:15: 'if' is not followed by whitespace. [WhitespaceAround]
{code}

this means you didn't actually build the project before checking in your latest changes.

russorat · 2017-02-07T08:35:11Z

docs/Sampling-batchaggregator.md

+
+Properties
+----------
+**sampleSize:** The number of records that needs to be sampled from the input records.


please add Either of 'samplePercentage' or 'sampleSize' should be specified for this plugin.

russorat · 2017-02-07T08:35:39Z

src/main/java/co/cask/hydrator/plugin/batch/aggreagtor/aggregator/Sampling.java

+    public static class SamplingConfig extends PluginConfig {
+
+        @Nullable
+        @Description("The number of records that needs to be sampled from the input records.")


please add: Either of 'samplePercentage' or 'sampleSize' should be specified for this plugin.

russorat · 2017-02-07T08:35:49Z

src/main/java/co/cask/hydrator/plugin/batch/aggreagtor/aggregator/Sampling.java

+        private Integer sampleSize;
+
+        @Nullable
+        @Description("The percenatage of records that needs to be sampled from the input records.")


please add: Either of 'samplePercentage' or 'sampleSize' should be specified for this plugin.

russorat · 2017-02-07T08:37:23Z

src/main/java/co/cask/hydrator/plugin/batch/aggreagtor/aggregator/Sampling.java

+        private Float overSamplingPercentage;
+
+        @Nullable
+        @Description("Random float value between 0 and 1 to be used in Systematic Sampling.")


add: If not provided, plugin will internally generate random value.

romy-khetan · 2017-02-07T09:43:37Z

@russorat I have updated the code and fixed checkstyle issues. Please review.

russorat · 2017-02-13T18:13:49Z

LGTM

romykhetan added 2 commits January 30, 2017 19:44

[HYDRATOR-552] Adding Sampling plugin

8c1e07d

Updated License year in Readme file

3e23f4c

russorat reviewed Feb 2, 2017

View reviewed changes

docs/Sampling-batchaggregator.md Outdated

i.e, Systematic Sampling and Reservoir Sampling.

Use Case

Copy link

russorat Feb 2, 2017 •

edited

Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this section

russorat reviewed Feb 2, 2017

View reviewed changes

romykhetan added 3 commits February 2, 2017 20:47

Adding validations for ranges and document modification

47e66b3

Merge branch 'develop' of github.com:hydrator/sampling-aggregator int…

8fcdca8

…o develop

Modified error message

7a9a30d

russorat reviewed Feb 7, 2017

View reviewed changes

Removed checkstyle issue and modified config property descriptions

aa19b0c

Updated pom

da4f67a

russorat merged commit f3783e2 into master Feb 13, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HYDRATOR-552] Adding Sampling plugin #1

[HYDRATOR-552] Adding Sampling plugin #1

romy-khetan commented Jan 30, 2017

russorat Feb 2, 2017 •

edited

Loading

russorat Feb 2, 2017

russorat Feb 2, 2017

russorat Feb 2, 2017

russorat Feb 2, 2017

russorat Feb 2, 2017

russorat Feb 2, 2017

russorat Feb 2, 2017 •

edited

Loading

russorat Feb 2, 2017

romy-khetan Feb 2, 2017

russorat Feb 7, 2017

russorat Feb 2, 2017

russorat Feb 2, 2017

russorat Feb 2, 2017

russorat Feb 2, 2017

russorat Feb 2, 2017

russorat Feb 2, 2017

russorat Feb 7, 2017

russorat Feb 7, 2017

russorat commented Feb 7, 2017 •

edited

Loading

russorat Feb 7, 2017

russorat Feb 7, 2017

russorat Feb 7, 2017

russorat Feb 7, 2017

romy-khetan commented Feb 7, 2017

russorat commented Feb 13, 2017

[HYDRATOR-552] Adding Sampling plugin #1

[HYDRATOR-552] Adding Sampling plugin #1

Conversation

romy-khetan commented Jan 30, 2017

russorat Feb 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

russorat Feb 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

russorat commented Feb 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

romy-khetan commented Feb 7, 2017

russorat commented Feb 13, 2017

russorat Feb 2, 2017 •

edited

Loading

russorat Feb 2, 2017 •

edited

Loading

russorat commented Feb 7, 2017 •

edited

Loading