KAFKA-3209: KIP-66: more single message transforms #2374

shikhar · 2017-01-13T22:03:16Z

Renames HoistToStruct SMT to HoistField.

Adds the following SMTs:
ExtractField
MaskField
RegexRouter
ReplaceField
SetSchemaMetadata
ValueToKey

Adds HTML doc generation and updates to connect.html.

…xtract transform

… name constants

asfbot · 2017-01-13T22:35:35Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/866/
Test FAILed (JDK 7 and Scala 2.10).

asfbot · 2017-01-13T22:35:37Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/868/
Test FAILed (JDK 8 and Scala 2.11).

asfbot · 2017-01-13T22:51:09Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/863/
Test FAILed (JDK 8 and Scala 2.11).

asfbot · 2017-01-13T22:56:08Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/861/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2017-01-13T23:19:11Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/866/
Test PASSed (JDK 8 and Scala 2.12).

ewencp · 2017-01-13T23:53:19Z

connect/transforms/src/main/java/org/apache/kafka/connect/transforms/InsertField.java

@@ -37,7 +37,7 @@

 public abstract class InsertField<R extends ConnectRecord<R>> implements Transformation<R> {

-    public interface Keys {
+    public interface ConfigName {


Seems this can be private instead? Ditto for other classes. I thought it was possibly for tests but it doesn't seem to be used anywhere anyway.

ewencp · 2017-01-13T23:59:48Z

connect/transforms/src/main/java/org/apache/kafka/connect/transforms/SetSchemaMetadata.java

+    /**
+     * Set the schema name, version or both on the record's key schema.
+     */
+    public static class Key<R extends ConnectRecord<R>> extends SetSchemaMetadata<R> {


Lack of multiple inheritance is really annoying in cases like this....

Curious, how would multiple inheritance help?

All these classes use the same pattern are defining the same overrides for operatingSchema and updatedRecord. If you put that into a single abstract class, then the definition of the Key and Value types would just be one line -- the class declaration that inherits from both SetSchemaMetadata and this new abstract class that provides the overrides.

It's obviously not a big deal here since the implementations are trivial, just kind of annoying code bloat.

ewencp · 2017-01-14T00:31:39Z

connect/transforms/src/main/java/org/apache/kafka/connect/transforms/ValueToKey.java

+    public R apply(R record) {
+        final Schema valueSchema = record.valueSchema();
+        if (valueSchema == null) {
+            if (!(record.value() instanceof Map)) {


We end up doing this same set of checks in a bunch of places. Any way we could reduce the duplicated code?

I'll factor out a couple of helper functions to call into requireMap and requireStructSchema

ewencp · 2017-01-14T00:39:02Z

connect/transforms/src/main/java/org/apache/kafka/connect/transforms/ValueToKey.java

+
+        Schema keySchema = valueToKeySchemaCache.get(valueSchema);
+        if (keySchema == null) {
+            final SchemaBuilder keySchemaBuilder = SchemaBuilder.struct();


We can probably skip it for now, but a cache for this conversion might be a nice improvement eventually. I'd guess that with schemas you're probably mostly seeing the same conversion.

There is a cache here though, valueToKeySchemaCache

Clearly I was reading this without my 👓

asfbot · 2017-01-14T01:04:08Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/861/
Test FAILed (JDK 8 and Scala 2.12).

asfbot · 2017-01-17T22:45:27Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/947/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-01-17T22:46:57Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/949/
Test PASSed (JDK 8 and Scala 2.11).

shikhar · 2017-01-17T23:39:11Z

@ewencp should be ready for review.

~~Currently the new generated/connect_transforms.html page is not being included from connect.html, I'll get that into a followup PR that adds a section giving an overview of SMT and usage.~~

asfbot · 2017-01-18T00:14:57Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/947/
Test FAILed (JDK 7 and Scala 2.10).

asfbot · 2017-01-18T00:36:55Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/955/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2017-01-18T01:07:52Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/953/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-01-18T01:29:55Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/957/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-01-18T01:30:21Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/959/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2017-01-18T01:34:34Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/957/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2017-01-18T02:05:00Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/953/
Test FAILed (JDK 7 and Scala 2.10).

asfbot · 2017-01-18T21:37:04Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/995/
Test FAILed (JDK 8 and Scala 2.11).

shikhar · 2017-01-19T18:37:59Z

connect/transforms/src/main/java/org/apache/kafka/connect/transforms/RegexRouter.java

+            final String topic = matcher.replaceFirst(replacement);
+            return record.newRecord(topic, record.kafkaPartition(), record.keySchema(), record.key(), record.valueSchema(), record.value(), record.timestamp());
+        }
+        return record;


Wondering if it may be useful here to have an option to drop records (return null here) that don't match the regex against the record.topic().

Hmm, not sure. I suspect I'd actually prefer an error in that case.

shikhar · 2017-01-19T18:47:30Z

@ewencp addressed comments

asfbot · 2017-01-19T18:48:31Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1031/
Test FAILed (JDK 8 and Scala 2.12).

asfbot · 2017-01-19T18:48:31Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/1033/
Test FAILed (JDK 8 and Scala 2.11).

asfbot · 2017-01-19T18:49:09Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/1031/
Test FAILed (JDK 7 and Scala 2.10).

asfbot · 2017-01-19T19:38:16Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1033/
Test FAILed (JDK 8 and Scala 2.12).

asfbot · 2017-01-19T19:46:26Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/1035/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2017-01-19T19:50:29Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/1033/
Test PASSed (JDK 7 and Scala 2.10).

shikhar · 2017-01-19T22:44:59Z

@ewencp sneaked in ReplaceField SMT as well. Won't add any more :-)

asfbot · 2017-01-19T23:33:08Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/1044/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2017-01-19T23:39:10Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/1042/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2017-01-20T00:07:56Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1042/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-01-20T00:54:33Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1049/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-01-20T00:56:34Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/1049/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2017-01-20T01:04:23Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/1051/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2017-01-20T19:23:14Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/1073/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2017-01-20T19:26:38Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/1071/
Test FAILed (JDK 7 and Scala 2.10).

asfbot · 2017-01-20T19:27:14Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/1071/
Test PASSed (JDK 8 and Scala 2.12).

ewencp

@shikhar This LGTM. I have one comment about whether we're inadvertently advertising internal APIs as external, but this isn't really a blocker. I'm going to commit as is and we can always follow up if needed.

ewencp · 2017-01-21T00:02:23Z

connect/transforms/src/main/java/org/apache/kafka/connect/transforms/util/SchemaUtil.java

+
+import java.util.Map;
+
+public class SchemaUtil {


Are transforms ending up in the javadocs? Just curious if we're making stuff like this appear as public APIs when we really just want this to be an internal utility.

Renames `HoistToStruct` SMT to `HoistField`. Adds the following SMTs: `ExtractField` `MaskField` `RegexRouter` `ReplaceField` `SetSchemaMetadata` `ValueToKey` Adds HTML doc generation and updates to `connect.html`. Author: Shikhar Bhushan <shikhar@confluent.io> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #2374 from shikhar/more-smt

Renames `HoistToStruct` SMT to `HoistField`. Adds the following SMTs: `ExtractField` `MaskField` `RegexRouter` `ReplaceField` `SetSchemaMetadata` `ValueToKey` Adds HTML doc generation and updates to `connect.html`. Author: Shikhar Bhushan <shikhar@confluent.io> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes apache#2374 from shikhar/more-smt

Shikhar Bhushan added 5 commits January 13, 2017 13:52

SetSchemaMetadata SMT

f34cc71

Support schemaless and rename HoistToStruct->Hoist; add the inverse E…

09af66b

…xtract transform

InsertField transform -- fix bad name for interface containing config…

022f492

… name constants

ValueToKey SMT

c5260a7

Clearer naming -- HoistField, ExtractField

06b47a8

ewencp reviewed Jan 14, 2017

View reviewed changes

Address review comments; docgen coming along

cdd8855

Shikhar Bhushan added 2 commits January 17, 2017 15:37

Auto-generate docs/generated/connect_transforms.html

04dc9f3

html tweaks

e90f989

consistent doc style

692901f

Add RegexRouter SMT

5772e95

consistent naming

2b99585

MaskField SMT

1063909

shikhar commented Jan 19, 2017

View reviewed changes

Config validation, try to fail-fast on no-op configs

16c9259

checkstyle

a374bf9

ReplaceField SMT and some consistency/cleanups refactoring

7ee22e5

Shikhar Bhushan added 2 commits January 19, 2017 16:00

Minor InsertField tweak -- docs and config check

aa325db

User doc for transformations in connect.html

be67396

Better word choice

7eae01a

ewencp approved these changes Jan 21, 2017

View reviewed changes

asfgit closed this in a8aa756 Jan 21, 2017

shikhar deleted the more-smt branch January 21, 2017 01:13

kkonstantine added the connect label Oct 16, 2020

KAFKA-3209: KIP-66: more single message transforms #2374

KAFKA-3209: KIP-66: more single message transforms #2374

Conversation

shikhar commented Jan 13, 2017 • edited

asfbot commented Jan 13, 2017

asfbot commented Jan 13, 2017

asfbot commented Jan 13, 2017

asfbot commented Jan 13, 2017

asfbot commented Jan 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asfbot commented Jan 14, 2017

asfbot commented Jan 17, 2017

asfbot commented Jan 17, 2017

shikhar commented Jan 17, 2017 • edited

asfbot commented Jan 18, 2017

asfbot commented Jan 18, 2017

asfbot commented Jan 18, 2017

asfbot commented Jan 18, 2017

asfbot commented Jan 18, 2017

asfbot commented Jan 18, 2017

asfbot commented Jan 18, 2017

asfbot commented Jan 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shikhar commented Jan 19, 2017

asfbot commented Jan 19, 2017

asfbot commented Jan 19, 2017

asfbot commented Jan 19, 2017

asfbot commented Jan 19, 2017

asfbot commented Jan 19, 2017

asfbot commented Jan 19, 2017

shikhar commented Jan 19, 2017

asfbot commented Jan 19, 2017

asfbot commented Jan 19, 2017

asfbot commented Jan 20, 2017

asfbot commented Jan 20, 2017

asfbot commented Jan 20, 2017

asfbot commented Jan 20, 2017

asfbot commented Jan 20, 2017

asfbot commented Jan 20, 2017

asfbot commented Jan 20, 2017

ewencp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shikhar commented Jan 13, 2017 •

edited

shikhar commented Jan 17, 2017 •

edited