KAFKA-5671 Followup: Remove reflections in unit test classes #3603

guozhangwang · 2017-08-01T00:35:51Z

Remove rest deprecation warnings in streams:jar.
Consolidate all unit test classes' reflections to access internal topology builder from packages other than o.a.k.streams. We need to refactor the hierarchies of StreamTask, StreamThread and KafkaStreams to remove these hacky reflections.
Minor fixes such as reference path, etc.
Minor edits on web docs for the describe function under developer-guide.

guozhangwang · 2017-08-01T00:36:11Z

asfgit · 2017-08-01T00:39:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6453/
Test FAILed (JDK 7 and Scala 2.11).

asfgit · 2017-08-01T00:43:26Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6438/
Test FAILed (JDK 8 and Scala 2.12).

asfgit · 2017-08-01T03:51:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6458/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-08-01T04:15:43Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6443/
Test PASSed (JDK 8 and Scala 2.12).

dguy · 2017-08-01T09:01:38Z

streams/src/main/java/org/apache/kafka/streams/kstream/KStreamBuilder.java

@@ -51,17 +50,17 @@
 * @deprecated Use {@link org.apache.kafka.streams.StreamsBuilder StreamsBuilder} instead
 */
 @Deprecated
-public class KStreamBuilder extends TopologyBuilder {
+public class KStreamBuilder extends org.apache.kafka.streams.processor.TopologyBuilder {


why this change?

To avoid importing the org.apache.kafka.streams.processor.TopologyBuilder which will introduce a WARNING during compilation; instead we will explicitly refer to the class within the class whenever we have to.

dguy · 2017-08-01T09:06:23Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/ProcessorContextImpl.java

     */
    @SuppressWarnings("deprecation")
    @Override
    public StateStore getStateStore(final String name) {
        if (currentNode() == null) {
-            throw new TopologyBuilderException("Accessing from an unknown node");
+            throw new org.apache.kafka.streams.errors.TopologyBuilderException("Accessing from an unknown node");


Again, why?

Ditto above.

dguy · 2017-08-01T09:13:31Z

streams/src/test/java/org/apache/kafka/test/StreamsTestUtils.java

@@ -78,4 +72,24 @@ public static Properties minimalStreamsConfig() {
        return results;
    }

+    // TODO: these three static functions are added because some non-TopologyBuilder unit tests need to access the internal topology builder,
+    //       which is usually a bad sign of design patterns between TopologyBuilder and StreamThread. We need to consider getting rid of them later
+    public static InternalTopologyBuilder internalTopologyBuilder(final StreamsBuilder streamsBuilder) throws NoSuchFieldException, IllegalAccessException {


Here and the next method. Can we wrap them with:

try { .../ }catch(final Exception e) { throw new RuntimeException(e); }

We can then get rig of the throws Exception from all the tests and from the KStreamTestDriver

Do we really need to use reflection? Can we not add a test class in the same package that exposes a package private field as public? When exploring the codebase is very useful to be able to see callers and reflection kills that.

@ijuma that is possible. We'd need a couple of test helper classes in the appropriate packages, one for StreamBuilder and one for InternalStreamsBuilder. We'd also need to make internalTopologyBuilder on StreamsBuilder package private.

…into K5671-followup-comments

guozhangwang · 2017-08-01T19:25:34Z

@dguy Comments addressed, could you review again?

asfgit · 2017-08-01T19:31:53Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6460/
Test FAILed (JDK 8 and Scala 2.12).

asfgit · 2017-08-01T20:25:05Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6475/
Test PASSed (JDK 7 and Scala 2.11).

bbejeck · 2017-08-01T20:26:11Z

@guozhangwang @dguy: sorry I'm a little late with this but I was thinking instead of a static helper class we used a wrapper class instead, something like:

 package org.apache.kafka.streams;

 public final class TestingStreamsBuilderWrapper extends StreamsBuilder {

    private final StreamsBuilder delegate;

    public TestingStreamsBuilderWrapper(StreamsBuilder delegate) {
        this.delegate = delegate;
    }

    public InternalTopologyBuilder getInternalTopologyBuilder() {
         return delegate.internalTopologyBuilder;
    }

    public InternalStreamsBuilder getInternalStreamsBuilder() {
        return delegate.internalStreamsBuilder;
    }

IMHO using this as a drop in replacement for StreamsBuilder might make the current tests easier to follow and subsequent tests easier to write.

guozhangwang · 2017-08-01T21:53:34Z

retest this please.

guozhangwang · 2017-08-01T21:55:52Z

@bbejeck The issue is that internalTopologyBuilder is a package-private field of InternalStreamsBuilder and internalStreamsBuilder is a package-private field of StreamsBuilder, which are in o.a.k.streams.kstream.internals and o.a.k.streams respectively. So if your helper class in only in o.a.k.streams it will not be able to access the first field.

bbejeck · 2017-08-01T22:33:50Z

@guozhangwang that's right, we'd need to add an additional helper class, let me know if you think the idea has legs, otherwise I'll just leave it as is.

asfgit · 2017-08-01T22:53:27Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6481/
Test PASSed (JDK 7 and Scala 2.11).

guozhangwang · 2017-08-01T23:09:56Z

@bbejeck In my latest commit I'm doing similar things but in an existing class than adding additional helper classes. LMK if you find that's good enough.

asfgit · 2017-08-01T23:21:43Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6466/
Test PASSed (JDK 8 and Scala 2.12).

asfgit · 2017-08-02T01:04:22Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6486/
Test FAILed (JDK 7 and Scala 2.11).

asfgit · 2017-08-02T01:04:23Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6471/
Test FAILed (JDK 8 and Scala 2.12).

dguy

Thanks @guozhangwang, LGTM.
I wonder if we should move the utility methods to get the InternalTopologyBuilder etc onto XXUtils classes rather than having them on specific Test classes? I don't have a strong opinion on this, so i'll leave it up to you to decide. Feel free to merge as is

dguy · 2017-08-02T09:38:07Z

@bbejeck that is what i was thinking, but we need to have at least a couple of helper classes in the appropriate packages (which is fine). I'm ok with how it is, though. TBH i'm not a big fan of either approach, but it is where we are at for the time being.

dguy · 2017-08-02T09:38:13Z

retest this please

asfgit · 2017-08-02T10:37:12Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6493/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-08-02T11:06:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6478/
Test PASSed (JDK 8 and Scala 2.12).

bbejeck · 2017-08-02T14:06:31Z

@guozhangwang yeah that works for me.

LGTM

bbejeck · 2017-08-02T14:06:47Z

@dguy agreed on both approaches, but for now we're not relying on reflection which is the important part IMHO.

guozhangwang · 2017-08-02T22:14:07Z

I think it is okay to keep it in the specific test class than creating two more XXTestUtil classes.

guozhangwang · 2017-08-02T22:14:11Z

Merged to trunk.

Remove reflections in unit test classes

86e1c22

guozhangwang added 2 commits July 31, 2017 18:16

update developer guide with describe

1c43c0d

minor fixes

b90fca5

dguy reviewed Aug 1, 2017

View reviewed changes

guozhangwang added 2 commits August 1, 2017 11:29

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka …

6be79de

…into K5671-followup-comments

github comments

0c798c4

rebase from trunk again

d1aba69

dguy approved these changes Aug 2, 2017

View reviewed changes

asfgit closed this in 125d69c Aug 2, 2017

guozhangwang deleted the K5671-followup-comments branch November 6, 2017 22:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-5671 Followup: Remove reflections in unit test classes #3603

KAFKA-5671 Followup: Remove reflections in unit test classes #3603

guozhangwang commented Aug 1, 2017 •

edited

Loading

guozhangwang commented Aug 1, 2017

asfgit commented Aug 1, 2017

asfgit commented Aug 1, 2017

asfgit commented Aug 1, 2017

asfgit commented Aug 1, 2017

dguy Aug 1, 2017

guozhangwang Aug 1, 2017

dguy Aug 1, 2017

guozhangwang Aug 1, 2017

dguy Aug 1, 2017

ijuma Aug 1, 2017

dguy Aug 1, 2017

guozhangwang commented Aug 1, 2017

asfgit commented Aug 1, 2017

asfgit commented Aug 1, 2017

bbejeck commented Aug 1, 2017

guozhangwang commented Aug 1, 2017

guozhangwang commented Aug 1, 2017

bbejeck commented Aug 1, 2017

asfgit commented Aug 1, 2017

guozhangwang commented Aug 1, 2017

asfgit commented Aug 1, 2017

asfgit commented Aug 2, 2017

asfgit commented Aug 2, 2017

dguy left a comment

dguy commented Aug 2, 2017

dguy commented Aug 2, 2017

asfgit commented Aug 2, 2017

asfgit commented Aug 2, 2017

bbejeck commented Aug 2, 2017

bbejeck commented Aug 2, 2017

guozhangwang commented Aug 2, 2017

guozhangwang commented Aug 2, 2017

KAFKA-5671 Followup: Remove reflections in unit test classes #3603

KAFKA-5671 Followup: Remove reflections in unit test classes #3603

Conversation

guozhangwang commented Aug 1, 2017 • edited Loading

guozhangwang commented Aug 1, 2017

asfgit commented Aug 1, 2017

asfgit commented Aug 1, 2017

asfgit commented Aug 1, 2017

asfgit commented Aug 1, 2017

dguy Aug 1, 2017

Choose a reason for hiding this comment

guozhangwang Aug 1, 2017

Choose a reason for hiding this comment

dguy Aug 1, 2017

Choose a reason for hiding this comment

guozhangwang Aug 1, 2017

Choose a reason for hiding this comment

dguy Aug 1, 2017

Choose a reason for hiding this comment

ijuma Aug 1, 2017

Choose a reason for hiding this comment

dguy Aug 1, 2017

Choose a reason for hiding this comment

guozhangwang commented Aug 1, 2017

asfgit commented Aug 1, 2017

asfgit commented Aug 1, 2017

bbejeck commented Aug 1, 2017

guozhangwang commented Aug 1, 2017

guozhangwang commented Aug 1, 2017

bbejeck commented Aug 1, 2017

asfgit commented Aug 1, 2017

guozhangwang commented Aug 1, 2017

asfgit commented Aug 1, 2017

asfgit commented Aug 2, 2017

asfgit commented Aug 2, 2017

dguy left a comment

Choose a reason for hiding this comment

dguy commented Aug 2, 2017

dguy commented Aug 2, 2017

asfgit commented Aug 2, 2017

asfgit commented Aug 2, 2017

bbejeck commented Aug 2, 2017

bbejeck commented Aug 2, 2017

guozhangwang commented Aug 2, 2017

guozhangwang commented Aug 2, 2017

guozhangwang commented Aug 1, 2017 •

edited

Loading