KAFKA-6474: Rewrite tests to use new public TopologyTestDriver [part 2] #4986

h314to · 2018-05-09T10:41:43Z

This PR is a further step towards the complete replacement of KStreamTestDriver with TopologyTestDriver.

Add task, processorTopology, and globalTopology access to TopologyTestDriverAccessor
Add condition to prevent NPE in ProcessorContextImpl
Refactor:
- KTableFilterTest
- KTableSourceTest
- KTableMapValuesTest
- KTableImplTest.

edit: To simplify the review process some straightforward changes were moved to another PR.

h314to · 2018-05-09T10:57:26Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableAggregateTest.java

 import java.util.HashMap;
 import java.util.Map;
+import java.util.Properties;

 import static org.junit.Assert.assertEquals;

 public class KTableAggregateTest {


A couple of tests in this class fail with TopologyTestDriver. My guess is that this is due to some problem with caching. Using TopologyTestDriver, regardless of the value I set for StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, I get the same results out of MockProcessor.processed as if the cache was 0. In fact, setting the KStreamTestDriver cache to zero makes it yield the exact same results as TopologyTestDriver. I'll keep looking into it. If you have any tip about what might be going wrong please let me know.

Not entirely sure, but seems to be a bug in TopologyTestDriver related to ThreadCache -- KStreamTestDriver creates a cache and uses it as member variable and passes it into store.init(...) -- the new TopologyTestDriver also creates a cache and passes it into GlobalProcessorContextImpl and the created StreamTask -- maybe, internally the cache is not forwarded correctly such that it is not used when stores are initialized.

I would need to do a deeper analysis why the cache is not forwarded to the stores correctly. Hope the pointes help. If not sufficient, please let us know and we can dig deeper into it from our side.

Thanks for the tips! I'll look into it a bit longer, to see if I can find out what's wrong. If I hit a dead end I'll let you know.

h314to · 2018-05-10T11:02:28Z

Rebased on current trunk to fix merge conficts.

h314to · 2018-05-14T10:13:23Z

streams/test-utils/src/main/java/org/apache/kafka/streams/TopologyTestDriver.java

-     * @param builder builder for the topology to be tested
-     * @param config the configuration for the topology
-     */
-    TopologyTestDriver(final InternalTopologyBuilder builder,


Removed this constructor after rebasing, since recent commits made it no longer necessary.

vvcephei · 2018-05-15T00:12:45Z

Hey @h314to , Thanks for this latest PR! I'll take a look as soon as I have a chance.

mjsax

Thanks for working on this! Couple of comments.

One more thing: parts of the changes seem to be straight forward -- can we extract them into a separate PR (again ;)) to make reviewing easier and reduce turn around time? The extracted PR could get merged quite quickly I believe.

Thanks a lot!

mjsax · 2018-05-20T21:53:59Z

streams/src/test/java/org/apache/kafka/streams/TopologyTestDriverWrapper.java

+        super(topology, config, initialWallClockTimeMs);
+    }
+
+    /** Get the processor context


nit: move to next line ("weird" formatting)

Yeah, it is wierd. Changed.

mjsax · 2018-05-20T21:54:11Z

streams/src/test/java/org/apache/kafka/streams/TopologyTestDriverWrapper.java

+        return context;
+    }
+
+    /** Identify the source node for a given topic


mjsax · 2018-05-20T21:54:23Z

streams/src/test/java/org/apache/kafka/streams/TopologyTestDriverWrapper.java

+     * @return the processor context
+     */
+    public ProcessorContext getProcessorContext(final String topicName) {
+        ProcessorContext context = task.context();


nit: add final

mjsax · 2018-05-20T21:57:35Z

streams/src/test/java/org/apache/kafka/streams/TopologyTestDriverWrapper.java

+        return names;
+    }
+
+    /** Get a processor by name


mjsax · 2018-05-20T22:10:39Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableAggregateTest.java

 import java.util.HashMap;
 import java.util.Map;
+import java.util.Properties;

 import static org.junit.Assert.assertEquals;

 public class KTableAggregateTest {


Not entirely sure, but seems to be a bug in TopologyTestDriver related to ThreadCache -- KStreamTestDriver creates a cache and uses it as member variable and passes it into store.init(...) -- the new TopologyTestDriver also creates a cache and passes it into GlobalProcessorContextImpl and the created StreamTask -- maybe, internally the cache is not forwarded correctly such that it is not used when stores are initialized.

I would need to do a deeper analysis why the cache is not forwarded to the stores correctly. Hope the pointes help. If not sufficient, please let us know and we can dig deeper into it from our side.

mjsax · 2018-05-20T22:14:53Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableImplTest.java


-        // two state store should be created
-        assertEquals(2, driver.allStateStores().size());
+            // two state store should be created


nit: fix typo store[s] ;)

mjsax · 2018-05-20T22:20:52Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableImplTest.java

-        assertTrue(driver.allProcessorNames().contains("KSTREAM-SINK-0000000007"));
-        assertTrue(driver.allProcessorNames().contains("KSTREAM-SOURCE-0000000008"));
+            // contains the corresponding repartition source / sink nodes
+            assertTrue(driver.getAllProcessorNames().contains("KSTREAM-SINK-0000000003"));


I am wondering if we should rewrite this using Topology#describe() instead of hooking into the test driver?

Actually same thought about driver.getAllStateStores() ?

That's a good idea. allProcessorNames is only used in this test, and its functionally is easily replicated by using Topology#describe().

In the case of allStateStores, it provides a bit more functionality than allProcessorNames since it returns a Map<String, StateStore>. It is also part of the public API. Getting all processor names could be done using TopologyWrapper to access the InternalTopologyBuilder, so it could in principle be removed. However, it could be useful for end users' testing, which do not have access to TopologyWrapper, so it might be nice to keep it around.

mjsax · 2018-05-20T22:29:13Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableKTableLeftJoinTest.java


-        processor.checkAndClearProcessResult("0:X0+null", "1:X1+null");
-        checkJoinedValues(getter, kv(0, "X0+null"), kv(1, "X1+null"), kv(2, null), kv(3, null));
+            processor.checkAndClearProcessResult("0:X0+null", "1:X1+null");


Wondering, if we should get rid of MockProcessor and update the test setup to pipe the result into a topic instead. Than we would use OutputVerifier instead of dealing with all those concatenated Strings.

That sounds way cleaner. If it's ok I would like to implement that in a subsequent PR, to avoid cramming even more changes into this one.

h314to

Thanks for the review! I've made the changes you recommended. I'm also splitting the straightforward changes (those which for which all the tests in a class do not require using TopologyTestDriverWrapper) into a new PR.

h314to · 2018-05-21T14:12:50Z

streams/src/test/java/org/apache/kafka/streams/TopologyTestDriverWrapper.java

+        super(topology, config, initialWallClockTimeMs);
+    }
+
+    /** Get the processor context


Yeah, it is wierd. Changed.

h314to · 2018-05-21T14:13:03Z

streams/src/test/java/org/apache/kafka/streams/TopologyTestDriverWrapper.java

+        return context;
+    }
+
+    /** Identify the source node for a given topic


h314to · 2018-05-21T14:13:13Z

streams/src/test/java/org/apache/kafka/streams/TopologyTestDriverWrapper.java

+        return names;
+    }
+
+    /** Get a processor by name


h314to · 2018-05-21T14:28:03Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableAggregateTest.java

 import java.util.HashMap;
 import java.util.Map;
+import java.util.Properties;

 import static org.junit.Assert.assertEquals;

 public class KTableAggregateTest {


Thanks for the tips! I'll look into it a bit longer, to see if I can find out what's wrong. If I hit a dead end I'll let you know.

h314to · 2018-05-21T14:28:55Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableImplTest.java


-        // two state store should be created
-        assertEquals(2, driver.allStateStores().size());
+            // two state store should be created


h314to · 2018-05-21T16:23:55Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableImplTest.java

-        assertTrue(driver.allProcessorNames().contains("KSTREAM-SINK-0000000007"));
-        assertTrue(driver.allProcessorNames().contains("KSTREAM-SOURCE-0000000008"));
+            // contains the corresponding repartition source / sink nodes
+            assertTrue(driver.getAllProcessorNames().contains("KSTREAM-SINK-0000000003"));


That's a good idea. allProcessorNames is only used in this test, and its functionally is easily replicated by using Topology#describe().

In the case of allStateStores, it provides a bit more functionality than allProcessorNames since it returns a Map<String, StateStore>. It is also part of the public API. Getting all processor names could be done using TopologyWrapper to access the InternalTopologyBuilder, so it could in principle be removed. However, it could be useful for end users' testing, which do not have access to TopologyWrapper, so it might be nice to keep it around.

h314to · 2018-05-21T16:56:15Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableKTableLeftJoinTest.java


-        processor.checkAndClearProcessResult("0:X0+null", "1:X1+null");
-        checkJoinedValues(getter, kv(0, "X0+null"), kv(1, "X1+null"), kv(2, null), kv(3, null));
+            processor.checkAndClearProcessResult("0:X0+null", "1:X1+null");


That sounds way cleaner. If it's ok I would like to implement that in a subsequent PR, to avoid cramming even more changes into this one.

mjsax · 2018-05-21T20:26:35Z

Meta comment: as it is easier to review smaller PRs, it might be worth the exclude some classes from this PR and tackle them individually via multiple PRs:

KTableAggregateTest to figure out the caching issue
don't introduce allProcessorNames() but use Topology#describe() -- exclude all classes that use allProcessorNames() and don't introduce allProcessorNames() in the first place
MockProcessor and OutputVerifier (this might be a follow up OR as the classes seem to overlap with second bullet point).

Whatever works best :) Let us know.

h314to · 2018-05-22T11:06:00Z

Yes, smaller PR are simpler to review. In order to reduce the size of this one I reverted the changes to the following classes, which will be tackled in subsequent PRs:

Reverted changes to KTableAggregateTest.
allProcessorNames is gone with my previous commit and there is no further need for it
Reverted changes to KTableKTableInnerJoinTest, KTableKTableOuterJoinTest, and KTableKTableLeftJoinTest. These are the ones MockProcessor is used more extensively, and thus, the ones which would benefit the most from being rewritten with OutputVerifier.

vvcephei

These changes look right to me. Just one question...

vvcephei · 2018-05-22T21:10:13Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/ProcessorContextImpl.java

@@ -71,7 +71,7 @@ public StateStore getStateStore(final String name) {
            return global;
        }

-        if (!currentNode().stateStores.contains(name)) {
+        if (currentNode.stateStores != null && !currentNode().stateStores.contains(name)) {


I find this case a little confusing. currentNode().stateStores == null would seem to imply also that we don't have access to the store, or any store for that matter. Why do we allow this case to pass though?

Yes, that's a bit messy. I'm going to change the way we get the processor context, and then we won't need this. Since there are already a few conflicts I'm also going to squash and rebase on current trunk.

h314to · 2018-05-23T17:32:52Z

I rebased on trunk and squashed. New changes are in a separate commit to ease review. I simplified the way the current node is set in getProcessorContext which allowed me to tidy up TopologyTestDriverWrapper a bit.

I also needed to fix a few calls to ConsumerRecordFactory#create because the recently added record header support (KAFKA-6850) made calls with null value ambiguous.

* Refactor: -KTableFilterTest.java -KTableImplTest.java -KTableMapValuesTest.java -KTableSourceTest.java * Add access to task, processorTopology, and globalTopology in TopologyTestDriver via TopologyTestDriverWrapper * Remove unnecessary constructor in TopologyTestDriver

…rent node

mjsax

Thanks for the update. Couple of minor comments.

mjsax · 2018-05-29T20:20:36Z

streams/src/test/java/org/apache/kafka/streams/TopologyTestDriverWrapper.java

+    /**
+     * Get the processor context
+     *
+     * @param processorName used to search for a processor connected to this StateStore, which is set as current node


Don't understand the comment. What StateStore does it refer to?

I think this is because this function is only for init the getters, which requires the state stores is connected, hence accessible in init to the node.

Yes, that's it. Thanks for clearing it up @guozhangwang . I tried to make it clearer while addressing your other comments.

mjsax · 2018-05-29T20:50:00Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableImplTest.java


-        // two state store should be created
-        assertEquals(2, driver.allStateStores().size());
+            // two state stores should be created


nit: remove comment -- clear from the code itself (my believe is, if a test needs comments you need to rewrite the test :) )

Good point. Done

mjsax · 2018-05-29T20:51:18Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableImplTest.java

-        // two state stores should be created
-        assertEquals(2, driver.allStateStores().size());
+        try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+            // two state stores should be created


nit: as above; remove

mjsax · 2018-05-29T20:51:33Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableImplTest.java

-        driver.setUp(builder, stateDir, null, null);
-        driver.setTime(0L);
+        try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+            // two state stores should be created


mjsax · 2018-05-29T20:54:29Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableImplTest.java

+        }
+    }
+
+    private TopologyDescription.Node getProcessor(final Topology topology, final String processorName) {


This is only used once -- would suggest to inline into assertTopologyContainsProcessor.

Additionally, the check for assertNotNull is not self-expressive. As the return type of assertTopologyContainsProcessor is void I would suggest to just call return (instead of return node;) if the node was found and replace return null;/assertNotNull with throw new AssertionError(...)

WDYT?

+1 from me.

Yes, that makes it much clearer. I changed it.

mjsax · 2018-05-29T20:58:34Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableImplTest.java


-        // three state store should be created, one for source, one for aggregate and one for reduce
-        assertEquals(3, driver.allStateStores().size());
+            // three state stores should be created, one for source, one for aggregate and one for reduce


mjsax · 2018-05-29T21:00:35Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableImplTest.java

-        assertTrue(driver.allProcessorNames().contains("KSTREAM-SOURCE-0000000004"));
-        assertTrue(driver.allProcessorNames().contains("KSTREAM-SINK-0000000007"));
-        assertTrue(driver.allProcessorNames().contains("KSTREAM-SOURCE-0000000008"));
+            // contains the corresponding repartition source / sink nodes


remove and rename test to shouldCreateSourceAndSinkNodesForRepartitioningTopic

guozhangwang

Thanks @h314to I have two meta comments regarding the newly added TopologyTestDriverWrapper:

The getProcessorContext function (I think its name should be setCurrentNodeForProcessorContext to be more accurate) is only used for initializing the value getters in our unit tests. I think we should not follow this pattern in unit test, i.e. enforcing ourselves to remember the processor name that creates the state store and set the processor node when calling init. Instead, in our internal code we call internalTopologyBuilder.connectProcessorAndStateStores to make sure the referenced node can also access to that store in its value getter. We could follow this pattern instead. I left a detailed comment as an example in KTableFilterTest.

Please let us know once you addressed @mjsax and my comments so that we can try complete removing the KStreamBuilder before 2.0

guozhangwang · 2018-06-04T00:56:20Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableImplTest.java

+        }
+    }
+
+    private TopologyDescription.Node getProcessor(final Topology topology, final String processorName) {


+1 from me.

guozhangwang · 2018-06-04T01:31:35Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableFilterTest.java

        KTableValueGetterSupplier<String, Integer> getterSupplier2 = table2.valueGetterSupplier();
        KTableValueGetterSupplier<String, Integer> getterSupplier3 = table3.valueGetterSupplier();

-        driver.setUp(builder, stateDir, Serdes.String(), Serdes.Integer());
+        try (final TopologyTestDriverWrapper driver = new TopologyTestDriverWrapper(builder.build(), props)) {


We should connect the state stores with this request processor instead of enforcing us to remember which processor to set while initializing. E.g. here we can do (note I removed the procNames):

private void doTestValueGetter(final StreamsBuilder builder, final KTableImpl<String, Integer, Integer> table2, final KTableImpl<String, Integer, Integer> table3, final String topic1) { final Topology topology = builder.build(); KTableValueGetterSupplier<String, Integer> getterSupplier2 = table2.valueGetterSupplier(); KTableValueGetterSupplier<String, Integer> getterSupplier3 = table3.valueGetterSupplier(); InternalTopologyBuilder topologyBuilder = TopologyWrapper.getInternalTopologyBuilder(topology); topologyBuilder.connectProcessorAndStateStores(table2.name, getterSupplier2.storeNames()); topologyBuilder.connectProcessorAndStateStores(table3.name, getterSupplier3.storeNames()); try (final TopologyTestDriverWrapper driver = new TopologyTestDriverWrapper(topology, props)) { KTableValueGetter<String, Integer> getter2 = getterSupplier2.get(); KTableValueGetter<String, Integer> getter3 = getterSupplier3.get(); getter2.init(driver.getProcessorContext(table2.name)); getter3.init(driver.getProcessorContext(table3.name)); // same below

Ditto elsewhere for initializing the getters.

This way is so much better. I wasn't happy with my implementation, but couldn't find a cleaner way to do it. Thank you!

guozhangwang · 2018-06-04T01:32:28Z

streams/src/test/java/org/apache/kafka/streams/TopologyTestDriverWrapper.java

+     * @param processorName used to search for a processor connected to this StateStore, which is set as current node
+     * @return the processor context
+     */
+    public ProcessorContext getProcessorContext(final String processorName) {


Better name as "setCurrentNodeInProcessorContext"? And then in java docs mention that it returns the processor context with current node set.

I agree. Changed the name and cleaned up the docs a bit.

guozhangwang · 2018-06-04T01:33:01Z

streams/src/test/java/org/apache/kafka/streams/TopologyTestDriverWrapper.java

+    /**
+     * Get the processor context
+     *
+     * @param processorName used to search for a processor connected to this StateStore, which is set as current node


I think this is because this function is only for init the getters, which requires the state stores is connected, hence accessible in init to the node.

guozhangwang · 2018-06-04T01:38:37Z

streams/src/test/java/org/apache/kafka/streams/TopologyTestDriverWrapper.java

+     * @param name the name to search for
+     * @return the processor matching the search name
+     */
+    public ProcessorNode getProcessor(final String name) {


Should we check globalTopology as well before give up and throw?

Yes, we should. Added the global topology check.

* Inline getProcessor in assertTopologyContainsProcessor in KTableImplTest * Rename KTableImplTest#testRepartition to shouldCreateSourceAndSinkNodesForRepartitioningTopic * Rename TopologyTestDriverWrapper#getProcessorContext to setCurrentNodeForProcessorContext * Search for processor in global topology before throwing * Change getters init in KTableFilterTest, KTableImplTest, KTableMapValuesTest, KTableSourceTest

mjsax

LGTM.

Thanks for the PR @h314to!

…2] (#4986) * KAFKA-6474: Rewrite tests to use new public TopologyTestDriver [part 2] * Refactor: -KTableFilterTest.java -KTableImplTest.java -KTableMapValuesTest.java -KTableSourceTest.java * Add access to task, processorTopology, and globalTopology in TopologyTestDriver via TopologyTestDriverWrapper * Remove unnecessary constructor in TopologyTestDriver * Change how TopologyTestDriverWrapper#getProcessorContext sets the current node Reviewers: John Roesler <john@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

guozhangwang · 2018-06-13T17:29:00Z

Merged to trunk, and cherry-picked to 2.0.

guozhangwang · 2018-06-13T17:30:03Z

We still have a bunch test classes that uses KStreamTestDriver. Since the 2.0 code freeze is today I think the final PR for completely getting rid of them will only go into trunk then.

@h314to Please ping us once you have the final PR to remove KStreamTestDriver ready.

…2] (apache#4986) * KAFKA-6474: Rewrite tests to use new public TopologyTestDriver [part 2] * Refactor: -KTableFilterTest.java -KTableImplTest.java -KTableMapValuesTest.java -KTableSourceTest.java * Add access to task, processorTopology, and globalTopology in TopologyTestDriver via TopologyTestDriverWrapper * Remove unnecessary constructor in TopologyTestDriver * Change how TopologyTestDriverWrapper#getProcessorContext sets the current node Reviewers: John Roesler <john@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

h314to commented May 9, 2018

View reviewed changes

mjsax added the streams label May 9, 2018

h314to force-pushed the fix/KAFKA-6474-part2 branch from a943901 to 4021dd9 Compare May 10, 2018 11:00

h314to force-pushed the fix/KAFKA-6474-part2 branch from 4021dd9 to a2ac5c4 Compare May 14, 2018 09:57

h314to commented May 14, 2018

View reviewed changes

h314to force-pushed the fix/KAFKA-6474-part2 branch from a2ac5c4 to 591ab05 Compare May 18, 2018 10:55

mjsax reviewed May 20, 2018

View reviewed changes

h314to commented May 21, 2018

View reviewed changes

h314to mentioned this pull request May 21, 2018

KAFKA-6474: Rewrite tests to use new public TopologyTestDriver [part 3] #5052

Merged

vvcephei approved these changes May 22, 2018

View reviewed changes

h314to force-pushed the fix/KAFKA-6474-part2 branch from e74b275 to acc26f9 Compare May 23, 2018 16:59

h314to added 2 commits May 25, 2018 03:06

Change how TopologyTestDriverWrapper#getProcessorContext sets the cur…

7525dc7

…rent node

h314to force-pushed the fix/KAFKA-6474-part2 branch from acc26f9 to 7525dc7 Compare May 25, 2018 02:19

mjsax reviewed May 29, 2018

View reviewed changes

guozhangwang reviewed Jun 4, 2018

View reviewed changes

mjsax approved these changes Jun 13, 2018

View reviewed changes

guozhangwang merged commit de4f4f5 into apache:trunk Jun 13, 2018

h314to deleted the fix/KAFKA-6474-part2 branch June 15, 2018 13:41

h314to mentioned this pull request Jul 29, 2018

KAFKA-6474: Rewrite tests to use new public TopologyTestDriver [part 4] #5433

Merged

KAFKA-6474: Rewrite tests to use new public TopologyTestDriver [part 2] #4986

KAFKA-6474: Rewrite tests to use new public TopologyTestDriver [part 2] #4986

Conversation

h314to commented May 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

h314to commented May 10, 2018

Choose a reason for hiding this comment

vvcephei commented May 15, 2018

mjsax left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

h314to left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjsax commented May 21, 2018

h314to commented May 22, 2018

vvcephei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

h314to commented May 23, 2018 • edited Loading

mjsax left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guozhangwang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjsax left a comment

Choose a reason for hiding this comment

guozhangwang commented Jun 13, 2018

guozhangwang commented Jun 13, 2018

h314to commented May 9, 2018 •

edited

Loading

h314to commented May 23, 2018 •

edited

Loading