KAFKA-4390: Replace MessageSet usage with client-side alternatives #2140

hachikuji · 2016-11-16T04:24:03Z

No description provided.

hachikuji · 2016-11-16T04:36:09Z

ping @junrao @guozhangwang @ijuma @apurvam

This patch modifies the server implementation to use the client-side Record objects for all internal processing. As you can see, this was a hefty bit of work, but fortunately most of the transformations are straightforward. The main thing to focus on is the implementation of LogValidator, which contains the offset assignment and record validation logic that was previously contained in ByteBufferMessageSet. I've been pretty careful to preserve the optimizations that were present previously (e.g. in-place assignment where possible), but don't take my word for it.

One quick note on naming. I've renamed the Records object and subclasses to LogBuffer. So MemoryRecords is now MemoryLogBuffer. The reason for this change was that it felt unintuitive for an instance of Records to be an Iterable<LogEntry>, with the LogEntry instances being the actual container for the records. A LogBuffer instead represents a range of the log and provides access to the log entries contained in it. That seemed more intuitive to me, but let me know if you agree or if you have other suggestions.

onurkaraman · 2016-11-16T05:02:37Z

Slightly related, slightly tangential: is there a specific reason why we put the new broker-specific java classes under clients/ ?

I'm talking about stuff like:
FileRecords
LeaderAndIsrRequest / LeaderAndIsrResponse
StopReplicaRequest / StopReplicaResponse
UpdateMetadataRequest / UpdateMetadataResponse

hachikuji · 2016-11-16T05:15:56Z

@onurkaraman Yeah, I've wondered a bit about that also. I'd be OK moving FileRecords to the server if people prefer that. I was thinking one potential benefit is that it opens the door to adding persistence to the client, which some users have requested (we have an ancient JIRA for this, but the use case might not be too compelling). In the end, I decided it wasn't that much code, so having it in clients didn't hurt too much and it kept all record-related stuff close together, which may make it easier to share common pieces.

ijuma · 2016-11-16T15:43:02Z

@hachikuji, thanks for tackling this.

About the naming question, I also found it a bit confusing how we sometimes have an offset and sometimes don't when talking about records. That is, MemoryRecords includes the offset (and record size in the underlying buffer) for each record while Record does not. It all becomes clearer when one realises that MemoryRecords (renamed to MemoryLogBuffer in the PR) actually contains LogEntry instances, each being a pair of offset and Record.

One thing to think about is whether this fits with the other Record classes we have and whether that matters (maybe it doesn't). For example, ConsumerRecord contains the offset while ProducerRecord does not. Also, it would have been a bit easier to review if the rename had been done in a separate PR, but probably too late for that. :)

About having the classes in clients, I think that's OK as they are in an internal common package.

ijuma

Just noticed a couple of things when I scanned the PR. Will do a proper review later.

ijuma · 2016-11-16T15:45:33Z

clients/src/main/java/org/apache/kafka/common/protocol/types/Type.java

@@ -427,49 +427,49 @@ public ByteBuffer validate(Object item) {
        }
    };

-    public static final Type RECORDS = new Type() {
+    public static final Type LOGBUFFER = new Type() {


Is it intentional that this is LOGBUFFER instead of LOG_BUFFER? Same for the toString implementation.

ijuma · 2016-11-16T15:48:36Z

clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java

@@ -123,7 +123,7 @@ public int timeout() {
        return timeout;
    }

-    public Map<TopicPartition, MemoryRecords> partitionRecords() {
+    public Map<TopicPartition, MemoryLogBuffer> partitionRecords() {


Should this method be renamed as well?

Ack. There are probably a few of these. I'll do another pass and try to find others.

hachikuji · 2016-11-16T21:11:59Z

@becketqin Would be nice to get your feedback also. Put on the coffee and lose yourself in code review!

becketqin · 2016-11-19T05:01:19Z

Wow, a 5000 line change... I'll take a look this weekend...

becketqin · 2016-11-21T07:13:06Z

@hachikuji I ran into some other stuff today and didn't finish reading the entire patch. Just some thoughts when I was reading the code. I think "Message" (and "Record" in the new clients) are a well established concept for Kafka. It is indeed a little weird that Records is an Iterable<LogEntry>, but I felt changing all the Reocrds to LogBuffer seems introducing a new concept (BTW FileLogBuffer sounds a little weird given it actually does not have a buffer). I would like to see if we can avoid solving the confusion by adding a new concept.

Not sure if there was any thinking on changing LogEntry to something like LogRecord which indicate it's something resides in the log? Then Records would contain a few LogRecord which contains (Offset + Record). We also have a pretty symmetric naming for ProducerRecord, ConsumerRecord and LogRecord clearly indicating where they are used. We can also consider renaming Records to LogRecords to make it clear. I am not sure if Records counts as a public interface or not, though. I know we have a page stating which packages are public and which are not, but I doubt if people follows that given we have an explicit internals package...

I'll save my other comments until I go through a full pass of the code in case some of them are not valid at all (I already found some...)

ijuma · 2016-11-21T09:32:06Z

@becketqin, that's a fair point about the rename and introducing a new concept. I have similar concerns and was wondering how we could make things clearer without that. Your suggestion looks promising.

onurkaraman · 2016-11-21T18:48:03Z

Does it make sense to separate the renaming from the actual task of this patch?

hachikuji · 2016-11-21T19:01:08Z

@becketqin Thanks for taking a look. I'm not sure I follow why you consider the renaming a conceptual change. The object works the same as before, but I felt the name fit closer to what the object actually represents, which is a range of bytes from the log. The name Records to me just suggests a container for Record objects.

The suggestion about LogRecord makes sense to me. I have actually done something similar in work building off of this patch. At the same time, I would like to preserve a concept of LogEntry as a container for records which sits between LogBuffer (or Records) and LogRecord (or Record). The basic idea is to treat the shallow messages as log entries, and the deep messages as log records (an uncompressed message is treated as a log entry containing just a single log record).

To give a bit more background, we're trying to generalize the concept of a message set so that 1) it uses a separate schema from individual messages, and 2) it's extended to uncompressed data. This allows us to amortize the cost of additional metadata which is invariant for the messages contained in a message set. I'm happy to provide some additional detail if you're interested (there will be a KIP on the way some time in the next few weeks).

@onurkaraman Yeah, we can do that if it makes this patch easier to get in. Let's see what others think. Sigh, any suggestion to reduce lines of code is likely to be popular to all except me.

hachikuji · 2016-11-22T17:25:40Z

I ran system tests on the latest patch and everything looks good: http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/2016-11-22--001.1479824556--hachikuji--KAFKA4390--24dc7ed/report.html. I will probably continue to add some additional test cases, but I'll leave the rest as is pending further review comments.

junrao · 2016-11-23T02:21:40Z

@hachikuji : Will also take a look at the patch. Just a quick note. Could you do some performance test to make sure there is no regression?

hachikuji · 2016-11-23T02:30:55Z

@junrao Thanks for taking a look. Performance testing is next on my plate after I fill in some of the gaps in test coverage.

hachikuji · 2016-11-29T19:22:00Z

Update: I've begun performance testing. I'm seeing a substantial performance degradation on the consumer side. I'll update this PR when I know more.

hachikuji · 2016-11-29T20:24:27Z

I found the cause of the performance regression. When handling a fetch, we must read through the log to find the starting position of a given offset (starting from the position given by the index). To do so, we only need to read the offset and size, but one of my recent commits accidentally changed this behavior to unnecessarily read the full record. I've fixed this in the last commit and now it looks like performance is within 5% of trunk for the producer and consumer. Perhaps still on the slower side though, so I'll continue investigating.

junrao

@hachikuji : Thanks for the patch. Made a pass on this. Some of the issues that I pointed out seem to be existing. We can decide whether to address them here or in followup jiras.

junrao · 2016-11-24T02:13:36Z

clients/src/main/java/org/apache/kafka/common/record/ByteBufferLogInputStream.java

+
+        public void setCreateTime(long timestamp) {
+            Record record = record();
+            if (record.magic() > 0) {


Should we assert record.magic() > 0?

junrao · 2016-11-24T02:14:27Z

clients/src/main/java/org/apache/kafka/common/record/FileLogBuffer.java

+            throw new KafkaException(String.format("Size of FileRecords %s has been truncated during write: old size %d, new size %d", file.getAbsolutePath(), size, newSize));
+
+        long position = start + offset;
+        long count = Math.min(length, this.size.get());


To be consistent, perhaps this.size and this.channel should just be size and channel?

junrao · 2016-11-24T02:14:45Z

clients/src/main/java/org/apache/kafka/common/record/MemoryLogBuffer.java

+    @Override
+    public long writeTo(GatheringByteChannel channel, long position, int length) throws IOException {
+        ByteBuffer dup = buffer.duplicate();
+        dup.position(new Long(position).intValue());


It seems that some of the changes are lost during rebase? For example, there was code in MemoryRecords for setting the buffer limit according to length, and cast position to int instead of creating a Long object.

junrao · 2016-11-24T02:14:55Z

clients/src/main/java/org/apache/kafka/common/record/MemoryLogBuffer.java

+
+            for (LogEntry deepEntry : shallowEntry) {
+                Record deepRecord = deepEntry.record();
+                messagesRead += 1;


This seems to be an existing issue. For uncompressed messages, do we double count messagesRead since we already increased the count in line 98?

Ack. I'll fix this and the one below and update the test cases.

junrao · 2016-11-24T02:15:05Z

clients/src/main/java/org/apache/kafka/common/record/MemoryLogBuffer.java

+            if (writeOriginalEntry) {
+                // There are no messages compacted out and no message format conversion, write the original message set back
+                shallowEntry.writeTo(buffer);
+                messagesRetained += 1;


This and line 144 don't seem to be correct. It seems that we should add the number of entries in retainedEntries?

junrao · 2016-11-30T01:53:50Z

core/src/main/scala/kafka/server/DelayedFetch.scala

@@ -152,7 +152,7 @@ class DelayedFetch(delayMs: Long,
    )

    val fetchPartitionData = logReadResults.map { case (tp, result) =>
-      tp -> FetchResponsePartitionData(result.errorCode, result.hw, result.info.messageSet)
+      tp -> FetchPartitionData(result.errorCode, result.hw, result.info.logBuffer)


unused import kafka.api.FetchResponsePartitionData

junrao · 2016-11-30T01:53:58Z

core/src/test/scala/unit/kafka/log/LogValidatorTest.scala

+      messageTimestampType = TimestampType.LOG_APPEND_TIME,
+      messageTimestampDiffMaxMs = 1000L)
+    val validatedLogBuffer = validatedResults.validatedEntries
+    assertEquals("message set size should not change", logBuffer.deepEntries.asScala.size, validatedLogBuffer.deepEntries.asScala.size)


"message set size" is bit ambiguous. Perhaps we should say "number of messages"?

junrao · 2016-11-30T01:54:08Z

core/src/test/scala/unit/kafka/log/LogValidatorTest.scala

+    }
+  }
+
+  /* check that offsets are assigned based on byte offset from the given base offset */


Not sure what's "byte offset".

junrao · 2016-11-30T01:54:15Z

core/src/test/scala/unit/kafka/message/MessageCompressionTest.scala

@@ -70,7 +70,7 @@ class MessageCompressionTest extends JUnitSuite {
    testCompressSize(GZIPCompressionCodec, messages, 396)

    if(isSnappyAvailable)
-      testCompressSize(SnappyCompressionCodec, messages, 502)
+      testCompressSize(SnappyCompressionCodec, messages, 1063)


Hmm, why do we have to change the expected size?

It puzzled me for a while when writing this code why the size was coming out different only for snappy, but it turns out that we've overridden the block size in the client code, instead of using the default as was done for the server code.

Good catch. We probably don't want the change the buffer size in the server to be the same as the client. We may consider changing the client to be the same as the server. See KAFKA-3704 for details.

junrao · 2016-11-30T01:54:24Z

core/src/test/scala/unit/kafka/message/MessageTest.scala

-        convertAndVerify(v, Message.MagicValue_V0, Message.MagicValue_V1)
-      } else if (v.magicValue == Message.MagicValue_V1) {
-        convertAndVerify(v, Message.MagicValue_V1, Message.MagicValue_V0)
+      if (v.codec == NoCompressionCodec) {


Is there a reason that we only test non-compressed message conversion now?

Hmm.. I think the test was broken or at least incomplete since Message.toFormatVersion only did shallow conversion. When I implemented this in the client code, I forbid shallow-only conversion because it results in bugs like we found in LogCleaner. We'll probably end up dropping this code after we remove Message.toFormatVersion as suggested above.

hachikuji · 2016-12-01T05:24:09Z

@junrao I really appreciate the thorough review. I've addressed the easier items and left a few replies. I'll get to the rest tomorrow.

By the way, in the near future, I'd like to squash commits to make rebasing a bit easier. It hasn't been too much of a problem yet, but it will get harder with more iterations.

guozhangwang · 2016-12-05T05:17:02Z

About the naming of Records to LogBuffer, I share the same concern with @becketqin and @ijuma . My proposal would be to rename LogEntry to RecordEntry or simply RecordAndOffset (seems more Scala-ish), and to me it is OK to have Records.iterator() return Iterator<RecordEntry>. As for ConsumerRecord and ProducerRecord, it would be best if them both contain a Record field, and then ConsumerRecord to contain a separate offset field, but since it is public APIs we have to leave it as is, which is not too bad to me.

Also, I'd like to suggest we separate the renaming out of this PR for the ease of reviewing, if it is still possible to revert it back.

hachikuji · 2016-12-05T17:46:28Z

@guozhangwang RecordEntry would work for me. Keep in mind that if KIP-98 is approved, it will do more than just track the offset, so I'd rather not use something specific like RecordAndOffset. Since my LogBuffer suggestion is not too popular, I'll go ahead and revert that change. Then the hierarchy will be Records -> RecordEntry -> Record. Does that sound reasonable?

hachikuji · 2016-12-05T21:32:29Z

I've gone ahead and squashed commits. You can still find the old commit history here: https://github.com/hachikuji/kafka/tree/KAFKA-4390-UNSQUASHED.

asfbot · 2016-12-12T18:55:52Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/81/
Test PASSed (JDK 8 and Scala 2.11).

…in filterTo

hachikuji · 2016-12-12T20:10:32Z

@junrao Latest round of comments addressed. Please take a look.

@ijuma On the question of Iterator vs Iterable, I'm pretty open. I used the former for consistency with the old code, but I agree it would be nice to be able to use the "foreach" syntax.

asfbot · 2016-12-12T20:45:59Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/86/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2016-12-12T21:54:53Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/85/
Test FAILed (JDK 7 and Scala 2.10).

asfbot · 2016-12-12T23:07:42Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/87/
Test FAILed (JDK 8 and Scala 2.11).

junrao

@hachikuji : Thanks for the latest patch. Just a few minor comments. Also, could you post the latest performance results? Assuming there is no degradation, the patch LGTM.

junrao · 2016-12-13T00:21:41Z

clients/src/main/java/org/apache/kafka/common/record/Records.java

- * A binary format which consists of a 4 byte size, an 8 byte offset, and the record bytes. See {@link MemoryRecords}
- * for the in-memory representation.
+ * Interface for accessing the records contained in a log. The log itself is represented as a sequence of log entries.
+ * Each log entry consists of a 4 byte size, an 8 byte offset, and a "shallow" {@link Record record}. If


a 4 byte size, an 8 byte offset => an 8 byte offset, a 4 byte size of the record

junrao · 2016-12-13T00:24:22Z

clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java

    }

    /**
-     * Close this batch for no more appends
+     * Filter this log buffer into the provided ByteBuffer.


Is the reference to "log buffer" still valid?

junrao · 2016-12-13T00:24:27Z

clients/src/main/java/org/apache/kafka/common/record/AbstractRecords.java

+    }
+
+    /**
+     * Get the records from this log buffer (note this requires "deep" iteration into the


Is the reference to "log buffer " still valid?

asfbot · 2016-12-13T01:24:53Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/92/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2016-12-13T01:25:19Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/91/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2016-12-13T01:32:06Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/90/
Test PASSed (JDK 7 and Scala 2.10).

hachikuji · 2016-12-13T09:06:53Z

@junrao Thanks for the reviews. I did some testing this evening. I thought I was seeing some performance difference initially in the producer, but it seems within the variance of the test runs. If I were guessing from the results, I'd say the non-compressed path is a tad slower while the compressed path might be a tad faster, but don't put much weight behind either conclusion. In any case, the results seem close enough that I'd recommend merging now. Note that I did add one commit to address a couple minor cleanups and tighten up the iteration code a little.

asfbot · 2016-12-13T10:30:14Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/96/
Test FAILed (JDK 7 and Scala 2.10).

asfbot · 2016-12-13T11:56:13Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/97/
Test FAILed (JDK 8 and Scala 2.12).

asfbot · 2016-12-13T11:56:34Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/98/
Test FAILed (JDK 8 and Scala 2.11).

ijuma · 2016-12-13T15:42:32Z

clients/src/main/java/org/apache/kafka/common/record/ByteBufferInputStream.java

-    public int read() {
-        if (!buffer.hasRemaining()) {
-            return -1;
+    private static class UnderlyingInputStream extends InputStream {


It's a bit annoying that we create so much indirection (DataLogInputStream -> ByteBufferInputStream -> UnderlyingInputStream -> ByteBuffer -> byte[]). In an ideal world, we would not bother with InputStream at all and would just operate at the ByteBuffer level. However, the gzip case is hard to do that way.

@ijuma Haha, yeah. One of the layers is sort of fake (DataInputStream should be a mixin), but the point is still valid.

junrao

@hachikuji : Thanks for the patch. LGTM. Just a couple of minor comments.

About the performance. We used to optimize the path for recompression on the broker side by implementing a chained ByteBuffer list to avoid copying during buffer overflow for writes. With the patch, we lose such optimization and simply recopies data to a bigger buffer during overflow. This will affect the performance of recompression when the estimated after-compression size is lower. Recompression can happen when (1) producer is old, and (2) the broker compression codec is different from the producer's, both should be uncommon. So, we can probably commit the patch as it is. If the recompression performance is a problem, we can always optimize the code for expanding the buffer in ByteBufferOutputStream later.

junrao · 2016-12-13T17:09:21Z

clients/src/main/java/org/apache/kafka/common/record/ByteBufferLogInputStream.java

+            this.buffer = buffer;
+            buffer.position(LOG_OVERHEAD);
+            this.record = new Record(buffer.slice());
+            buffer.position(OFFSET_OFFSET);


Do we need to change the position of buffer? Perhaps we could instead just change the position in the slice passed to Record().

Currently Record expects the position of the ByteBuffer to be at 0. I was tempted to change this assumption, but decided to leave it for now (it's a bit annoying to change all the accessors to assume relative positioning). We could accomplish the same result using mark() and reset() if that seems any better.

junrao · 2016-12-13T17:10:13Z

clients/src/main/java/org/apache/kafka/common/record/Records.java

- * A binary format which consists of a 4 byte size, an 8 byte offset, and the record bytes. See {@link MemoryRecords}
- * for the in-memory representation.
+ * Interface for accessing the records contained in a log. The log itself is represented as a sequence of log entries.
+ * Each log entry consists of a 4 byte size, an 8 byte offset, a 4 byte record size, and a "shallow" {@link Record record}.


"a 4 byte size," needs to be removed.

asfbot · 2016-12-13T18:41:04Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/106/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2016-12-13T18:41:52Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/104/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2016-12-13T19:28:51Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/105/
Test PASSed (JDK 8 and Scala 2.12).

Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>, Jun Rao <junrao@gmail.com> Closes apache#2140 from hachikuji/KAFKA4390

dengziming · 2021-01-25T05:30:44Z

clients/src/main/java/org/apache/kafka/common/record/FileRecords.java

        long position = start + offset;
-        long count = Math.min(length, this.size - offset);
+        long count = Math.min(length, size.get());


Should this be Math.min(length, size.get() - offset)?

@dengziming : Thanks for the comment. This does seem like a bug. Would you be interested in submitting a separate PR to have this fixed?

ijuma reviewed Nov 16, 2016

View reviewed changes

hachikuji force-pushed the KAFKA4390 branch from e4de9ab to 83552f9 Compare November 16, 2016 19:58

hachikuji force-pushed the KAFKA4390 branch from 99419d3 to de01949 Compare November 18, 2016 17:32

hachikuji force-pushed the KAFKA4390 branch 2 times, most recently from f002ac6 to b646ef5 Compare November 28, 2016 04:49

hachikuji force-pushed the KAFKA4390 branch from b646ef5 to 85b3834 Compare November 29, 2016 20:17

junrao reviewed Nov 30, 2016

View reviewed changes

hachikuji force-pushed the KAFKA4390 branch from 85b3834 to 8c82bae Compare December 1, 2016 03:09

hachikuji force-pushed the KAFKA4390 branch 2 times, most recently from 42b51b2 to fb79917 Compare December 5, 2016 21:31

hachikuji force-pushed the KAFKA4390 branch from e71a4d0 to be1d08c Compare December 6, 2016 01:07

hachikuji force-pushed the KAFKA4390 branch from 788f1bd to 4e8cf94 Compare December 12, 2016 18:15

Jason Gustafson added 3 commits December 12, 2016 11:26

Forgotten renamings, cleanup, and buggy offset/timestamp calculation …

a382de5

…in filterTo

Missed one renaming

ef03d29

FileRecords positions should be integers

ffff0bf

Cleanup and expand Records javadoc

a817a76

junrao reviewed Dec 13, 2016

View reviewed changes

Fix some stale references to log buffer

ae0866b

Minor cleanup and tightening

a96750f

ijuma reviewed Dec 13, 2016

View reviewed changes

junrao reviewed Dec 13, 2016

View reviewed changes

Fix Records comment on log entry fields

620ff2c

asfgit closed this in 67f1e5b Dec 13, 2016

dengziming reviewed Jan 25, 2021

View reviewed changes

dengziming mentioned this pull request Jan 26, 2021

KAFKA-12233: Align the length passed to FileChannel by FileRecords.writeTo #9970

Merged

3 tasks

KAFKA-4390: Replace MessageSet usage with client-side alternatives #2140

KAFKA-4390: Replace MessageSet usage with client-side alternatives #2140

Conversation

hachikuji commented Nov 16, 2016

hachikuji commented Nov 16, 2016

onurkaraman commented Nov 16, 2016

hachikuji commented Nov 16, 2016 • edited Loading

ijuma commented Nov 16, 2016 • edited Loading

ijuma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hachikuji commented Nov 16, 2016

becketqin commented Nov 19, 2016

becketqin commented Nov 21, 2016

ijuma commented Nov 21, 2016

onurkaraman commented Nov 21, 2016

hachikuji commented Nov 21, 2016

hachikuji commented Nov 22, 2016

junrao commented Nov 23, 2016

hachikuji commented Nov 23, 2016 • edited Loading

hachikuji commented Nov 29, 2016

hachikuji commented Nov 29, 2016

junrao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hachikuji Dec 1, 2016 • edited Loading

Choose a reason for hiding this comment

hachikuji commented Dec 1, 2016

guozhangwang commented Dec 5, 2016 • edited Loading

hachikuji commented Dec 5, 2016

hachikuji commented Dec 5, 2016

asfbot commented Dec 12, 2016

hachikuji commented Dec 12, 2016 • edited Loading

asfbot commented Dec 12, 2016

asfbot commented Dec 12, 2016

asfbot commented Dec 12, 2016

junrao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asfbot commented Dec 13, 2016

asfbot commented Dec 13, 2016

asfbot commented Dec 13, 2016

hachikuji commented Dec 13, 2016

asfbot commented Dec 13, 2016

asfbot commented Dec 13, 2016

asfbot commented Dec 13, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junrao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asfbot commented Dec 13, 2016

asfbot commented Dec 13, 2016

asfbot commented Dec 13, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hachikuji commented Nov 16, 2016 •

edited

Loading

ijuma commented Nov 16, 2016 •

edited

Loading

hachikuji commented Nov 23, 2016 •

edited

Loading

hachikuji Dec 1, 2016 •

edited

Loading

guozhangwang commented Dec 5, 2016 •

edited

Loading

hachikuji commented Dec 12, 2016 •

edited

Loading