type assertion failed on clientset #24656

mfanjie · 2016-04-22T08:55:54Z

I am working on federation scheduler, my code need list & watch clusters, replciasets federation apiserver, while my code can not work properly, the followings are detail description:

If I use List function to get all replicasets, from result you can see that on ReplicaSetList, the type meta is there {ReplicaSetList extensions/v1beta1} , but if you look at the detail of every element, you will see the typemeta property of the replicaset was lost.(I can get that info by curl, that mean the entity has the info on apiserver side.)
Code sample:

client12 := release_1_2.NewForConfigOrDie(kubeconfig)
r2, err := client12.ExtensionsClient.ReplicaSets(api.NamespaceAll).List(api.ListOptions{ResourceVersion: "0"})
    fmt.Println("result2: %v", r2)
    fmt.Println(err)

Result:

result2: %v &{{ReplicaSetList extensions/v1beta1} {/apis/extensions/v1beta1/replicasets 9} [{{ } {frontend  default /apis/extensions/v1beta1/namespaces/default/replicasets/frontend ec9b7324-079e-11e6-a2a4-acbc32c12e53 9 2 2016-04-21 16:56:23 +0800 CST <nil> <nil> map[app:guestbook tier:frontend] map[]} {0xc82053c390 0xc82053a6a0 {{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[tier:frontend app:guestbook] map[]} {[] [{php-redis gcr.io/google_samples/gb-frontend:v3 [] []  [{ 0 80 TCP }] [{GET_HOSTS_FROM dns <nil>}] {map[] map[cpu:{0.100000000 DecimalSI} memory:{104857600.000000000 BinarySI}]} [] <nil> <nil> <nil> /dev/termination-log IfNotPresent <nil> false false false}] Always 0xc82053c680 <nil> ClusterFirst map[]    false false false 0xc8200cd680 []}}} {0 0 0}}]}

Watch function has the same problem

Sample Code

WatchFunc: func(options api.ListOptions) (watch.Interface, error) {
            return factory.KubeClientSet.ExtensionsClient.ReplicaSets(api.NamespaceAll).Watch(options)
        }

which when it get updated data from apiserver, the following error is thrown.

E0422 16:52:37.233041   29457 reflector.go:378] k8s.io/kubernetes/federation/plugin/pkg/federated-scheduler/factory/factory.go:237: expected type *v1beta1.ReplicaSet, but watch event object had type *extensions.ReplicaSet

And if I dump the object being watched, I see the example info like List function.

This problem never happened when I get single object instead of list or watch, if I get object by name, I can get the object with correct typemeta.
I tried both internalclientset, release_1_2, and release_1_3, same problem happened. Based on current info I suspect there is some problem on decoder.

The text was updated successfully, but these errors were encountered:

mfanjie · 2016-04-22T09:32:47Z

I am debugging on codec to make sure if there is problem, so far the Codec UT with embedded body works. Meanwhile appreciate anyone can give me some suggest.

ghost · 2016-04-22T15:40:02Z

@nikhiljindal Can you provide any advice to @mfanjie here?

ncdc · 2016-04-22T15:58:19Z

@mfanjie can you provide more of your sample code that has WatchFunc: ...? e.g. the imports and the entire object containing WatchFunc?

mfanjie · 2016-04-22T16:18:27Z

I am still investigating on the issue, from my current findings, I highly suspect the issue happens here.
https://github.com/kubernetes/kubernetes/blob/master/pkg/client/clientset_generated/release_1_3/typed/extensions/v1beta1/extensions_client.go#L104
From here I see extensions is registered as groupVersion and this groupVersion is input parameter of codec, so I think all receiving data will be decoded as extension group.
But in Federation apiserver code, the SubReplicaSet is defined as v1beta1 type, to keep the same structure with sub rs, I need to get the same v1beta1/ReplicaSet.

// SubReplicaSet represents the configuration of a replica set scheduled to a Cluster.
type SubReplicaSet struct {
    unversioned.TypeMeta `json:",inline"`
    v1.ObjectMeta        `json:"metadata,omitempty"`

    // Spec defines the desired behavior of this SubReplicaSet.
    Spec extensionsv1.ReplicaSetSpec `json:"spec,omitempty"`

    // Status is the current status of this SubReplicaSet. This data may be
    // out of date by some window of time.
    Status extensionsv1.ReplicaSetStatus `json:"status,omitempty"`
}

So when I watch federation apiserver, and try to decode the response, it threw assertion failure.
We should either change register v1beta1 group version in extensions_client.go or we should change SubReplicaSet to unversioned.

This is current investigation updates, I did not get time to verify yet. @jianhuiz can you check this further?

@ncdc can you check my comment to see if I am on direction?
If you need the full code, here it is, please ignore the silly code for testing purpose. :)

https://github.com/k8s-incubator/kubernetes/blob/wip-federated-scheduler/federation/plugin/pkg/federated-scheduler/factory/factory.go#L324

ncdc · 2016-04-22T16:36:44Z

You're importing extensions "k8s.io/kubernetes/pkg/apis/extensions/v1beta1". You may need to just import k8s.io/kubernetes/pkg/apis/extensions instead.

mfanjie · 2016-04-22T16:41:57Z

@caesarxuchao please kindly advise

mfanjie · 2016-04-22T16:54:00Z

@ncdc as subreplicaset is defined with v1beta1 properties, and subreplicaset is created based on replica set properties. So I want to get replica set with the version of v1beta1.
Other wise it is hard to create subreplicaset.

ncdc · 2016-04-22T16:57:58Z

Understood, however, I was reacting to this error: reflector.go:378] k8s.io/kubernetes/federation/plugin/pkg/federated-scheduler/factory/factory.go:237: expected type *v1beta1.ReplicaSet, but watch event object had type *extensions.ReplicaSet which means it's returning extensions.ReplicaSet (internal), not v1beta1.ReplicaSet.

caesarxuchao · 2016-04-22T17:53:27Z

@mfanjie, I submitted a PR a couple of days ago to change the codec used by release_1_3:
https://github.com/kubernetes/kubernetes/blob/master/pkg/client/clientset_generated/release_1_3/typed/extensions/v1beta1/extensions_client.go#L118

Are you using this latest code? Does it work? If not, I think there's something wrong in the codec code, I can take a look later.

mfanjie · 2016-04-22T20:55:22Z

maybe my code need rebase then, I will figure out later. Thanks.

发自我的 iPhone

在 2016年4月23日，上午1:54，Chao Xu <notifications@github.com mailto:notifications@github.com> 写道：

@mfanjiehttps://github.com/mfanjie, I submitted a PR a couple of days ago to change the codec used by release_1_3:
https://github.com/kubernetes/kubernetes/blob/master/pkg/client/clientset_generated/release_1_3/typed/extensions/v1beta1/extensions_client.go#L118

Are you using this latest code? Does it work? If not, I think there's something wrong in the codec code, I can take a look later.

―
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com//issues/24656#issuecomment-213528982

jianhuiz · 2016-04-22T23:26:36Z

@caesarxuchao I met the same issue this morning when I rebased to the latest master and regenerated the clientset. I then reverted to the previous generated code, it works fine.
It is actually caused by using the serializer, it works when I changed it back to the LegacyCodec
#24395

mfanjie · 2016-04-23T03:20:41Z

@caesarxuchao by rebasing the code, the object can not be decoded anymore, here is the logs:

I0423 11:15:25.406593   41408 factory.go:340] watching subrs: &{0xc820177d70 0xc8200a5440 {0 0} false}
proto: no encoder for TypeMeta unversioned.TypeMeta [GetProperties]
proto: no encoder for TypeMeta unversioned.TypeMeta [GetProperties]
proto: tag has too few fields: "-"
proto: no coders for struct *reflect.rtype
proto: no encoder for sec int64 [GetProperties]
proto: no encoder for nsec int32 [GetProperties]
proto: no encoder for loc *time.Location [GetProperties]
proto: no encoder for Time time.Time [GetProperties]
proto: no coders for intstr.Type
proto: no encoder for Type intstr.Type [GetProperties]
-----------------list1 in createReplicaSetLW: %v TypeMeta:<kind:"PodList" apiVersion:"v1" > metadata:<selfLink:"/api/v1/pods" resourceVersion:"6" >
proto: no encoder for TypeMeta unversioned.TypeMeta [GetProperties]
proto: no encoder for TypeMeta unversioned.TypeMeta [GetProperties]
proto: no encoder for Amount *inf.Dec [GetProperties]
proto: no encoder for Format resource.Format [GetProperties]
proto: no encoder for unscaled big.Int [GetProperties]
proto: no encoder for scale inf.Scale [GetProperties]
-----------------list in createReplicaSetLW: %v %!v(PANIC=reflect.Value.Interface: cannot return value obtained from unexported field or method)
%!v(PANIC=reflect.Value.Interface: cannot return value obtained from unexported field or method) <nil>
-----------------

caesarxuchao · 2016-04-25T05:06:54Z

@mfanjie the problem seems to be caused by recent changes related to protobuf, but I thought protobuf is not k8s default serialization format yet, did you have any configuration that might trigger it?

Also, we have an e2e test that exercises release_1_3 clientset's watch and list function: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/generated_clientset.go#L183-L192. The test is passing. Could you compare your code with it? Perhaps that can help you debug your issue.

mfanjie · 2016-04-25T05:09:44Z

@caesarxuchao I manually merge all code of #24395 to my branch, things got better, I am still debugging on it, I did not update this issue as I did not confirm if it worked or not yet. Will update again soon.

mfanjie · 2016-04-25T09:36:38Z

@caesarxuchao I confirm after I merge all code change of #24395, it works properly, not a blocker of my work anymore, thank you for your prompt help. Closing this issue now.

mfanjie · 2016-04-25T15:42:37Z

The problem was gone when I use the old codebase and merge #24395 manually, but when I rebase the latest code from master branch, the error is till there. Reopen this issue for tracking, I am using the workaround to test now.

I0423 11:15:25.406593   41408 factory.go:340] watching subrs: &{0xc820177d70 0xc8200a5440 {0 0} false}
proto: no encoder for TypeMeta unversioned.TypeMeta [GetProperties]
proto: no encoder for TypeMeta unversioned.TypeMeta [GetProperties]
proto: tag has too few fields: "-"
proto: no coders for struct *reflect.rtype
proto: no encoder for sec int64 [GetProperties]
proto: no encoder for nsec int32 [GetProperties]
proto: no encoder for loc *time.Location [GetProperties]
proto: no encoder for Time time.Time [GetProperties]
proto: no coders for intstr.Type
proto: no encoder for Type intstr.Type [GetProperties]
-----------------list1 in createReplicaSetLW: %v TypeMeta:<kind:"PodList" apiVersion:"v1" > metadata:<selfLink:"/api/v1/pods" resourceVersion:"6" >
proto: no encoder for TypeMeta unversioned.TypeMeta [GetProperties]
proto: no encoder for TypeMeta unversioned.TypeMeta [GetProperties]
proto: no encoder for Amount *inf.Dec [GetProperties]
proto: no encoder for Format resource.Format [GetProperties]
proto: no encoder for unscaled big.Int [GetProperties]
proto: no encoder for scale inf.Scale [GetProperties]
-----------------list in createReplicaSetLW: %v %!v(PANIC=reflect.Value.Interface: cannot return value obtained from unexported field or method)
%!v(PANIC=reflect.Value.Interface: cannot return value obtained from unexported field or method) <nil>
-----------------

caesarxuchao · 2016-04-25T17:27:54Z

Have you run hack/update-all.sh after you add your subresource? Could you try to run it again after you rebase?

I suspect the problem you encountered is not directly related to clientset.

mfanjie · 2016-04-26T06:59:21Z

It's not a blocker anymore. The only left problem now is when I try to fmt.Println(rs), it will go panic. As long as I avoid printing the entire object, my code was working properly.

Attempting to schedule: %+v %!v(PANIC=reflect.Value.Interface: cannot return value obtained from unexported field or method)

mfanjie · 2016-04-26T09:22:26Z

Here is some test code, clusters can be dumped correctly, while replicasets can not be.

cs, _ := f.FederatedClientSet.FederationClient.Clusters().List(api.ListOptions{ResourceVersion: "0"})
    rss, _ := f.KubeClientSet.ExtensionsClient.ReplicaSets("default").List(api.ListOptions{ResourceVersion: "0"})
    fmt.Println("clusters %v", cs)
    fmt.Println("rss %v", rss)

Here is the stack trace:

#0  reflect.ValueOf (i=..., ~r1=...) at /usr/local/Cellar/go/1.5.2/libexec/src/reflect/value.go:2078
#1  0x00000000000860d5 in fmt.(*pp).catchPanic (p= []uint8, arg=..., verb=118) at /usr/local/Cellar/go/1.5.2/libexec/src/fmt/print.go:639
#2  0x000000000005c56e in runtime.call32 () at /usr/local/Cellar/go/1.5.2/libexec/src/runtime/asm_amd64.s:437
#3  0x000000000002c2d9 in runtime.gopanic (e=...) at /usr/local/Cellar/go/1.5.2/libexec/src/runtime/panic.go:423
#4  0x000000000013e767 in reflect.valueInterface (v=..., safe=true, ~r2=...) at /usr/local/Cellar/go/1.5.2/libexec/src/reflect/value.go:912
#5  0x000000000013e658 in reflect.Value.Interface (v=..., i=...) at /usr/local/Cellar/go/1.5.2/libexec/src/reflect/value.go:901
#6  0x00000000006ec89d in github.com/gogo/protobuf/proto.writeStruct (w=0xc8203ca2a0, sv=..., ~r2=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:361
#7  0x00000000006ed7b3 in github.com/gogo/protobuf/proto.writeAny (w=0xc8203ca2a0, v=..., props=0xc8204c3e40, ~r3=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:491
#8  0x00000000006ecb03 in github.com/gogo/protobuf/proto.writeStruct (w=0xc8203ca2a0, sv=..., ~r2=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:372
#9  0x00000000006ed7b3 in github.com/gogo/protobuf/proto.writeAny (w=0xc8203ca2a0, v=..., props=0xc8204b9340, ~r3=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:491
#10 0x00000000006ebfaa in github.com/gogo/protobuf/proto.writeStruct (w=0xc8203ca2a0, sv=..., ~r2=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:298
#11 0x00000000006ed7b3 in github.com/gogo/protobuf/proto.writeAny (w=0xc8203ca2a0, v=..., props=0xc8204b8b00, ~r3=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:491
#12 0x00000000006ecb03 in github.com/gogo/protobuf/proto.writeStruct (w=0xc8203ca2a0, sv=..., ~r2=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:372
#13 0x00000000006ed7b3 in github.com/gogo/protobuf/proto.writeAny (w=0xc8203ca2a0, v=..., props=0xc8204b6b00, ~r3=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:491
#14 0x00000000006eb8be in github.com/gogo/protobuf/proto.writeStruct (w=0xc8203ca2a0, sv=..., ~r2=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:239
---Type <return> to continue, or q <return> to quit---
#15 0x00000000006ed7b3 in github.com/gogo/protobuf/proto.writeAny (w=0xc8203ca2a0, v=..., props=0xc82049a9a0, ~r3=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:491
#16 0x00000000006ecb03 in github.com/gogo/protobuf/proto.writeStruct (w=0xc8203ca2a0, sv=..., ~r2=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:372
#17 0x00000000006ed7b3 in github.com/gogo/protobuf/proto.writeAny (w=0xc8203ca2a0, v=..., props=0xc82049a6e0, ~r3=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:491
#18 0x00000000006ecb03 in github.com/gogo/protobuf/proto.writeStruct (w=0xc8203ca2a0, sv=..., ~r2=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:372
#19 0x00000000006ed7b3 in github.com/gogo/protobuf/proto.writeAny (w=0xc8203ca2a0, v=..., props=0xc8204978c0, ~r3=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:491
#20 0x00000000006ecb03 in github.com/gogo/protobuf/proto.writeStruct (w=0xc8203ca2a0, sv=..., ~r2=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:372
#21 0x00000000006f07bd in github.com/gogo/protobuf/proto.marshalText (w=..., pb=..., compact=true, ~r3=...)
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:763
#22 0x00000000006f0ad9 in github.com/gogo/protobuf/proto.CompactTextString (pb=..., ~r1="")
    at /Users/fmeng/go-tools/src/github.com/gogo/protobuf/proto/text.go:791
#23 0x000000000072e0d8 in k8s.io/kubernetes/pkg/apis/extensions/v1beta1.(*ReplicaSet).String (m=0xc820221b00, ~r0="")
    at /Users/fmeng/code/kubernetes/src/k8s.io/kubernetes/pkg/apis/extensions/v1beta1/generated.pb.go:285
#24 0x0000000000086ee3 in fmt.(*pp).handleMethods (p=0xc8200671e0, verb=118, depth=0, handled=true)
    at /usr/local/Cellar/go/1.5.2/libexec/src/fmt/print.go:730
#25 0x0000000000087475 in fmt.(*pp).printArg (p=0xc8200671e0, arg=..., verb=118, depth=0, wasString=false)
    at /usr/local/Cellar/go/1.5.2/libexec/src/fmt/print.go:806
#26 0x000000000008eae8 in fmt.(*pp).doPrint (p=0xc8200671e0, a= []interface {} = {...}, addspace=true, addnewline=true)
    at /usr/local/Cellar/go/1.5.2/libexec/src/fmt/print.go:1254
---Type <return> to continue, or q <return> to quit---
#27 0x0000000000082497 in fmt.Fprintln (w=..., a= []interface {} = {...}, n=55360, err=...) at /usr/local/Cellar/go/1.5.2/libexec/src/fmt/print.go:254
#28 0x0000000000082593 in fmt.Println (a= []interface {} = {...}, n=14801760, err=...) at /usr/local/Cellar/go/1.5.2/libexec/src/fmt/print.go:264
#29 0x000000000033708f in k8s.io/kubernetes/federation/plugin/pkg/federated-scheduler.(*Scheduler).scheduleOne (s=0xc820364000)
    at /Users/fmeng/code/kubernetes/src/k8s.io/kubernetes/federation/plugin/pkg/federated-scheduler/scheduler.go:99
#30 0x00000000003368a1 in k8s.io/kubernetes/federation/plugin/pkg/federated-scheduler.(*Scheduler).Run (s=0xc820364000)
    at /Users/fmeng/code/kubernetes/src/k8s.io/kubernetes/federation/plugin/pkg/federated-scheduler/scheduler.go:86
#31 0x0000000000064620 in k8s.io/kubernetes/federation/plugin/cmd/federated-scheduler/app.Run.func2 ()
    at /Users/fmeng/code/kubernetes/src/k8s.io/kubernetes/federation/plugin/cmd/federated-scheduler/app/server.go:123
#32 0x00000000000638d8 in k8s.io/kubernetes/federation/plugin/cmd/federated-scheduler/app.Run (s=0xc820165520, ~r1=...)
    at /Users/fmeng/code/kubernetes/src/k8s.io/kubernetes/federation/plugin/cmd/federated-scheduler/app/server.go:126
#33 0x0000000000002104 in main.main () at /Users/fmeng/code/kubernetes/src/k8s.io/kubernetes/federation/plugin/cmd/federated-scheduler/scheduler.go:47

caesarxuchao · 2016-04-26T18:32:21Z

@mfanjie may I ask how did you solve the proto problem?

mfanjie · 2016-04-27T01:56:06Z

@caesarxuchao the previous version does not have protobuf properties, so i think the update-all.sh was not invoked, I get the latest from from @jianhuiz and the type assertion problem was gone.
and currently the only remaining issue is panic when printing replicaset, printing cluster does not have any issue.

lavalamp · 2016-06-07T21:57:49Z

The panic appears to be due to a private field, which shouldn't exist. It looks like the hard problems are taken care of? I'm lowering priority.

caesarxuchao · 2016-06-07T22:08:47Z

@mfanjie the type assertion problem is fixed. Is there any remaining problem?

mfanjie · 2016-06-08T02:42:29Z

@caesarxuchao no any problem now, thanks, close the issue.

…herry-pick-24650-to-release-4.4 [release-4.4] Bug 1811198: /readyz should start returning failure on shutdown initiation Origin-commit: cd62f4529c30cb648bfcc5aea024620a0d72122e

mfanjie mentioned this issue Apr 22, 2016

Cluster Federation: task list #23653

Closed

20 tasks

ghost added area/cluster-federation priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Apr 22, 2016

ghost added this to the v1.3 milestone Apr 22, 2016

ghost assigned nikhiljindal Apr 22, 2016

mfanjie closed this as completed Apr 25, 2016

mfanjie reopened this Apr 25, 2016

davidopp added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label May 27, 2016

davidopp removed the team/control-plane label May 27, 2016

lavalamp removed this from the v1.3 milestone Jun 7, 2016

lavalamp added priority/backlog Higher priority than priority/awaiting-more-evidence. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Jun 7, 2016

mfanjie closed this as completed Jun 8, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

type assertion failed on clientset #24656

type assertion failed on clientset #24656

mfanjie commented Apr 22, 2016 •

edited

mfanjie commented Apr 22, 2016

ghost commented Apr 22, 2016

ncdc commented Apr 22, 2016

mfanjie commented Apr 22, 2016

ncdc commented Apr 22, 2016

mfanjie commented Apr 22, 2016

mfanjie commented Apr 22, 2016

ncdc commented Apr 22, 2016

caesarxuchao commented Apr 22, 2016

mfanjie commented Apr 22, 2016

jianhuiz commented Apr 22, 2016 •

edited

mfanjie commented Apr 23, 2016 •

edited

caesarxuchao commented Apr 25, 2016

mfanjie commented Apr 25, 2016

mfanjie commented Apr 25, 2016

mfanjie commented Apr 25, 2016

caesarxuchao commented Apr 25, 2016 •

edited

mfanjie commented Apr 26, 2016 •

edited

mfanjie commented Apr 26, 2016

caesarxuchao commented Apr 26, 2016

mfanjie commented Apr 27, 2016

lavalamp commented Jun 7, 2016

caesarxuchao commented Jun 7, 2016

mfanjie commented Jun 8, 2016

type assertion failed on clientset #24656

type assertion failed on clientset #24656

Comments

mfanjie commented Apr 22, 2016 • edited

mfanjie commented Apr 22, 2016

ghost commented Apr 22, 2016

ncdc commented Apr 22, 2016

mfanjie commented Apr 22, 2016

ncdc commented Apr 22, 2016

mfanjie commented Apr 22, 2016

mfanjie commented Apr 22, 2016

ncdc commented Apr 22, 2016

caesarxuchao commented Apr 22, 2016

mfanjie commented Apr 22, 2016

jianhuiz commented Apr 22, 2016 • edited

mfanjie commented Apr 23, 2016 • edited

caesarxuchao commented Apr 25, 2016

mfanjie commented Apr 25, 2016

mfanjie commented Apr 25, 2016

mfanjie commented Apr 25, 2016

caesarxuchao commented Apr 25, 2016 • edited

mfanjie commented Apr 26, 2016 • edited

mfanjie commented Apr 26, 2016

caesarxuchao commented Apr 26, 2016

mfanjie commented Apr 27, 2016

lavalamp commented Jun 7, 2016

caesarxuchao commented Jun 7, 2016

mfanjie commented Jun 8, 2016

mfanjie commented Apr 22, 2016 •

edited

jianhuiz commented Apr 22, 2016 •

edited

mfanjie commented Apr 23, 2016 •

edited

caesarxuchao commented Apr 25, 2016 •

edited

mfanjie commented Apr 26, 2016 •

edited