Cannot stop python client with Ctrl-C #3820

yokoi-h · 2015-10-14T07:35:50Z

I'm using grpc stream between a server process implemented by Golang and a client process in Python.
The client can receive values from the server with the stream. However when it comes to disconnecting from the client, it doesn't work well.

I made sample programs using the stream like below.
https://gist.github.com/yokoi-h/06ab4d802fa918884d38

Components in the sample program are as follows.

client1(add_count.py): sends count to server
server(sample_server.go): receives count and adds it to a variable inside
client2(monitor_count.py): receives total count from the server
- client2(monitor_count.py) <----- server(sample_server.go) <---- client1(add_count.py)
- these components are running in a different terminal.

If client1 sends count to server, then client2 can receive total count via the server.
But when hit Ctrl-C to quit client2 in the terminal, it's blocked and not possible to quit except using kill.

I would like to know how to finish a client that is reading stream with Ctrl-C.
If you can provide the example to stop with Ctrl-C, it would be great.

Thank you

nathanielmanistaatgoogle · 2015-10-14T13:35:04Z

Thanks for the report. I see you're calling early_adopter_ methods in your code - what happens if you change those to beta_? You may need to update to gRPC 0.11 or later and re-run protoc.

yokoi-h · 2015-10-16T05:52:49Z

Hi,
Thank you for your reply. I should have look at the latest tutorial.

I cloned gRPC git repository and checked out release-0_11_1 tag, and then compiled and installed in an Ubuntu 14.04 VM.
I tried modified script that uses beta_, but the same problem occurred. I was not able to quit the script with Ctrl-C.

Is there any solution or measure to analyze for this?

Thank you

kyleconroy · 2015-10-17T22:26:13Z

I'm also seeing this problem with my Python client. Here's my code https://gist.github.com/kyleconroy/9c2ba448d46f2ec93900

shuowusc · 2015-10-26T23:17:25Z

We are also seeing this problem (Python2.7, beta API) for both client and server. Client side is easier to trigger. It seems to have something to do with internal broken connection objects that lingered around and causing destructor to block wait forever.

hsaliak · 2016-04-21T22:53:51Z

@nathanielmanistaatgoogle what's the status of this issue?

nathanielmanistaatgoogle · 2016-04-23T21:55:48Z

@hsaliak I expect it to be resolved this quarter; likely before mid-May.

GuoJing · 2016-05-31T10:10:27Z

still not fixed in 0.14.0.

jaytaylor · 2016-06-15T23:47:26Z

+1

Definitely still not fixed in HEAD.

nathanielmanistaatgoogle · 2016-06-16T15:02:14Z

@jaytaylor: while working on this we've been finding several causes of it rather than just one. Please go into more detail - platform, Python version, gRPC commit point, and if possible code to reproduce?

GuoJing · 2016-06-16T16:21:51Z

hi

I think this is easy to find the code to reproduce, I wrote the code here

#5640

GuoJing

http://guojing.me http://guojing.me/
The one who love coding.
--> ret <--
echo null
rm -rf /

在 2016年6月16日，下午11:02，Nathaniel Manista notifications@github.com 写道：

@jaytaylor https://github.com/jaytaylor: while working on this we've been finding several causes of it rather than just one. Please go into more detail - platform, Python version, gRPC commit point, and if possible code to reproduce?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub #3820 (comment), or mute the thread https://github.com/notifications/unsubscribe/AAWKQWtwyo_JuZY4QRCnfEWMp89dnBXVks5qMWV4gaJpZM4GOc70.

tseaver · 2016-06-28T20:23:28Z

@kpayson64 Thanks! I can confirm that the indefinite hang-on-exit I've seen with gcloud-python + GAX + gRPC is now gone, leaving in its place a much more benign bunch of logging spew. :)

D0628 16:18:07.661735081   10465 iomgr.c:99]                 Waiting for 3 iomgr objects to be destroyed
D0628 16:18:08.663269688   10465 iomgr.c:99]                 Waiting for 3 iomgr objects to be destroyed
D0628 16:18:09.664535323   10465 iomgr.c:99]                 Waiting for 3 iomgr objects to be destroyed
D0628 16:18:10.665863923   10465 iomgr.c:99]                 Waiting for 3 iomgr objects to be destroyed
D0628 16:18:11.667386620   10465 iomgr.c:99]                 Waiting for 3 iomgr objects to be destroyed
D0628 16:18:12.668871835   10465 iomgr.c:99]                 Waiting for 3 iomgr objects to be destroyed
D0628 16:18:13.670009823   10465 iomgr.c:99]                 Waiting for 3 iomgr objects to be destroyed
D0628 16:18:14.671078765   10465 iomgr.c:99]                 Waiting for 3 iomgr objects to be destroyed
D0628 16:18:15.672284821   10465 iomgr.c:99]                 Waiting for 3 iomgr objects to be destroyed
D0628 16:18:16.673620721   10465 iomgr.c:118]                Failed to free 3 iomgr objects before shutdown deadline: memory leaks are likely
D0628 16:18:16.673701411   10465 iomgr.c:80]                 LEAKED OBJECT: tcp-client:ipv4:216.58.194.106:443 fd=6 0x7fea300045a8
D0628 16:18:16.673723755   10465 iomgr.c:80]                 LEAKED OBJECT: tcp-client:ipv4:216.58.218.138:443 fd=12 0x7fea30005168
D0628 16:18:16.673746282   10465 iomgr.c:80]                 LEAKED OBJECT: tcp-client:ipv4:216.58.194.42:443 fd=23 0x7fea30006888
D0628 16:18:16.673830891   10465 metadata.c:238]             WARNING: 1 metadata elements were leaked
D0628 16:18:16.673872448   10465 metadata.c:238]             WARNING: 3 metadata elements were leaked
D0628 16:18:16.673903482   10465 metadata.c:238]             WARNING: 2 metadata elements were leaked
D0628 16:18:16.673927907   10465 metadata.c:238]             WARNING: 2 metadata elements were leaked
D0628 16:18:16.673955019   10465 metadata.c:238]             WARNING: 1 metadata elements were leaked
D0628 16:18:16.673982805   10465 metadata.c:238]             WARNING: 9 metadata elements were leaked
D0628 16:18:16.674006559   10465 metadata.c:238]             WARNING: 3 metadata elements were leaked
D0628 16:18:16.674031913   10465 metadata.c:238]             WARNING: 3 metadata elements were leaked
D0628 16:18:16.674055336   10465 metadata.c:238]             WARNING: 2 metadata elements were leaked
D0628 16:18:16.674124369   10465 metadata.c:238]             WARNING: 2 metadata elements were leaked
D0628 16:18:16.674160625   10465 metadata.c:238]             WARNING: 2 metadata elements were leaked
D0628 16:18:16.674188014   10465 metadata.c:238]             WARNING: 3 metadata elements were leaked
D0628 16:18:16.674212776   10465 metadata.c:238]             WARNING: 2 metadata elements were leaked
D0628 16:18:16.674313468   10465 metadata.c:238]             WARNING: 7 metadata elements were leaked
D0628 16:18:16.674342804   10465 metadata.c:238]             WARNING: 4 metadata elements were leaked
D0628 16:18:16.674363545   10465 metadata.c:238]             WARNING: 2 metadata elements were leaked
D0628 16:18:16.674384519   10465 metadata.c:251]             WARNING: 2 metadata strings were leaked
D0628 16:18:16.674404965   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:17:40 GMT
D0628 16:18:16.674425889   10465 metadata.c:255]             LEAKED: 3
D0628 16:18:16.674449029   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.674470734   10465 metadata.c:255]             LEAKED: 443:quic
D0628 16:18:16.674491956   10465 metadata.c:251]             WARNING: 2 metadata strings were leaked
D0628 16:18:16.674518349   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:18:05 GMT
D0628 16:18:16.674541054   10465 metadata.c:255]             LEAKED: quic=":443"; ma=2592000; v="34,33,32,31,30,29,28,27,26,25"
D0628 16:18:16.674566538   10465 metadata.c:251]             WARNING: 5 metadata strings were leaked
D0628 16:18:16.674587736   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:17:55 GMT
D0628 16:18:16.674609252   10465 metadata.c:255]             LEAKED: /google.logging.v2.ConfigServiceV2/CreateSink
D0628 16:18:16.674633129   10465 metadata.c:255]             LEAKED: x-goog-api-client
D0628 16:18:16.674654447   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:17:35 GMT
D0628 16:18:16.674677346   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:17:29 GMT
D0628 16:18:16.674699780   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.674719762   10465 metadata.c:255]             LEAKED: Requested entity was not found.
D0628 16:18:16.674741234   10465 metadata.c:251]             WARNING: 2 metadata strings were leaked
D0628 16:18:16.674763658   10465 metadata.c:255]             LEAKED: logging.googleapis.com:443
D0628 16:18:16.674786116   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:18:02 GMT
D0628 16:18:16.674809218   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.674831358   10465 metadata.c:255]             LEAKED: 29600m
D0628 16:18:16.674853004   10465 metadata.c:251]             WARNING: 3 metadata strings were leaked
D0628 16:18:16.674875729   10465 metadata.c:255]             LEAKED: /google.logging.v2.LoggingServiceV2/ListLogEntries
D0628 16:18:16.674933951   10465 metadata.c:255]             LEAKED: Bearer ya29.CjAPA_i1yKBinTNZ2b9wS5FgZElPA8fZG4Xg24nF9mkKcyVlTSNZvNGKGLQVKah_X0U
D0628 16:18:16.674954541   10465 metadata.c:255]             LEAKED: 5
D0628 16:18:16.674974456   10465 metadata.c:251]             WARNING: 2 metadata strings were leaked
D0628 16:18:16.674993382   10465 metadata.c:255]             LEAKED: alt-svc
D0628 16:18:16.675010531   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:17:36 GMT
D0628 16:18:16.675029725   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.675048026   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:17:56 GMT
D0628 16:18:16.675066608   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.675084165   10465 metadata.c:255]             LEAKED: 30S
D0628 16:18:16.675104658   10465 metadata.c:251]             WARNING: 5 metadata strings were leaked
D0628 16:18:16.675123718   10465 metadata.c:255]             LEAKED: alternate-protocol
D0628 16:18:16.675142083   10465 metadata.c:255]             LEAKED: attachment
D0628 16:18:16.675159526   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:17:59 GMT
D0628 16:18:16.675177156   10465 metadata.c:255]             LEAKED: Invalid resource: id is missing
D0628 16:18:16.675194886   10465 metadata.c:255]             LEAKED: /google.logging.v2.MetricsServiceV2/ListLogMetrics
D0628 16:18:16.675217420   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.675236510   10465 metadata.c:255]             LEAKED: /google.logging.v2.MetricsServiceV2/CreateLogMetric
D0628 16:18:16.675259646   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.675278980   10465 metadata.c:255]             LEAKED: /google.logging.v2.LoggingServiceV2/WriteLogEntries
D0628 16:18:16.675306325   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.675325605   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:17:58 GMT
D0628 16:18:16.675345407   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.675364507   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:17:48 GMT
D0628 16:18:16.675384439   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.675402924   10465 metadata.c:255]             LEAKED: Requested metric was not found
D0628 16:18:16.675422288   10465 metadata.c:251]             WARNING: 4 metadata strings were leaked
D0628 16:18:16.675440743   10465 metadata.c:255]             LEAKED: /google.logging.v2.LoggingServiceV2/DeleteLog
D0628 16:18:16.675520524   10465 metadata.c:255]             LEAKED: /google.logging.v2.MetricsServiceV2/UpdateLogMetric
D0628 16:18:16.675546900   10465 metadata.c:255]             LEAKED: Bearer ya29.CjAPAzw3OKeovYMPda0Dx5OOoAtN8irv6v_WXnxPZqHU0Ryo2cs_gyxXbnVdnrUaGh8
D0628 16:18:16.675567006   10465 metadata.c:255]             LEAKED: /google.logging.v2.ConfigServiceV2/GetSink
D0628 16:18:16.675588591   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.675608092   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:18:00 GMT
D0628 16:18:16.675626798   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.675644204   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:18:06 GMT
D0628 16:18:16.675663752   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.675682391   10465 metadata.c:255]             LEAKED: Python-gRPC-0.15.0 grpc-c/0.15.0 (manylinux; chttp2)
D0628 16:18:16.675703338   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.675722238   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:17:31 GMT
D0628 16:18:16.675740482   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.675758867   10465 metadata.c:255]             LEAKED: /google.logging.v2.ConfigServiceV2/DeleteSink
D0628 16:18:16.675778098   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.675794892   10465 metadata.c:255]             LEAKED: /google.logging.v2.ConfigServiceV2/UpdateSink
D0628 16:18:16.675813321   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.675831682   10465 metadata.c:255]             LEAKED: gax/0.12.1 gapic/0.1.0 gax/0.12.1 python/2.7.11
D0628 16:18:16.675852964   10465 metadata.c:251]             WARNING: 6 metadata strings were leaked
D0628 16:18:16.675870778   10465 metadata.c:255]             LEAKED: /google.logging.v2.ConfigServiceV2/ListSinks
D0628 16:18:16.675891221   10465 metadata.c:255]             LEAKED: Requested sink was not found
D0628 16:18:16.675909202   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:18:01 GMT
D0628 16:18:16.675927607   10465 metadata.c:255]             LEAKED: Bearer ya29.CjAPA5-BwfB1hVHRXhtfoCZPg6D-7EYUUxcVTSPm3hvnTOd3Z25uwnFbd1MSyH_Tz58
D0628 16:18:16.675945394   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:17:27 GMT
D0628 16:18:16.675963916   10465 metadata.c:255]             LEAKED: /google.logging.v2.MetricsServiceV2/DeleteLogMetric
D0628 16:18:16.675984275   10465 metadata.c:251]             WARNING: 2 metadata strings were leaked
D0628 16:18:16.676002156   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:17:53 GMT
D0628 16:18:16.676017818   10465 metadata.c:255]             LEAKED: Tue, 28 Jun 2016 20:17:39 GMT
D0628 16:18:16.676033360   10465 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0628 16:18:16.676048260   10465 metadata.c:255]             LEAKED: /google.logging.v2.MetricsServiceV2/GetLogMetric

kpayson64 · 2016-06-28T20:36:50Z

@tseaver Good to hear!

It looks like you are running into an issue I've been investigating where outstanding calls are not properly cleaned up on exit. I've opened #7121 for tracking.

zj8487 · 2016-07-04T10:28:36Z

Ctrl + C can not make the client to exit with grpcio 0.15.0
and it is not to appear all the time

Traceback (most recent call last):
  File "greeter_client.py", line 70, in <module>
    run()
  File "greeter_client.py", line 61, in run
    response = stub.SayHello.future(helloworld_pb2.HelloRequest(name='you'))
  File "/home/xxxx/.virtualenvs/xxxx/local/lib/python2.7/site-packages/grpc/_channel.py", line 477, in future
    request, timeout, metadata)
  File "/home/xxx/.virtualenvs/xxxx/local/lib/python2.7/site-packages/grpc/_channel.py", line 440, in _prepare
    state = _RPCState(_UNARY_UNARY_INITIAL_DUE, None, None, None, None)
  File "/home/xxxx/.virtualenvs/xxxx/local/lib/python2.7/site-packages/grpc/_channel.py", line 109, in __init__
    self.due = set(due)
KeyboardInterrupt
^C^CE0704 17:57:48.988460597   29195 chttp2_transport.c:1810]    close_transport: {"created":"@1467626268.959884871","description":"Failed parsing HTTP/2","file":"src/core/ext/transport/chttp2/transport/chttp2_transport.c","file_line":1726,"referenced_errors":[{"created":"@1467626268.959881826","description":"EOF","file":"src/core/lib/iomgr/tcp_posix.c","file_line":233}]}
^C^C^C

zj8487 · 2016-07-04T10:30:28Z

@kpayson64 Is it still a issue on the master branch?

kpayson64 · 2016-07-06T15:07:14Z

@zj8487
This should be fixed in the master branch.

minhaz1 · 2016-08-02T01:08:29Z

I'm still having this issue with grpcio-1.0.0rc1.

kpayson64 · 2016-08-02T01:13:43Z

@minhaz1
Can you post the code that is causing this issue?

There is a known issue with Python2.7 hanging on exit with outstanding streaming/async calls.
#6999, but the decision was to not fix it.

minhaz1 · 2016-08-02T01:18:53Z

@kpayson64 that actually looks exactly like the problem I'm having.

nathanielmanistaatgoogle · 2016-08-02T01:54:03Z

@minhaz1: as the person who wrote that leaving the problem in place is "highly unlikely to have any effect in the real world", I'd be curious to hear more about how significant an obstacle it is for you. Also what is it that has you using Python 2.7?

pgrosu · 2016-08-02T02:10:09Z

@nathanielmanistaatgoogle Just a friendly aside, which versions of Python is gRPC supported for? I thought 2.7 was in the set of supported versions.

hsaliak · 2016-08-02T02:20:44Z

@pgrosu we support Python 2.7+ and 3.4+. However as noted in #6999 condition.wait() ignoring SIGINT is an issue that exists in Python 2.7 itself.

pgrosu · 2016-08-02T02:28:37Z

@hsaliak Fair enough, but it would be nice to add another column to the Repository Structure & Status table, and listing the supported versions for each language.

minhaz1 · 2016-08-02T02:51:31Z

@nathanielmanistaatgoogle what I am working on is going to be an internal library and consumers of the library will most likely be running as python 2.x. It isn't really a huge issue, as it does still handle other signals like SIGTERM correctly. It just made it inconvenient for testing things locally because I would have to Ctrl-Z + and kill %%. I just added a signal handler for my test client code to handle SIGINT and then send SIGTERM to the current process so I can Ctrl-C for local testing.

nathanielmanistaatgoogle · 2016-08-02T03:01:38Z

@minhaz1: good to know and thank you for sharing.

hsaliak · 2016-08-02T03:02:15Z

@pgrosu thanks for that suggestion, filed #7604 to track

pgrosu · 2016-08-02T08:16:29Z

@hsaliak Thank you - very appreciated.

alamaison · 2016-08-02T21:06:12Z

@minhaz1: you can just ctrl+\ to kill a process. If that doesnt work, try
ctrl+|

On Tue, 2 Aug 2016, 09:16 Paul Grosu, notifications@github.com wrote:

@hsaliak https://github.com/hsaliak Thank you - very appreciated.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#3820 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAkbBzgf2hbDAKqwvatJVoKHS0RpsJqOks5qbvzxgaJpZM4GOc70
.

nathanielmanistaatgoogle added the lang/Python label Oct 14, 2015

hsaliak added this to the GA milestone Nov 5, 2015

ctiller assigned nathanielmanistaatgoogle Nov 5, 2015

nathanielmanistaatgoogle mentioned this issue Nov 18, 2015

Fixes to stub lifecycle #4149

Closed

This was referenced Dec 3, 2015

Stub and server lifecycle fixes #4268

Closed

Stub and server lifecycle fixes #4269

Merged

hsaliak added the priority/P1 label Apr 21, 2016

nicolasnoble modified the milestones: GA, 2016/05/21 Milestone Apr 21, 2016

tbetbetbe mentioned this issue Apr 27, 2016

'_at_exit' hook hangs, prevents clean shutdown googleapis/gax-python#96

Closed

nathanielmanistaatgoogle mentioned this issue May 20, 2016

grpc client does not respect SIGINT when it can't connect to the server #4148

Closed

jeanlucmongrain mentioned this issue May 31, 2016

Python Client 0.14.0 don't recover from connection failure #6730

Closed

hsaliak assigned kpayson64 and unassigned nathanielmanistaatgoogle Jun 8, 2016

kpayson64 mentioned this issue Jun 10, 2016

Added check for signals in poll() #6873

Merged

kpayson64 mentioned this issue Jun 22, 2016

Updated several threads to use CleanupThread for clean exit #7001

Merged

kpayson64 closed this as completed Jun 27, 2016

nathanielmanistaatgoogle mentioned this issue Jul 4, 2016

Memory leak in GRPC python client! #6619

Closed

lock bot locked as resolved and limited conversation to collaborators Oct 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot stop python client with Ctrl-C #3820

Cannot stop python client with Ctrl-C #3820

yokoi-h commented Oct 14, 2015

nathanielmanistaatgoogle commented Oct 14, 2015

yokoi-h commented Oct 16, 2015

kyleconroy commented Oct 17, 2015

shuowusc commented Oct 26, 2015

hsaliak commented Apr 21, 2016

nathanielmanistaatgoogle commented Apr 23, 2016

GuoJing commented May 31, 2016

jaytaylor commented Jun 15, 2016

nathanielmanistaatgoogle commented Jun 16, 2016

GuoJing commented Jun 16, 2016

tseaver commented Jun 28, 2016

kpayson64 commented Jun 28, 2016

zj8487 commented Jul 4, 2016 •

edited

Loading

zj8487 commented Jul 4, 2016

kpayson64 commented Jul 6, 2016

minhaz1 commented Aug 2, 2016

kpayson64 commented Aug 2, 2016

minhaz1 commented Aug 2, 2016

nathanielmanistaatgoogle commented Aug 2, 2016

pgrosu commented Aug 2, 2016

hsaliak commented Aug 2, 2016

pgrosu commented Aug 2, 2016

minhaz1 commented Aug 2, 2016

nathanielmanistaatgoogle commented Aug 2, 2016

hsaliak commented Aug 2, 2016

pgrosu commented Aug 2, 2016

alamaison commented Aug 2, 2016

Cannot stop python client with Ctrl-C #3820

Cannot stop python client with Ctrl-C #3820

Comments

yokoi-h commented Oct 14, 2015

nathanielmanistaatgoogle commented Oct 14, 2015

yokoi-h commented Oct 16, 2015

kyleconroy commented Oct 17, 2015

shuowusc commented Oct 26, 2015

hsaliak commented Apr 21, 2016

nathanielmanistaatgoogle commented Apr 23, 2016

GuoJing commented May 31, 2016

jaytaylor commented Jun 15, 2016

nathanielmanistaatgoogle commented Jun 16, 2016

GuoJing commented Jun 16, 2016

GuoJing

tseaver commented Jun 28, 2016

kpayson64 commented Jun 28, 2016

zj8487 commented Jul 4, 2016 • edited Loading

zj8487 commented Jul 4, 2016

kpayson64 commented Jul 6, 2016

minhaz1 commented Aug 2, 2016

kpayson64 commented Aug 2, 2016

minhaz1 commented Aug 2, 2016

nathanielmanistaatgoogle commented Aug 2, 2016

pgrosu commented Aug 2, 2016

hsaliak commented Aug 2, 2016

pgrosu commented Aug 2, 2016

minhaz1 commented Aug 2, 2016

nathanielmanistaatgoogle commented Aug 2, 2016

hsaliak commented Aug 2, 2016

pgrosu commented Aug 2, 2016

alamaison commented Aug 2, 2016

zj8487 commented Jul 4, 2016 •

edited

Loading