Specification file for logging implementation #94

GowthamShanmugam · 2016-12-19T19:25:29Z

Signed-off-by: GowthamShanmugam gshanmug@redhat.com

tendrl-bug-id: #28

GowthamShanmugam · 2016-12-19T19:25:47Z

@brainfunked @shtripat please review

GowthamShanmugam · 2016-12-20T08:33:03Z

@shtripat please review this

shtripat · 2016-12-21T10:51:10Z

specs/logging_implementation.adoc

@@ -0,0 +1,174 @@
+= Specification for logging implementation
+
+Logs and alerts need to be treated as messages emitted by a component.


"...emitted by Tedrl modules" may be I would say

i will change it

'..by Tendrl'?

shtripat · 2016-12-21T10:53:49Z

specs/logging_implementation.adoc

@@ -0,0 +1,174 @@
+= Specification for logging implementation


"Generic message mechanism for logs and alerts in Tendrl"

i will change it

shtripat · 2016-12-21T10:56:43Z

specs/logging_implementation.adoc

+
+Logs and alerts need to be treated as messages emitted by a component.
+
+== Problem description


The problem statement for this case is something like

The logs and alerts in Tendrl are handled differently and there is no standard mechanism which makes sure no internal details are exposed as part of alerts/notifications sent to end user. Ideally logs messages and alerts are both messages which are emitted from an application. Tendrl needs to treat them as one entity and a common handler should decide what to be done with the message which is emitted and received at handler."

The below text talks more of implementation how and what you want to do rather :)

shtripat · 2016-12-21T10:57:59Z

specs/logging_implementation.adoc

+* Notifications and alerts need to be delivered to the node agent alert sockets.
+* Commons needs to implement a logging library that hides all the implementation
+  details and configuration options from any of the components using the library.
+  Components should be able to blindly emit messages to the common logging module


We should rather call it Message Handler module in Tendrl I feel

shtripat · 2016-12-21T11:05:01Z

specs/logging_implementation.adoc

+Logs and alerts need to be treated as messages.
+
+== Proposed change
+* Create a new class called "Message" in common with attributes name, type,


I feel we would need info about the module which is raising this message so that module specific logging configuration can be used for logging the messages. I am coming from a point where each module like node_agent, ceph_integration, gluster_integartion etc can have their own log-levels and loggers defined.
Does not look very clear how this scenario will be taken care with this model..

shtripat · 2016-12-21T11:07:25Z

specs/logging_implementation.adoc

+```
+* Expected output format for the alert in node_agent socket:
+```
+{


This looks very minimal and alerts might need additional fields related to acknowledgment etc. You may refer skyring for detailed object structure may be.

ack required for events right?

shtripat · 2016-12-21T11:08:49Z

specs/logging_implementation.adoc

+  message, priority(empty for log), alert_id ,log_level(empty for alert),
+  function_name, line_number.
+* Using name retrieve the particular logging handler.
+* Create a new function for Message class and process the object values.


I feel there should be a generic class defined as MessageHandler and extend this into two classes namely LogMessageHandler and AlertHandler. This class would have a handle() function which is specific in nature and takes care of the actual logic.

@brainfunked told me we have to log alert also in log file. so spiting like LogMessageHandler and AlertHandler is good?

yes i feel same .. we cant split

shtripat · 2016-12-21T11:10:02Z

specs/logging_implementation.adoc

+
+=== Tendrl/node_agent impact
+
+* Import the Message class in all the modules and Change log statements to


I feel we should add the framework in one PR and do the mass change in different PR. Makes life easier and doesnt halt everybody else :)

ok .. i will change as per your suggestion

brainfunked · 2016-12-22T09:51:44Z

specs/logging_implementation.adoc

@@ -0,0 +1,174 @@
+= Specification for logging implementation


brainfunked · 2016-12-22T09:52:03Z

specs/logging_implementation.adoc

@@ -0,0 +1,174 @@
+= Specification for logging implementation
+
+Logs and alerts need to be treated as messages emitted by a component.


'..by Tendrl'?

brainfunked · 2016-12-22T09:55:10Z

specs/logging_implementation.adoc

+* A schema needs to be defined for emitted messages. This could be a json schema.
+  The schema needs to include metadata that allows the exact point of origin of
+  the message to be known.
+* Message priorities, types etc. need to be defined at the time of emission.


'..at the point of origin'.

brainfunked · 2016-12-22T11:10:38Z

specs/logging_implementation.adoc

+  and it should be able to deliver them to either syslog, log files, alert
+  sockets etc. depending upon the metadata and configuration.
+* Syslog must be the default logging configuration. Logging to individual
+  configuration files should be thoroughly discouraged.


'..discouraged by default.'

brainfunked · 2016-12-22T11:14:23Z

specs/logging_implementation.adoc

+
+Logs and alerts need to be treated as messages emitted by a component.
+
+== Problem description


Additionally, the updates to in-flight operations, #55, need to be logged under INFO. The updates themselves should just be messages emitted that have the same structure and appropriate metadata. This metadata includes operation uuids. This implies that the details of the jobs being created also need to be logged. This is where tracing and debugging is addressed. The updates being logged provide an id. This id pertains to a specific job. This job being logged enables end-to-end tracing of the job and it's updates across the deployment. This is very important, since it allows an administrator to filter the logs by id and get all the required details.

t0ffel

Hi, I'm with Common Logging. I was asked to take a look at this PR.
I'm not very familiar with tendrl, so please forgive me if my questions occasionally sound stupid.

t0ffel · 2016-12-22T18:51:41Z

specs/logging_implementation.adoc

+
+== Problem description
+
+* Logs and alerts need to be treated as messages emitted by a component.


Apologies in advance as I am a bit out of context on Tendrl, but what is the difference between logs and alerts within the scope of this specification?

Log nothing but a information, erros. etc. It helps an end_user for the debugging the problem. Alert also a message but it is immediately forward to end_user. Alerts can be debug with the help of log message.

t0ffel · 2016-12-22T19:23:08Z

specs/logging_implementation.adoc

+
+== Proposed change
+* Create a new class called "Message" in common with attributes name, type,
+  message, priority(empty for log), alert_id ,log_level(empty for alert),


Several questions:

What about timestamp?

Is there a place for extra metadata that might accompany the message?

Do you plan to have possible values/expected format for these fields? i.e.

does the log_level have 1-1 mapping to syslog's severity?

What is the format of the message? Can it be a multiline backtrace?

1.ya i have toadd timestamp also,
2.I have declared class called Message for holding the message and other details. And alert and log can be handled from that object.
3.
* yes log_level have 1-1 mapping to syslog's : log level also came from object, with the help of log level i am calling syslog function.
@brainfunked your comments pleases

t0ffel · 2016-12-22T19:32:05Z

specs/logging_implementation.adoc

+```
+* Import the Message class in all the modules and Change log statements to
+  populate the instance of message class and pass the object to function.
+* Expected output format for log and alert in log file:


what about sending logs to syslog? What will the logs/alerts look like in syslog/journal? Will you be using any special systemd journal fields?

no we are not using any special journal,
using loglevel we are calling exact syslog function to log the message

alert is always warning. For log Based n loglevel it is decided (info, error....)

t0ffel · 2016-12-22T19:33:50Z

specs/logging_implementation.adoc

+* Expected output format for log and alert in log file:
+```
+Format: "%(asctime)s - %(levelname)s - %(message)s"
+Message: "%(filename)s - %(function)s - %(lineno)s - %(message)s"


I'm a bit confused. In https://github.com/Tendrl/specifications/pull/94/files#diff-5a251eddb4570c9ab1b30f91c2d17180R54 it says that json will be stored.
Also what exactly do you mean by separate lines for Format and Message?

What is my idea is Create class called Message, This message class have all the attributes like
"name" (log_handler_name) (each module have different logging conf, so we are going to log message in common place, so we need log_handler for that corresponding module).
"type" (alert or log) (we log alert also but alert have one more step, alert is passed to enduser immediately ).
"log level" - (it tells what is the log_level like info, error.... for log the message in syslog, alert is default warning).
"timestamp"
"severity" (for log it is empty)
"aler_id"
"tag" (for alert)
"message" - message is nothing but text what they expected to log or alert. Developer will hard code the message. Why i gave message format means
if you give log format Format: "%(asctime)s - % (filename) - %(function) - %(linenumber) %(levelname)s - %(message)s" then log message logged with filename, functionname, loneno where the loglevel function called.

Here we are collecting all object in handler function (in message class) and logging from that function itself. So log message will take handler function's name, filename, lineno in log file. To avoid this the handler will collect the filename, function name, lineno automatically where the object is came from and it will append these details with message and then log that message in log file. So that appending format is Message: "%(filename)s - %(function)s - %(lineno)s - %(message)s".

we are using message object to collect all info, no need any json, i have to modify that line

format is nothing but we give logging format when we create log handler,
message format nothing but appending extra info with message automatically

GowthamShanmugam · 2016-12-28T08:53:59Z

@shtripat @brainfunked @nnDarshan @anmolbabu please review this

GowthamShanmugam · 2016-12-28T08:54:16Z

@brainfunked @shtripat @anmolbabu @nnDarshan

GowthamShanmugam · 2016-12-28T10:14:06Z

@brainfunked @shtripat @nnDarshan please review this

shtripat · 2016-12-29T04:41:35Z

specs/logging_implementation.adoc

+Logs and alerts need to be treated as messages emitted by Tendrl.
+
+== Problem description
+* The logs, alerts and flight_operations_updates in Tendrl are handled differently


in flight operations updates

shtripat · 2016-12-29T04:44:39Z

specs/logging_implementation.adoc

+     log_level(info/error..)
+     message
+     message_trace_back_flag(True/False)
+     message_uuid (optional for log)


It should not optional for logs may be we should use current job id as message id so that tracing of the error becomes easier while debugging.

@shtripat yes actually my logic also same why i mentioned here as optional means, for flight operation i am using job id as message id, for alerting i am using alert id as message id but what about normal exception logs. It does not required any id. What i try to mentioned here is for normal log messages it is optional

May be make the sentences more clearer in that case

@shtripat ok

shtripat · 2016-12-29T04:45:31Z

specs/logging_implementation.adoc

+  object values.
+* If the message type is Alert then store the message in log file with uuid (log level
+  for alert is always warning). and put the object in node_agent socket.
+* If the message type is flight_operation then log the message in file with message


@shtripat ok i will change it

shtripat · 2016-12-29T04:45:59Z

specs/logging_implementation.adoc

+=== Tendrl/node_agent impact
+
+* Create socket to receive alert.
+* Import message class from common and change all log statement (it is handled in another PR).


Put the link to other PR here

@shtripat For this specification is any required?

All I meant is instead if saying handled in another PR put the reference of that PR

@shtripat ok i will do it

shtripat · 2016-12-29T04:47:29Z

specs/logging_implementation.adoc

+  Message: ""%(asctime)s - %(filename)s - %(function)s - %(lineno)s - %(levelname)s
+  - %(message)s"
+
+  (No need to specify any log format in configuration file. We can remove the format


Need to POC this I feel

@shtripat done

anmolbabu · 2016-12-29T09:05:05Z

specs/logging_implementation.adoc

+  are emitted from an application. Tendrl needs to treat them as one entity and
+  a common handler should decide what to be done with the message which is emitted
+  and received at handler.
+== Use Cases


You probably need to leave a line above

@anmolbabu done

anmolbabu · 2016-12-29T09:14:52Z

specs/logging_implementation.adoc

+  we have to find the function name, line number, file name and append this data
+  with message manually).
+  ```
+  from inspect import getframeinfo, stack


This is not looking properly in html generated from adoc use asciidoctor <file_name>.adoc after installing asciidoctor. The command generates a html file in current directory and that you can open in any browser and check it. You probably need to leave a line before or not sure if is a valid symbol in adoc. Please check it.

@anmolbabu this symbol should start from first column without any space, then it will work

@GowthamShanmugam better follow a line gap and start from first column of line always.

@GowthamShanmugam please check https://github.com/GowthamShanmugam/specifications/blob/4bdc50f125b05162fcf36117266b2fcba21ae745/specs/logging_implementation.adoc
for seeing how the code part looks currently. It looks mis-aligned and not as intended.

i will correct it

anmolbabu · 2016-12-29T13:53:32Z

specs/logging_implementation.adoc

+
+== Notifications/Monitoring impact
+
+None


We should probably say:

The collectd plugin to handle collectd detected threshold breaches should be customized to call the utility function described in this adoc instead of .directly posting the alert to socket

anmolbabu · 2016-12-29T13:55:44Z

specs/logging_implementation.adoc

+
+=== Tendrl/gluster_integration impact
+
+* Import message class from common and change all log statement (it will be in another PR).


We should also be adding this:

gluster-integration needs to invoke the utility described in this PR for every alert detected by it.

Similarly for ceph-integration also

i will add this

GowthamShanmugam · 2017-01-02T10:11:06Z

@brainfunked @shtripat @nthomas-redhat @nnDarshan @anmolbabu please review this

mkudlej · 2017-01-02T11:26:27Z

Please add usmqe/usmqe-tests#14 as reference for testing alerts.

GowthamShanmugam · 2017-01-02T11:33:24Z

@mkudlej done

GowthamShanmugam · 2017-01-03T06:33:55Z

@anmolbabu @shtripat @brainfunked @nthomas-redhat please review

shtripat · 2017-01-03T07:48:45Z

specs/logging_implementation.adoc

+* Notifications sent to the administrator should provide sufficient context to
+  enable debugging of the problem via investigation into the log files.
+* Notifications and alerts need to be delivered to the node agent alert sockets.
+* Commons needs to implement a logging library that hides all the implementation


commons module??

@shtripat *common

I mean you should say "common module"

shtripat · 2017-01-03T07:57:17Z

specs/logging_implementation.adoc

+     for alert is always warning). and put the object in node_agent socket.
+  2. If the message type is flight operation then log the message in file with message
+     uuid (uuid is used to trace the filght operation).
+  3. If the type is Log then process then store message in log file..


@shtripat i will change it

shtripat · 2017-01-03T07:58:33Z

specs/logging_implementation.adoc

+     "type"           : "string",
+     "tags"           : "string",
+     "ackedby"        : "string",
+     "acked"          : "boolean"


Should we have ack_comment as well here?

@shtripat ack_comment? why this is required , @anmolbabu your suggestion please

While acking a notification user might provide some comment and that we can store.

@anmolbabu your suggestion please

@GowthamShanmugam @shtripat comment looks like is required but I missed it somehow in my spec that actually talks about alert structure. But its my opinion that, instead of having changes to the alert structure and details around it be effected in 2 places this spec and also https://github.com/Tendrl/specifications/blob/master/specs/pluggable_alert_delivery.adoc. So I feel we can open a bug to capture this point in pluggable_alert_delivery.adoc and just refer that spec in this spec for details about alerts. @shtripat what do you suggest

shtripat · 2017-01-03T07:59:17Z

specs/logging_implementation.adoc

+
+import uuid
+
+class alert(Message):


Class name starts with caps

@shtripat i will change it

Please refer my comment @ line 208

@anmolbabu ok

GowthamShanmugam · 2017-02-13T12:28:26Z

@TimothyAsir @brainfunked @anmolbabu @nthomas-redhat @shtripat At the time of logging a message, if the socket fails then log message never reach to logging framework. In this case we have to intimate the end user like socket is down and log messages are not logged successfully. Other wise the log message will lost. The problem here is without socket we can log the message anywhere, we need some mechanism to handle this case.

shtripat · 2017-02-14T13:17:18Z

@GowthamShanmugam I am not sure how to tackle, but as a preferable option I would like to log these kind of failures in some common tendrl files say /var/log/tendrl/tendrl.log. Just a wierd thought...

nthomas-redhat · 2017-02-15T00:03:07Z

@TimothyAsir @brainfunked @anmolbabu @nthomas-redhat @shtripat At the time of logging a message, if the socket fails then log message never reach to logging framework. In this case we have to intimate the end user like socket is down and log messages are not logged successfully. Other wise the log message will lost. The problem here is without socket we can log the message anywhere, we need some mechanism to handle this case.

@GowthamShanmugam
In case of logger failure :
log errors to stderr and messages to stdout , this should ultimately go into systemd journals.
we should be able to alert to etcd directly via the node agent in case there's a problem with the socket .
Errors should be logged to stderr always regardless of logger failed or not

GowthamShanmugam · 2017-02-15T06:12:03Z

@nthomas-redhat i will change as per your suggestion

brainfunked · 2017-02-21T08:35:42Z

specs/logging_implementation.adoc

@@ -0,0 +1,308 @@
+= Generic message mechanism for logs and alerts in Tendrl"


Remove the " at the end.

brainfunked · 2017-02-21T08:48:28Z

specs/logging_implementation.adoc

+
+* Store the operation details in etcd:
+```
+1. Operation messages with inorder_id


The etcd message directory locations are missing. Add them.

For job specific updates, the following structure is expected:

/queue/<job_id> where job_id is the UUID of the job with no prefixes or suffixes.

/queue/<job_id>/parameters containing all the parameters for the job.

/queue/<job_id>/messages containing the sequentially generated keys, each containing a single message pertaining to the job.

In addition to these, there needs to be a specific file that updates with the job's overall status and marks the job as finished, either successfully or with errors, with the error details etc. @anivargi your inputs are needed here. The concern is to avoid multiple queries per poll for a job, if possible.

@brainfunked job_key is not available for flows, i had discuss with @shtripat and @nnDarshan they said when the job is received by component update parameter with job_key.
Because no need to change signature of any function for call job_key separately.

@brainfunked
My understanding is all the parameter of jobs are stored under a non-sequential key i.e
/queue/<job_id>/parameters, this is done by api during job submission. i have to this with @anivargi help.

Message handler module should place the job updates under /queue/<job_id>/messages/ for particular job.

And /queue/<job_id>/finished contains job status like job finished with error, errors, etc...

is this correct?

@GowthamShanmugam
Here is how it needs to be
/queue/<job_id> will be a directory
/queue/<job_id>/status is the status of the current job i.e new, processing, finished, error
/queue/<job_id>/errors is the errors caused in the job if the status is error.
/queue/<job_id>/parameters is the parameters of the job which is queued.
/queue/<job_id>/messages will be a directory containing all the messages.
/queue/<job_id>/messages/sequence will be the sequence number and value will be the message string

Other possible attributes under /queue/<job_id>/.. could be integration_id, run, flow, type, node_ids.

Hope its clear now.
//@brainfunked

@anivargi where would the "run" , "node_ids" etc items go as per this new structure?

@r0h4n the run would come in as /queue/<job_id>/run and node_ids will come in as /queue/<job_id>/node_ids. Note that node_ids would be a comma separated value since we cannot represent an array in etcd value like we do in json today.

tendrl-bug-id: Tendrl#28 Signed-off-by: GowthamShanmugam <gshanmug@redhat.com>

GowthamShanmugam · 2017-02-27T08:47:06Z

@brainfunked @r0h4n @shtripat Spec is updated

anivargi · 2017-03-02T13:01:36Z

@GowthamShanmugam slight changes are going to be have to made as follows:

/queue/:job_id will be a directory
/queue/:job_id/payload will be the json dump of the job with the same structure we have today
/queue/:job_id/status will be the global status which needs to be updated as per the currentstatus of the job, default will be 'new'
/queue/:job_id/errors will be a message if the status is error
/queue/:job_id/messages will be the log messages related to the job.

The request_id is not need anymore for the API to know about the job messages.
CC: @r0h4n @brainfunked

anivargi · 2017-03-02T15:39:30Z

@brainfunked @GowthamShanmugam I would also like to recommend if we could put the messages under something like /messages/jobs/:job_id, would that be a option?

anivargi · 2017-03-02T16:12:50Z

@GowthamShanmugam We need to move the messages per job under /messages/jobs

/messages/jobs/:job_id will be a directory
/messages/jobs/:job_id/:sequence will have value of the message

Other non job related messages will reside under
/messages/events

CC: @brainfunked

GowthamShanmugam · 2017-03-06T15:13:21Z

@brainfunked @anivargi @r0h4n as per above discussion i have modified job structure https://github.com/Tendrl/commons/pull/175/files

r0h4n · 2017-03-07T20:44:19Z

@anivargi I think we have missed the "parent_id" field in this job structure

GowthamShanmugam · 2017-03-15T10:02:28Z

@r0h4n @anivargi as per current job structure parent id also comes under payload

GowthamShanmugam · 2017-03-16T07:19:04Z

@anivargi @brainfunked
Messages are stored in different places

If cluster_id present
/clusters/cluster_id/Messages
else
/nodes/node_id/Messages
If job_id present
stored in /Messages/jobs
also /Messages/events
else:
only /Messages/events
is this ok (or) job_updates should be stored in /Messages/jobs only not in /Messages/events, /clusters/cluster_id/Messages or /nodes/node_id/Messages.

which is correct?

GowthamShanmugam force-pushed the logging branch from 46903d3 to 71f74d2 Compare December 20, 2016 08:19

GowthamShanmugam mentioned this pull request Dec 21, 2016

Logging implementation #28

Closed

shtripat reviewed Dec 21, 2016

View reviewed changes

brainfunked suggested changes Dec 22, 2016

View reviewed changes

brainfunked mentioned this pull request Dec 22, 2016

Updates for in flight operations #55

Closed

t0ffel reviewed Dec 22, 2016

View reviewed changes

GowthamShanmugam force-pushed the logging branch from 71f74d2 to be115c3 Compare December 28, 2016 08:53

GowthamShanmugam closed this Dec 28, 2016

GowthamShanmugam reopened this Dec 28, 2016

shtripat reviewed Dec 29, 2016

View reviewed changes

GowthamShanmugam force-pushed the logging branch 2 times, most recently from 4750c33 to 4bdc50f Compare December 29, 2016 08:52

anmolbabu reviewed Dec 29, 2016

View reviewed changes

GowthamShanmugam force-pushed the logging branch 2 times, most recently from 55412d0 to aa3916a Compare January 2, 2017 10:09

mkudlej mentioned this pull request Jan 2, 2017

alert socket test usmqe/usmqe-tests#14

Closed

GowthamShanmugam force-pushed the logging branch from aa3916a to 7e709dc Compare January 2, 2017 11:33

GowthamShanmugam force-pushed the logging branch from 7e709dc to 9f19062 Compare January 3, 2017 06:43

shtripat reviewed Jan 3, 2017

View reviewed changes

GowthamShanmugam force-pushed the logging branch from 4669e88 to 5323b91 Compare January 31, 2017 09:57

nthomas-redhat mentioned this pull request Feb 15, 2017

Logging implementation for tendrl Tendrl/node-agent#204

Closed

GowthamShanmugam force-pushed the logging branch from 5323b91 to e045085 Compare February 18, 2017 08:04

brainfunked suggested changes Feb 21, 2017

View reviewed changes

Specification file for logging implementation

9e1e24d

tendrl-bug-id: Tendrl#28 Signed-off-by: GowthamShanmugam <gshanmug@redhat.com>

GowthamShanmugam force-pushed the logging branch from e045085 to 9e1e24d Compare February 27, 2017 08:45

Update logging_implementation.adoc

59d8b04

r0h4n merged commit ca1bb5c into Tendrl:master Apr 28, 2017

		@@ -0,0 +1,174 @@
		= Specification for logging implementation

		Logs and alerts need to be treated as messages emitted by a component.


		Logs and alerts need to be treated as messages emitted by a component.

		== Problem description


		=== Tendrl/node_agent impact

		* Import the Message class in all the modules and Change log statements to


		== Problem description

		* Logs and alerts need to be treated as messages emitted by a component.


		=== Tendrl/gluster_integration impact

		* Import message class from common and change all log statement (it will be in another PR).

		@@ -0,0 +1,308 @@
		= Generic message mechanism for logs and alerts in Tendrl"

Specification file for logging implementation #94

Specification file for logging implementation #94

Conversation

GowthamShanmugam commented Dec 19, 2016 • edited Loading

GowthamShanmugam commented Dec 19, 2016

GowthamShanmugam commented Dec 20, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GowthamShanmugam Dec 21, 2016 • edited Loading

Choose a reason for hiding this comment

GowthamShanmugam Dec 22, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t0ffel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GowthamShanmugam Dec 23, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GowthamShanmugam Dec 23, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GowthamShanmugam commented Dec 28, 2016

GowthamShanmugam commented Dec 28, 2016

GowthamShanmugam commented Dec 28, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GowthamShanmugam Dec 29, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GowthamShanmugam Dec 29, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anmolbabu Dec 29, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GowthamShanmugam commented Jan 2, 2017

GowthamShanmugam commented Dec 19, 2016 •

edited

Loading

GowthamShanmugam Dec 21, 2016 •

edited

Loading

GowthamShanmugam Dec 22, 2016 •

edited

Loading

GowthamShanmugam Dec 23, 2016 •

edited

Loading

GowthamShanmugam Dec 23, 2016 •

edited

Loading

GowthamShanmugam Dec 29, 2016 •

edited

Loading

GowthamShanmugam Dec 29, 2016 •

edited

Loading

anmolbabu Dec 29, 2016 •

edited

Loading

mkudlej commented Jan 2, 2017 •

edited

Loading