Setting isBulk to true, causes "Error: Callback was already called." #13

sloops77 · 2017-02-26T08:50:27Z

we get this issue once we switched bulk delivery on.

AWS Linux
node 6 LTS
winston 2.2.0
NODE_ENV="staging"
here is the stack trace

Error: Callback was already called.
at <unknown> (/usr/src/app/node_modules/winston/node_modules/async/lib/async.js line 38 col 31)
if (called) throw new Error("Callback was already called.");
at <unknown> (/usr/src/app/node_modules/winston/lib/winston/logger.js line 236 col 7) next();
at logged (/usr/src/app/node_modules/winston-loggly-bulk/lib/winston-loggly.js line 113 col 5) callback(null, true);
at <unknown> (/usr/src/app/node_modules/node-loggly-bulk/lib/loggly/client.js line 179 col 9) callback(new Error('Unspecified error from Loggly: ' + ex));
at Request._callback (/usr/src/app/node_modules/node-loggly-bulk/lib/loggly/common.js line 198 col 9) success(res, body);
at Request.self.callback (/usr/src/app/node_modules/node-loggly-bulk/node_modules/request/request.js line 198 col 22) self._callback.apply(self, arguments) at emitTwo (events.js line 106 col 13)
at Request.emit (events.js line 191 col 7)
at Request.<anonymous> (/usr/src/app/node_modules/node-loggly-bulk/node_modules/request/request.js line 1035 col 10) self.emit('complete', response, response.body)
at emitOne (events.js line 101 col 20)
at Request.emit (events.js line 188 col 7)
at IncomingMessage.<anonymous> (/usr/src/app/node_modules/node-loggly-bulk/node_modules/request/request.js line 962 col 12) self.emit('end', chunk)
at emitNone (events.js line 91 col 20)
at IncomingMessage.emit (events.js line 185 col 7)
at endReadableNT (_stream_readable.js line 974 col 12)
at _combinedTickCallback (internal/process/next_tick.js line 74 col 11)
at process._tickDomainCallback [as _tickCallback] (internal/process/next_tick.js line 122 col 9)

The text was updated successfully, but these errors were encountered:

sloops77 · 2017-02-26T08:55:52Z

maybe related to wavded/winston-mail#33

mostlyjason · 2017-02-27T17:43:13Z

Thanks for reporting!

sloops77 · 2017-03-15T12:26:59Z

hey guys - any update?

mostlyjason · 2017-03-15T21:16:41Z

Hey we haven't had a chance to look at this one yet. We have some other tasks ahead of this one in the queue, including #12. Hoping we can look at this next month. If you find a solution before then please share

reharik · 2017-07-13T14:49:27Z

Sorry to me too, but me too. Really glad someone posted this cuz my stack trace is not available so I was kind of in the dark about what was causing it.

This is an example of a fix for loggly/winston-loggly-bulk#12 with a small amount of code change. Note that these lines: ``` if (timerFunction === null) { timerFunction = setInterval(function () { sendBulkLogs(); },30000); } ``` produce the same bug when `isBulk` is `true`. See loggly/winston-loggly-bulk#13

mostlyjason · 2017-09-22T18:58:14Z

We just released version 2.0.1 which we believe contains a fix for this. Please reopen if it happens again after updating.

ianaya89 · 2017-12-13T19:04:13Z

Hey, I am having the same issue with the version 2.0.1.

Heroku
node 8 LTS
winston 2.1.1

Stack Trace:

2017-12-13T18:55:19.838077+00:00 app[web.1]: Error Code- 504 "Gateway Time-out"
2017-12-13T18:55:19.875543+00:00 app[web.1]: 2017-12-13T18:55:19.874Z - error: uncaughtException: Callback was already called. date=Wed Dec 13 2017 18:55:19 GMT+0000 (UTC), pid=20, uid=5548, gid=5548, cwd=/app, execPath=/app/.heroku/node/bin/node, version=v8.9.3, argv=[/app/.heroku/node/bin/node, /app/bin/lkm-api], rss=66162688, heapTotal=39178240, heapUsed=31654440, external=18388932, loadavg=[4.5546875, 5.01171875, 5.40673828125], uptime=133822
2017-12-13T18:55:19.875545+00:00 app[web.1]: Error: Callback was already called.
2017-12-13T18:55:19.875547+00:00 app[web.1]:     at /app/node_modules/winston/node_modules/async/lib/async.js:38:31
2017-12-13T18:55:19.875548+00:00 app[web.1]:     at /app/node_modules/winston/lib/winston/logger.js:238:16
2017-12-13T18:55:19.875549+00:00 app[web.1]:     at logged (/app/node_modules/winston-loggly-bulk/lib/winston-loggly.js:129:5)
2017-12-13T18:55:19.875549+00:00 app[web.1]:     at /app/node_modules/node-loggly-bulk/lib/loggly/client.js:199:9
2017-12-13T18:55:19.875550+00:00 app[web.1]:     at Request._callback (/app/node_modules/node-loggly-bulk/lib/loggly/common.js:189:9)
2017-12-13T18:55:19.875551+00:00 app[web.1]:     at Request.self.callback (/app/node_modules/request/request.js:186:22)
2017-12-13T18:55:19.875551+00:00 app[web.1]:     at emitTwo (events.js:126:13)
2017-12-13T18:55:19.875552+00:00 app[web.1]:     at Request.emit (events.js:214:7)
2017-12-13T18:55:19.875552+00:00 app[web.1]:     at Request.<anonymous> (/app/node_modules/request/request.js:1163:10)
2017-12-13T18:55:19.875553+00:00 app[web.1]:     at emitOne (events.js:116:13)
2017-12-13T18:55:19.875553+00:00 app[web.1]:     at Request.emit (events.js:211:7)
2017-12-13T18:55:19.875554+00:00 app[web.1]:     at IncomingMessage.<anonymous> (/app/node_modules/request/request.js:1085:12)
2017-12-13T18:55:19.875555+00:00 app[web.1]:     at Object.onceWrapper (events.js:313:30)
2017-12-13T18:55:19.875555+00:00 app[web.1]:     at emitNone (events.js:111:20)
2017-12-13T18:55:19.875556+00:00 app[web.1]:     at IncomingMessage.emit (events.js:208:7)
2017-12-13T18:55:19.875556+00:00 app[web.1]:     at endReadableNT (_stream_readable.js:1056:12)
2017-12-13T18:55:19.875557+00:00 app[web.1]:     at _combinedTickCallback (internal/process/next_tick.js:138:11)
2017-12-13T18:55:19.875557+00:00 app[web.1]:     at process._tickDomainCallback (internal/process/next_tick.js:218:9)
2017-12-13T18:55:19.912412+00:00 app[web.1]: npm ERR! code ELIFECYCLE
2017-12-13T18:55:19.912822+00:00 app[web.1]: npm ERR! errno 1
2017-12-13T18:55:19.914500+00:00 app[web.1]: npm ERR! lkm-api@1.0.0 start: `./bin/lkm-api`
2017-12-13T18:55:19.914664+00:00 app[web.1]: npm ERR! Exit status 1
2017-12-13T18:55:19.914908+00:00 app[web.1]: npm ERR!
2017-12-13T18:55:19.927219+00:00 app[web.1]: npm ERR! Failed at the lkm-api@1.0.0 start script.
2017-12-13T18:55:19.940964+00:00 app[web.1]:
2017-12-13T18:55:19.927411+00:00 app[web.1]: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
2017-12-13T18:55:19.941165+00:00 app[web.1]: npm ERR! A complete log of this run can be found in:
2017-12-13T18:55:19.941315+00:00 app[web.1]: npm ERR!     /app/.npm/_logs/2017-12-13T18_55_19_929Z-debug.log

Thanks!

mostlyjason · 2017-12-13T20:20:06Z

@ianaya89 thanks for reporting this we'll have someone take a look at it. In the meantime, can you tell us which version of node-loggly-bulk you are using?

jfitzpatrick13 · 2017-12-14T15:21:55Z

Same issue as ianaya89
"winston": "2.4.0",
"winston-loggly-bulk": "2.0.1"

Error Code- 504 "Gateway Time-out"

        if (called) throw new Error("Callback was already called.");
                    ^

Error: Callback was already called.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! XXX_api@1.0.0 start: node dist/app.js
npm ERR! Exit status 1

ianaya89 · 2017-12-14T15:33:30Z

@mostlyjason Thanks for your reply. The version seems to be 2.0.1. But just to be clear, I am not using it directly, is a dependency of winston-loggly-bulk.

Shwetajain148 · 2017-12-15T11:05:06Z

Hi @ianaya89 and @jfitzpatrick13, Can you please confirm if you are using version 2.2.1 of node-loggly-bulk which is latest one. Also, please let me know the node and npm version that you are running.

Thank you.

cc: @mostlyjason

jfitzpatrick13 · 2017-12-15T13:36:01Z

@Shwetajain148,
Yes, the main package that I'm using is winston-loggly-bulk 2.0.1
It brings in
node-loggly-bulk 2.2.1
winston 2.4.0

My setup is
node 6.10.1
npm 3.10.10

Thanks

rushisan · 2017-12-19T07:46:32Z

The same problem problem
Error: Callback was already called.
at /app/node_modules/winston/node_modules/async/lib/async.js:38:31
at /app/node_modules/winston/lib/winston/logger.js:238:16
at logged (/app/node_modules/winston-loggly-bulk/lib/winston-loggly.js:129:5)
at /app/node_modules/node-loggly-bulk/lib/loggly/client.js:199:9
at Request._callback (/app/node_modules/node-loggly-bulk/lib/loggly/common.js:189:9)
at Request.self.callback (/app/node_modules/request/request.js:186:22)
at emitTwo (events.js:126:13)
at Request.emit (events.js:214:7)
at Request. (/app/node_modules/request/request.js:1163:10)
at emitOne (events.js:116:13)
at Request.emit (events.js:211:7)
at IncomingMessage. (/app/node_modules/request/request.js:1085:12)
at Object.onceWrapper (events.js:313:30)
at emitNone (events.js:111:20)
at IncomingMessage.emit (events.js:208:7)
at endReadableNT (_stream_readable.js:1056:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickDomainCallback (internal/process/next_tick.js:218:9)

I am using
winston 2.4.0
winston-loggly-bulk 2.0.1

Ubuntu 16.04
Node 8.9.1

Looks like it happens when loggly servers returns an error, or some network problems.

ianaya89 · 2017-12-19T16:06:53Z

Hi @Shwetajain148, I am using:

node: 8.9.1
npm: 5.6.0
node-loggly-bulk: 2.0.1

alimvedats · 2017-12-20T16:06:44Z

Same issue
node: 9.3.0
npm: 5.6.0
node-loggly-bulk: 2.0.1

alexeyts · 2018-01-03T17:40:50Z

Same issue here:
node: 8.9.0
node-loggly-bulk: 2.0.1

blyork · 2018-01-05T16:15:18Z

Same issue also:
node: 8.4.0
npm: 5.3.0

`-- winston-loggly-bulk@2.0.1
  +-- node-loggly-bulk@2.2.1
  | +-- json-stringify-safe@5.0.1 deduped
  | +-- moment@2.19.4
  | `-- request@2.81.0 deduped
  `-- winston@2.4.0 deduped

magland · 2018-01-05T21:00:48Z

Same issue. Using winston-loggly-bulk

amrshakya · 2018-01-06T00:16:31Z

Same issue. Using winston-loggly-bulk.

matteo-harpoon · 2018-01-08T10:57:24Z

Same issue. Using winston-loggly-bulk.
node: 7.9.0
npm: 5.0.3

winston: ^2.3.1
winston-loggly-bulk: ^2.0.1

backtrace:

node_modules/winston/node_modules/async/lib/async.js:38
node_modules/winston/lib/winston/logger.js:238
node_modules/winston-loggly-bulk/lib/winston-loggly.js:129 in logged
node_modules/node-loggly-bulk/lib/loggly/client.js:199
node_modules/node-loggly-bulk/lib/loggly/common.js:189 in Request._callback
node_modules/request/request.js:186 in Request.self.callback
events.js:106 in emitTwo
events.js:194 in Request.emit
node_modules/request/request.js:1163
events.js:96 in emitOne
events.js:191 in Request.emit
node_modules/request/request.js:1085
events.js:293 in Object.onceWrapper
events.js:91 in emitNone
events.js:188 in IncomingMessage.emit
_stream_readable.js:975 in endReadableNT
internal/process/next_tick.js:80 in _combinedTickCallback
internal/process/next_tick.js:128 in process._tickDomainCallback

blyork · 2018-01-08T17:58:10Z

I traced this issue down to loggly/node-loggly-bulk#31.

node-loggly-bulk's log function is calling its callback twice when the Loggly servers return a bad status code and then a success on a retry. This is causing a call to a async.forEach in winston.log to throw a error when transportLog calls next twice.

I am working on a fix.

mostlyjason · 2018-01-08T18:03:24Z

Awesome thanks @blyork! This has been hard to nail down. I look forward to the fix you mentioned.

CC @Shwetajain148

dguillamot · 2018-01-10T17:36:11Z

We are also running into this. Looking forward to an updated npm release with a fix! Thanks!

rushisan · 2018-01-11T19:28:45Z

any update on this issue? we are getting this problem every day several times, and it restarts our application, very annoying.

hodonsky · 2018-01-25T17:58:36Z

So close.... bump can we please give this PR a look?!

xolubi · 2018-02-02T17:17:24Z

Can't wait for this to get merged!

wlindley · 2018-02-15T19:30:02Z

We really need this one, too!

magland · 2018-02-15T19:39:04Z

I am waiting patiently, but cannot use loggly until this is fixed.

Samstiles · 2018-02-15T19:39:46Z

cc @mostlyjason

dmr-loggly · 2018-02-15T21:43:22Z

Hey everyone, we apologize for the delay in responding to this issue. I'm working right now on making sure it's brought to the right people's attention to be fixed ASAP.

Shwetajain148 · 2018-02-21T11:02:02Z

Hey everyone, Greetings! I have published a beta package version 2.0.4 of winston-loggly-bulk which contains the fix for this callback error and also improves the retry mechanism of the library on different errors.

Feel free to test the below beta package in your environment and let me know your feedback. I will release the package in production when you guys will confirm that everything works fine.

Install beta package using the below command:

npm install winston-loggly-bulk@beta

This beta package internally install node-loggly-bulk's beta package version 2.2.2.

cc: @mchaudhary

hodonsky · 2018-02-21T18:47:27Z

I will let you know how it goes. Thank you all!

Shwetajain148 · 2018-03-08T10:22:29Z

Hi @ianaya89 @jfitzpatrick13 @rushisan @alimvedats @alexeyts @blyork @magland @amrshakya @matteo-harpoon @dguillamot @Samstiles @relicmelex @xolubi..

Did you guys get some time to test the above beta package that I published which contains the fix for the callback error?

I want to listen from you so that I can go ahead and release the fix in production.

Comment Ref- #13 (comment)

hodonsky · 2018-03-08T18:04:50Z

Unfortunately I haven't had any slew of errors to trigger what would have been the bug. I'm doing some testing soon and will let you know if there are any negative effects. If you don't hear back from me in 10 days, assume all is well and it didn't get triggered, and I just forgot to come back here and tell you.

alexeyts · 2018-03-08T18:20:59Z

As this issue constantly happened in our production system and we did not succeed fixing it in a reasonable timeframe, we abandoned the use of this library.
We instead use syslog to report the logs.

Samstiles · 2018-03-08T18:30:09Z

Unfortunately we're in the same boat as @alexeyts :( We had no other choice and don't have the bandwidth now to swap back

davidknoll · 2018-03-08T18:47:28Z

I think mine was an isolated case, I haven't seen this crash in my app's console output again. -- Sent from my iPhone

magland · 2018-03-09T09:28:37Z

I plan to test it soon and will let you know. Thx for the fix

Shwetajain148 · 2018-03-15T10:55:12Z

Hi everyone! I have released the node-loggly-bulk's package version 2.2.2 which contains the fix for the callback error and also improves the error retry mechanism of the library. You can see the instructions here: https://www.loggly.com/docs/node-js-logs/ to setup nodejs logging and by running the command below-

npm install winston-loggly-bulk

Thanks!

duhseekoh · 2018-03-15T15:41:50Z

Giving this a go. FYI, if you're using yarn, you'll need to remove then add winston-loggly-bulk, otherwise yarn ain't gonna update the transitive node-loggly-bulk dep.

hodonsky · 2018-03-16T17:39:30Z

@Shwetajain148 this action rolled me back to winston-loggly-bulk@2.0.2 when the fix was in 2.0.4-beta. Is this intentional because the update is in a dependency instead?

duhseekoh · 2018-03-16T19:38:50Z

@relicmelex 2.0.2 is correct for winston-loggly-bulk. @Shwetajain148 made the fix in node-loggly-bulk which is a dependency of winston-loggly-bulk. So nothing in winston-loggly-bulk changes...

rushisan · 2018-03-21T15:23:58Z

Looks like node-loggly-bulk v2.2.2 did the trick. No errors for last 30 hours.

DJphilomath · 2018-03-21T18:56:10Z

I've also encountered this error. I'm still in my trial period and was not so happy that the library crashed my server. I'll see if this does the trick.

davidknoll · 2018-03-21T22:50:01Z

I can confirm the new version fixed it for me- since my last comment I've had several more instances in my logs of intermittent trouble connecting to Loggly and it's no longer crashing the app.

Shwetajain148 · 2018-03-28T12:48:52Z

Hi everyone, since this issue has been handled in node-loggly-bulk's 2.2.2 release, I am going ahead and closing this one.

Thanks!

jcasner · 2018-04-06T17:06:30Z

We upgraded to node-loggly-bulk@2.2.2 2 days ago and are now seeing even worse problems. We are seeing more frequent 502, 504, ETIMEDOUT, and ECONNRESET errors. While the retry logic seems to be firing, we're also noticing that the retried message doesn't appear in our Loggly logs, only the messages from node-loggly-bulk about retrying.

2018-04-06 09:15:41.522
{ json: { level: "info", message: "<SOURCE IP> - [06/Apr/2018:16:15:41 +0000] "GET / HTTP/1.1" 200 5223 "-" "SiteUptime" ", timestamp: "2018-04-06T16:15:41.522Z" }, http: { clientHost: "<HOST IP>", contentType: "application/json" } }
2018-04-06 09:15:00.948
{ json: { level: "info", message: "log #22 sent successfully after 3 retries", timestamp: "2018-04-06T16:15:00.948Z" }, http: { clientHost: "<HOST IP>", contentType: "application/json" } }
2018-04-06 09:14:56.915
{ json: { level: "info", message: "log #22 - Trying again in 4000[ms], attempt no. 3", timestamp: "2018-04-06T16:14:56.915Z" }, http: { clientHost: "<HOST IP>", contentType: "application/json" } }
2018-04-06 09:14:56.915
{ json: { level: "info", message: "log #22 - failed on error: Error: read ECONNRESET", timestamp: "2018-04-06T16:14:56.915Z" }, http: { clientHost: "<HOST IP>", contentType: "application/json" } }
2018-04-06 09:13:27.508
{ json: { level: "info", message: "<SOURCE IP> - - [06/Apr/2018:16:13:27 +0000] "GET / HTTP/1.1" 200 5223 "https://www.google.ca/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" ", timestamp: "2018-04-06T16:13:27.508Z" }, http: { clientHost: "<HOST IP>", contentType: "application/json" } }

Or, maybe the actual message is in there, but there's no way to tie back to the Log #. Up until this upgrade, we would get the occasional Callback already fired, but now we're experiencing the system crashes that others have reported. I'm not sure this issue is resolved...

DJphilomath · 2018-04-06T17:10:04Z

We have also been having a lot of connection errors after the upgrade

2018-04-06T17:07:27.635Z - Waking up ... will check
log #55 - failed on error: Error: read ECONNRESET
log #55 - Trying again in 2000[ms], attempt no. 2
2018-04-06T17:07:37.636Z - Waking up ... will check
log #55 - failed on error: Error: socket hang up
log #55 - Trying again in 4000[ms], attempt no. 3
log #59 - failed on error: Error: read ECONNRESET
log #59 - Trying again in 2000[ms], attempt no. 2
log #55 sent successfully after 3 retries
log #59 - failed on error: Error: read ECONNRESET
log #59 - Trying again in 4000[ms], attempt no. 3
2018-04-06T17:07:47.637Z - Waking up ... will check
2018-04-06T17:07:57.639Z - Waking up ... will check

hodonsky · 2018-04-06T17:16:07Z

I've been flooding my loggly pipe with load tests, and I haven't gotten a single one, can you copy in your implementation details as the error log it appears isn't super helpful as to 'why' it's happening here? @DJphilomath Also it looks like for you it's every 10 seconds pretty on the nose, do you have a timeout somewhere or a load balancer with connection draining?

davidknoll · 2018-04-06T17:19:19Z

I've been seeing a lot more errors on the console, but they're not crashing the app like before. Haven't actively been checking Loggly though. -- Sent from my iPhone

jcasner · 2018-04-06T18:25:55Z

I've dug a bit deeper. Seems the app isn't actually crashing, but the CPU is spiking, which is leading to autoscaling events, so the affected server gets rotated out. That was hard to diagnose because the logs aren't making it through to Loggly :/

I have filed a report with Loggly also, since it seems to be bursts of network issues. It seems like the issue is degraded network activity on their side which leads to this library getting overloaded with retries. Here are the times we've noticed degraded network performance from Loggly:

4/4/18 - 14:26 - 14:32 PST - 4694 errors
4/4/18 - 16:07 - 16:12 PST - 2648 errors 
4/4/18 - 20:24 - 20:29 PST - 19306 errors
4/6/18 - 02:05 - 02:35 PST - 34522 errors
4/6/18 - 08:40 - 08:45 PST - 4205 errors

Here's our winston setup:

import winston from 'winston';
import config from '../config';

require('winston-loggly-bulk');

const logger = new winston.Logger({
  level: config.logLevel,
  exitOnError: false,
});

module.exports.stream = {
  write: (message, encoding) => logger.info(message),
};

// Override the built-in console methods with winston hooks
switch (config.environment) {
  case 'dev-local':
    logger.add(winston.transports.Console, {
      handleExceptions: true,
      json: false,
      colorize: true,
      timestamp: true,
      prettyPrint: true,
    });
    break;
  default:
    logger.add(winston.transports.Loggly, {
      token: '<AppToken>',
      subdomain: '<subdomain>',
      tags: [`${config.environment}`],
      json: true,
      timestamp: true,
    });
    break;
}

console.log = (...args) => logger.info(...args);
console.info = (...args) => logger.info(...args);
console.warn = (...args) => logger.warn(...args);
console.error = (...args) => logger.error(...args);
console.debug = (...args) => logger.debug(...args);

This was referenced Aug 12, 2017

Clear timerFunctionForBufferedLogs if not needed loggly/node-loggly-bulk#14

Closed

Active handle/Interval prevents node script exit on Ubuntu 14.04 #12

Closed

Shwetajain148 mentioned this issue Aug 16, 2017

Clear all open interval and resolve callback error loggly/node-loggly-bulk#15

Merged

mostlyjason closed this Sep 22, 2017

mostlyjason reopened this Dec 13, 2017

Shwetajain148 mentioned this issue Jan 23, 2018

Resolve callback error and modified the entire retry logic loggly/node-loggly-bulk#34

Merged

10 of 10 tasks complete

dmr-loggly self-assigned this Feb 15, 2018

Shwetajain148 closed this Mar 28, 2018

loggly / winston-loggly-bulk

Join GitHub today

GitHub is where the world builds software

Setting isBulk to true, causes "Error: Callback was already called." #13

Setting isBulk to true, causes "Error: Callback was already called." #13

Comments

sloops77 commented Feb 26, 2017 • edited

sloops77 commented Feb 26, 2017

mostlyjason commented Feb 27, 2017

sloops77 commented Mar 15, 2017

mostlyjason commented Mar 15, 2017

reharik commented Jul 13, 2017

mostlyjason commented Sep 22, 2017

ianaya89 commented Dec 13, 2017

mostlyjason commented Dec 13, 2017

jfitzpatrick13 commented Dec 14, 2017 • edited

ianaya89 commented Dec 14, 2017

Shwetajain148 commented Dec 15, 2017

jfitzpatrick13 commented Dec 15, 2017

rushisan commented Dec 19, 2017 • edited

ianaya89 commented Dec 19, 2017

alimvedats commented Dec 20, 2017

alexeyts commented Jan 3, 2018

blyork commented Jan 5, 2018

magland commented Jan 5, 2018

amrshakya commented Jan 6, 2018

matteo-harpoon commented Jan 8, 2018

blyork commented Jan 8, 2018

mostlyjason commented Jan 8, 2018

dguillamot commented Jan 10, 2018

rushisan commented Jan 11, 2018

hodonsky commented Jan 25, 2018

xolubi commented Feb 2, 2018

wlindley commented Feb 15, 2018

magland commented Feb 15, 2018

Samstiles commented Feb 15, 2018

dmr-loggly commented Feb 15, 2018

Shwetajain148 commented Feb 21, 2018 • edited

hodonsky commented Feb 21, 2018

Shwetajain148 commented Mar 8, 2018

hodonsky commented Mar 8, 2018

alexeyts commented Mar 8, 2018

Samstiles commented Mar 8, 2018

davidknoll commented Mar 8, 2018

magland commented Mar 9, 2018

Shwetajain148 commented Mar 15, 2018

duhseekoh commented Mar 15, 2018

hodonsky commented Mar 16, 2018 • edited

duhseekoh commented Mar 16, 2018 • edited

rushisan commented Mar 21, 2018

DJphilomath commented Mar 21, 2018

davidknoll commented Mar 21, 2018

Shwetajain148 commented Mar 28, 2018

jcasner commented Apr 6, 2018

DJphilomath commented Apr 6, 2018 • edited

hodonsky commented Apr 6, 2018 • edited

davidknoll commented Apr 6, 2018

jcasner commented Apr 6, 2018

Essential cookies

Always active

Analytics cookies

sloops77 commented Feb 26, 2017 •

edited

jfitzpatrick13 commented Dec 14, 2017 •

edited

rushisan commented Dec 19, 2017 •

edited

Shwetajain148 commented Feb 21, 2018 •

edited

hodonsky commented Mar 16, 2018 •

edited

duhseekoh commented Mar 16, 2018 •

edited

DJphilomath commented Apr 6, 2018 •

edited

hodonsky commented Apr 6, 2018 •

edited