Logdna agent 2.0 breaks for on-prem installs

[darakian]

Hey all,

Just got through an incident where logs were no longer arriving for an "on-prem " install of your service. The issue is something to do with the 2.0 release which we install through your repos. We were seeing logs of the sort

[error] { Error: An error occured while sending logs to the cloud.
    at process.nextTick (/usr/bin/node_modules/@logdna/logger/lib/logger.js:719:25)
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)
  meta:
   { actual: 'Request failed with status code 403',
     code: 403,
     firstLine: 'Oct  9 17:25:47 ip-172-24-4-240 polkitd[1056]: Registered Authentication Agent for unix-process:27029:844371184 (system bus name :1.14602 [/usr/bin/pkttyagent --notify-fd 6 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)',
     lastLine: '2020-10-09 17:25:47 0000 [info] got SIGTERM signal, shutting down...',
     retrying: false,
     attempts: 1 } }

with the 2.0 agent. After downgrading back to 1.6.5 we see

2020-10-09 17:43:46 0000 [info] logdna-agent/1.6.5 started on some_host (some_ip)
2020-10-09 17:43:46 0000 [info] streaming /var/log/: 49 file(s)
2020-10-09 17:43:49 0000 [info] got SIGTERM signal, shutting down...
2020-10-09 17:43:50 0000 [info] logdna-agent/1.6.5 started on some_host (some_ip)
2020-10-09 17:43:50 0000 [info] streaming /var/log/: 49 file(s)
2020-10-09 17:43:56 0000 [info] got SIGTERM signal, shutting down...
2020-10-09 17:43:56 0000 [info] logdna-agent/1.6.5 started on some_host (some_ip)
2020-10-09 17:43:56 0000 [info] streaming /var/log/: 49 file(s)

This seems to be the case across all rhel/debian based hosts. Our logs are sent to a domain of the form a.b.c.d which differs from your regular logs.logdna.com path. Perhaps that's the issue.

[darakian]

Perhaps the removal of the LOGDNA_LOGHOST variable here

46875a1

Just a note, it would be really nice to see the host being used in the event of failure in the agent logs.

[darinspivey]

Thanks for the suggestion, @darakian. We will be fixing the environment variables to be the way they were before. Your suggestion to have the host as part of the error is a good one! That change would go in @logdna/logger package. Feel free to open an issue in that repository and we'll get it in shortly. That being said, the logger only has a full URL (not the individual parts of host, endpoint, etc), however we can provide that URL in its entirety as part of those errors.