No logs in Papertrail?

[sirjonathan]

I activated and configured the plugin on a personal site to test it out. So far, I'm not seeing any logs sent to Papertrail. Here are the steps I've taken:

1. Installed and activated the plugin.
2. Created a new account on Papertrail.
3. Updated the "Papertrail" settings page in Stream to include my "Destination".
4. Confirmed that new actions are logged to Stream.
5. Refreshed the Papertrail dashboard.

So far, nothing.

Am I missing something obvious? Is there more information I can provide?

[Japh]

Hmm... a screenshot of your Stream to Papertrail settings screen would be great. Also, if you could check your PHP error log for anything that might be relevant.

[troy]

What @Japh said. My first thought would be a subtle misconfiguration or typo somewhere, and then a firewall rule that drops the log messages. This doc has a few troubleshooting ideas. The ideas are geared for system-level loggers, but the telnet reachability test and/or tcpdump transmission test may help.

If it isn't specific to stream-papertrail, feel free to contact support@ Papertrail's domain.

[sirjonathan]

Here's a screenshot: https://cloudup.com/c_HFMTlcw8r

And here's the full value of the "Destination" field: logs3.papertrailapp.com:11757

I don't have any firewall plugins installed. My hosting provider is RackSpace Cloud Sites.

Here's what I found in the error log:

PHP Warning: socket_sendto() [<a href='function.socket-sendto'>function.socket-sendto</a>]: unable to write to socket [1]: Operation not permitted in /mnt/target03/355327/355349/www.jonathanwold.com/web/content/blog/wp-content/plugins/stream-to-papertrail/inc/class-stream-papertrail-api.php on line 98

[Japh]

Thanks, @sirjonathan, I had a sneaking suspicion that might've been the problem. I'll look into it!

[troy]

I wonder what restriction Rackspace Cloud Servers has to prohibit this. I know SOCK_RAW (more) requires root, but I don't think 'SOCK_DGRAM` (ie, UDP) should, though I'm not certain of that. I could imagine an SELinux policy forbidding outbound UDP packets, but I don't know what return value/code it would manifest as.

[Japh]

Sounds like perhaps some hosts only have certain ports open, and primarily for TCP only. (80, 21, 443, maybe 22.)

Wonder if there's another way to log without using UDP...

[Japh]

@sirjonathan Would you mind asking Rackspace Cloud Sites support if they can look into this for you? They may be willing to open the port for UDP traffic even.

Short of that, the only other alternative seems to be setting up a proxy service that the plugin could make HTTP requests to, and which will forward those on as UDP to Papertrail.

Perhaps I could make an alternative Loggly plugin for people who aren't able to use Papertrail... only trouble is, it's pricey to get Loggly to archive to S3.

[Japh]

I may need to simply make a note in the plugin's description that if you come across this error, then sadly this plugin won't work for you 😞

[troy]

proxy service that the plugin could make HTTP requests to

The reason Papertrail doesn't support HTTP is the reason that this probably shouldn't either: without a fair amount of logic on the client (Stream) side, it blocks the app. Here's more.

It's not impossible to implement log submission using reliable delivery without blocking, but it's fairly tough. The sender can't just transmit the logs directly, since that write and ack will often take more time than is acceptable for the browser-facing request (and could time out entirely). It needs to queue them somewhere and then drain that queue independently of the browser-facing request. Also, even the queue operations need to have timeouts enforced, since the queue itself may not respond fast enough. Finally, something needs to consider whether the socket has timed out and needs to reconnect, ensure it's only sending one of each queued item when the connection is re-established, and decide what to do when the queue fills up.

Suddenly what looks like a simple log sender becomes its own log framework, and one with fairly complex tradeoffs for implementors to consider, and tradeoffs which only emerge in certain situations (high-volume logging, destination unreachable, etc).

Here's an example of what happens when logging is unintentionally blocking. In their case, that was the syslog() call to a local pipe serviced by rsyslog, then draining to a TCP destination. Same underlying problem. (Note that rsyslog has implemented the aspects above - queueing, etc - but a simpler sender like this one wouldn't.)

[Japh]

Prior to switching from Loggly to Papertrail, I had it working with Loggly in a non-blocking fashion over HTTP.

WordPress' built in remote request function allows you to specify that it needs to be non-blocking.