Unable to run any jobs - Cannot read property 'setEncoding' of undefined

[whiskerch]

Summary

I have created installed a new single server instance of Cronicle, however when I run any job it failes with the following message:

# Job ID: jjnmw30sl05
# Event Title: Test
# Hostname: ip-10-0-1-143.eu-west-2.compute.internal
# Date/Time: 2018/10/24 08:20:56 (GMT+0)

/opt/cronicle/node_modules/pixl-json-stream/json-stream.js:40
		this.streamIn.setEncoding('utf8');
		              ^

TypeError: Cannot read property 'setEncoding' of undefined
    at __construct.init (/opt/cronicle/node_modules/pixl-json-stream/json-stream.js:40:17)
    at new __construct (/opt/cronicle/node_modules/pixl-json-stream/json-stream.js:31:8)
    at __construct.<anonymous> (/opt/cronicle/bin/shell-plugin.js:32:16)
    at __construct.emit (events.js:182:13)
    at Socket.<anonymous> (/opt/cronicle/node_modules/pixl-json-stream/json-stream.js:75:12)
    at Socket.emit (events.js:182:13)
    at addChunk (_stream_readable.js:283:12)
    at readableAddChunk (_stream_readable.js:260:13)
    at Socket.Readable.push (_stream_readable.js:219:10)
    at Pipe.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

# Job failed at 2018/10/24 08:20:57 (GMT+0).
# Error: Child 27355 crashed with code: 1
# End of log.

Steps to reproduce the problem

I installed cronicle as per the installation documentation and tried to run a test job

Your Setup

As per installation instructions

Operating system and version?

Centos 7.5.1804

Node.js version?

10.12.0

Cronicle software version?

Are you using a multi-server setup, or just a single server?

Single server

Are you using the filesystem as back-end storage, or S3/Couchbase?

Filesystem

Can you reproduce the crash consistently?

Yes

Log Excerpts

# Job ID: jjnmw30sl05
# Event Title: Test
# Hostname: ip-10-0-1-143.eu-west-2.compute.internal
# Date/Time: 2018/10/24 08:20:56 (GMT+0)

/opt/cronicle/node_modules/pixl-json-stream/json-stream.js:40
		this.streamIn.setEncoding('utf8');
		              ^

TypeError: Cannot read property 'setEncoding' of undefined
    at __construct.init (/opt/cronicle/node_modules/pixl-json-stream/json-stream.js:40:17)
    at new __construct (/opt/cronicle/node_modules/pixl-json-stream/json-stream.js:31:8)
    at __construct.<anonymous> (/opt/cronicle/bin/shell-plugin.js:32:16)
    at __construct.emit (events.js:182:13)
    at Socket.<anonymous> (/opt/cronicle/node_modules/pixl-json-stream/json-stream.js:75:12)
    at Socket.emit (events.js:182:13)
    at addChunk (_stream_readable.js:283:12)
    at readableAddChunk (_stream_readable.js:260:13)
    at Socket.Readable.push (_stream_readable.js:219:10)
    at Pipe.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

# Job failed at 2018/10/24 08:20:57 (GMT+0).
# Error: Child 27355 crashed with code: 1
# End of log.```
<!-- Please provide a crash log, and/or any other log excerpts you consider relevant. -->

<!-- Thank you for helping to make Cronicle better! -->

[jhuckaby]

Thank you for this issue report. I have never seen anything like this before, and cannot fathom how it is possibly crashing at that point in the code. That is completely bizarre.

From the stack trace it looks like you are using the Shell Plugin. Can you share the shell script code you are trying to run? I am trying to determine if there is something weird about your shell syntax that is causing a crash at this spot.

Thanks.

[whiskerch]

Hi

The script I run is:

#!/bin/sh

# Enter your shell script code here
hostname

I am running a docker version on my mac and it works great - with the same script

The AWS EC2 instance I am running Centos on, has been hardened, but node had raised no errors while installing, nor did Cronicle as far as I can see

[jhuckaby]

That really makes no sense. That's the world's simplest shell script 😆

It's so strange, because I use both AWS and CentOS. I've been running Cronicle on AWS instances with CentOS 7 in live production for 3 years, and I've never seen anything like this. It's so odd.

It looks like something has gone haywire with the STDIN / STDOUT pipes between the two processes. Cronicle does a child process exec to the Shell Plugin script, and it is somehow not able to connect up the pipes. I'm really scratching my head on this one.

The AWS EC2 instance I am running Centos on, has been hardened

What do you mean by "hardened" may I ask?

[whiskerch]

As part of the build I run this script this ansible script on the server: https://github.com/MindPointGroup/RHEL7-CIS

It locks down a whole load of settings on the server, there is a good chance this is what is causing the problem, however going through the settings it applies, I cannot see where or how it is causing the problem

Chris

[jhuckaby]

Ah, interesting. I've never used such a hardening script before, so I don't know what this is doing. However, I could take a guess...

The way the Cronicle Shell Plugin works is, it takes your raw shell code from the UI, and writes it to a temporary file (the temp dir it uses is whatever Node's os.tmpdir() returns, usually /tmp or something like that). It then sets this temp file to have execute permissions (0755), and then runs it. I'll bet your Ansible hardening script does something to prevent any executables from running out of the server's temp directory.

[whiskerch]

That'll be it - my /tmp directory is setup to prevent any code being executed in it.

I'll look into relaxing that rule
Thanks!

[AlexAbraham1]

Is it possible to point the cronicle script to use a directory other than /tmp?

Thanks!

[jhuckaby]

I believe you can set an environment variable to do this. Cronicle queries the Node.js os.tmpdir() function to get the temp dir path, and I believe that just reads it from the TMPDIR global root environment variable (I think -- not sure on this). So try setting that variable maybe?

You might be able to specify it in the Cronicle configuration file here: job_env

[danwild]

Sorry for digging up an old thread - but I just bumped into this error (which looks the same as #220 ), and it turns out the issue was that my custom plugin files had insufficient permissions for cronicle to read.

[jhuckaby]

@danwild Thank you for letting me know about this detail. I will try to reproduce this and come up with a more graceful error condition in a future release.