Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Add RPC via stderr.

  • Loading branch information...
commit 941eb9bb998bbc6a11d8e53ab8957b4d1a18b836 1 parent 615e5ca
@seven1m authored
Showing with 100 additions and 148 deletions.
  1. +79 −145 README.md
  2. +4 −1 app/models/run.coffee
  3. +16 −2 lib/script.coffee
  4. +1 −0  package.json
View
224 README.md
@@ -53,206 +53,140 @@ To start up the web server, run:
coffee bin/web
-<a name="job-api"></a>
-## Job API
+## Job Scripts
-Each job runs in a separate Node.js process with limited context, which has the effect of sandboxing the running code from the parent worker process.
+A job is simply a script on the file system plus some metadata including schedule (cron), hooks, target worker, etc.
-The idea is to make it difficult for someone to do naughty things to your worker process, the host machine, and your network, but **I explicitly disclaim the secureness of this system -- In fact, I recommend you NOT make this system available via the public web, and that you restrict access with .htaccess or some other technique (authentication/authorization is not yet present).**
+A script can be written in any language, e.g. bash, ruby, python, etc., though CoffeeScript is recommended in order to utilize the existing Mongoose Queue model.
-The following functions are available to your running code:
+Here's perhaps the simplest possible script:
-### done
+```bash
+#!/usr/bin/env bash
-Call `done()` at the end of every job so the db connections can be cleaned up, otherwise your job may be marked as failed.
-
-### emit
-
-*Arguments:*
-
-* event
-* data
-
-Emits an event that other jobs can watch, consequently allowing one job to trigger another job. Data passed as the second argument is available to any triggered jobs as the `data` variable.
+echo 'done!'
+```
-### progress
+### Updating Jobs
-*Arguments:*
+When you setup Sooner.io under the Configuration section above, you set up a bare git repository living inside the Sooner.io directory called `scripts.git`, along with a cloned copy in `scripts-working-copy`.
-* current
-* max (optional, defaults to 100)
+This is designed so that you can push directly to the scripts repo via ssh from your local workstation. Doing so will trigger the post-update git hook and update the working copy.
-If you wish to track incremental progress of your job, you may call, e.g. `progress(5, 10)` (this will show a progress bar at half-way). The first argument is the current number of units of work complete, while the second is the total number of units of work. The second argument is optional and can be used to change the maximum on the fly.
+First, on your local machine, clone the scripts repo:
-### queue
+ git clone username@server:/var/www/apps/sooner.io/scripts.git sooner-scripts
-*Arguments:*
+You will initially see an empty directory. Create your script there, mark it as executable (required), and commit it. Then push:
-* name
+ vim myScript.coffee
+ chmod +x myScript.coffee
+ git add myScript.coffee
+ git commit -m "Add myScript"
+ git push origin master
-Returns the [Mongoose](http://mongoosejs.com/) object for the named queue collection.
+The output you see next should indicate the creation of a new job. Next you should visit the web interface and enable the job (all new jobs are disabled by default) and configure the other settings as desired.
-See the [Querying](http://mongoosejs.com/docs/query.html) and [Updating](http://mongoosejs.com/docs/updating-documents.html) docs for help.
+An alternative means of updating scripts is to work directly on the server (not optimal, but handy when testing small changes):
-Each entry in the queue has the following fields defined:
+ cd /var/www/apps/sooner.io/scripts-working-copy
+ vim myScript.coffee
+ coffee ../bin/deployer.coffee update
-* `_id`
-* `status`
-* `data`
-* `createdAt`
+Just be sure to and and commit your change once finished and `git push`.
-You should only set the `status` and `data` fields yourself.
+### API
-### db.connect
+There is something of an API that scripts can utilize to tell Sooner.io to do specific things. Namely, there are two methods called `progress` and `emit` (documented below).
-*Arguments:*
+Making calls to those methods is done via RPC over one of two channels:
-* connectionName
-* callback
+1. via [dnode](https://github.com/substack/dnode) which has library support for Node.js, Perl, Ruby, PHP, and Java.
+2. via stderr output (low-tech, but should work with just about any language)
-*Example:*
+#### API calls via dnode
-Connects to a named database (PostgreSQL only at the moment).
+Here is an example of using dnode from within CoffeeScript:
```coffeescript
-db.connect 'foo', (conn) ->
- # use conn here
+#!/usr/bin/env coffee
+
+dnode = require 'dnode'
+dnode.connect process.argv[2], (remote, conn) ->
+ console.log 'half way'
+ remote.progress 50
+ done = ->
+ console.log 'done!'
+ conn.end()
+ setTimeout done, 2000
```
-The `connectionName` is a named connection provided in `config.json` under `dbConnections`. Following is an example connection called "foo":
+A few things to note:
-```json
-{
- "dbConnections": {
- "foo": "postgres://postgres@localhost/foo"
- }
-}
-```
+* The first argument passed to your script is the unix socket that dnode can use to communicate with the parent process. This socket is automatically cleaned up once your script is finished executing.
+* Speaking of "finished", you will need to call `conn.end()` once you are finished working in order to close the dnode socket connection.
-### conn.query
+#### API calls via stderr
-*Arguments:*
+A lower-tech way of communicating is to echo to `stderr`, which can be done in bash with `echo "foo" 1>&2`. Since stderr may additionally be used for other messages, you must prefix your RPC call with `>>> sooner.` so that Sooner.io knows what you mean. This works from bash like so:
-* sql
-* params (optional)
-* callback
+```bash
+#!/usr/bin/env bash
-*Example:*
+echo 'half way'
+echo '>>> sooner.progress(50)' 1>&2
+echo '>>> sooner.emit("foo-event")' 1>&2
+sleep 2
-```coffeescript
-db.connect 'foo', (conn) ->
- conn.query 'select * from foo where bar=$1', ['baz'], (rows) ->
- # use rows here
+echo 'done!'
```
-### conn.end
-
-*No arguments*
-
-Closes the database connection.
-
-### shell.spawn
+#### emit
*Arguments:*
-* commandName
-* args (array)
-
-Returns a [ChildProcess](http://nodejs.org/api/child_process.html).
-
-The `commandName` is a named shell command provided in `config.json` under `shellCommands`. Following is an example named shell command for "ls":
+* event
+* data
-```json
-{
- "shellCommands": {
- "listDir": "ls"
- }
-}
-```
+Emits an event that other jobs can watch, consequently allowing one job to trigger another job. Data passed as the second argument is available to any triggered jobs as the `data` variable.
-### shell.run
+#### progress
*Arguments:*
-* commandName
-* args (array)
-* callback
-
-This is a higher level function that executes spawn, then waits for the process to finish. `stdout` and `stderr` are both captured to a single string, then passed via the `callback` function.
-
-*Example:*
-
-```coffeescript
-db.run 'listDir', ['/tmp'], (code, output) ->
- # code = return code of the completed process
- # output = stdout+stderr
-```
-
-### ftp.connect
+* current
+* max (optional, defaults to 100)
-*Arguments:*
+If you wish to track incremental progress of your job, you may call, e.g. `progress(5, 10)` (this will show a progress bar at half-way). The first argument is the current number of units of work complete, while the second is the total number of units of work. The second argument is optional and can be used to change the maximum on the fly.
-* connectionName
-* callback
+### Queue Access
-This is a light wrapper around [node-ftp](https://github.com/mscdex/node-ftp).
+The Queue model (available in `app/models/queue.coffee`) is something you can use to store and retrieve work to be done. Queues are browsable, filterable, and sortable via the web interface, so they are great for keeping track of work done and/or to-be-done.
-`callback` is passed an FTPConnection object with the following methods:
+Here's how you would access a Queue from a CoffeeScript script:
-* `list(path, callback)`
-* `mkdir(name, callback)`
-* `put(inSTream, filename, callback)`
-* `get(filename, callback)`
-* `rename(oldFilename, newFilename, callback)`
-* `end()`
+```coffeescript
+queue = require __dirname + '/../app/models/queue'
-Setup FTP server connection details in `config.json`:
+q = queue('profiles')
-```json
-{
- "ftpServers": {
- "foo": {
- "host": "ftp.example.com",
- "username": "user",
- "password": "secret"
- }
- }
-}
+q.where('status', 'pending').run (err, profiles) ->
+ # do work here
```
-### fs.readStream
-
-*Arguments:*
-
-* path
+Essentially, the `queue` function returns a [Mongoose](http://mongoosejs.com/) model attached to a similarly-named MongoDB collection (prepended with `queue_`) to which you are free to query, insert, update, and delete.
-Returns an opened read stream. See the Node.js [documentation](http://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options).
+See the [Querying](http://mongoosejs.com/docs/query.html) and [Updating](http://mongoosejs.com/docs/updating-documents.html) Mongoose docs for help.
-### fs.writeStream
-
-*Arguments:*
-
-* path
-
-Returns an opened write stream. See the Node.js [documentation](http://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options).
-
-### xml.stringToJSON
-
-*Arguments:*
-
-* string
-* callback
-
-Converts XML in string into a JSON object and runs `callback(err, json)`.
-
-### xml.fileToJSON
-
-*Arguments:*
+Each entry in the queue has the following fields defined:
-* path
-* callback
+* `_id`
+* `status`
+* `data`
+* `createdAt`
+* `updatedAt`
-Reads file contents from path and converts XML into a JSON object and runs `callback(err, json)`.
+You should only set the `status` and `data` fields yourself. You should store an object in the `data` attribute with as much information as you need to track your work. If you manage to keep the data object flat (only a single-layer JS object), all the attributes will be easily visible via the web interface in table form (though it is indeed possible to store nested objects as well)
## License
View
5 app/models/run.coffee
@@ -94,7 +94,10 @@ schema.methods.run = (callback) ->
if err or !job then throw ['error getting job:', err]
script = new Script @fullPath(),
- progress: console.log
+ emit: (event, data) =>
+ console.log "job #{@name} emitted '#{event}' with data", data
+ _.debounce(GLOBAL.hook.emit, 50, true)(event, data)
+ progress: _.debounce(_.bind(@setProgress, @), 50, true)
# FIXME: race condition
models.run.where('status', 'busy').where('jobId', @jobId).count (err, runningCount) =>
View
18 lib/script.coffee
@@ -2,10 +2,13 @@ childProcess = require 'child_process'
fs = require 'fs'
temp = require 'temp'
dnode = require 'dnode'
+carrier = require 'carrier'
EventEmitter2 = require('eventemitter2').EventEmitter2
class Script extends EventEmitter2
+ RPCRE: /(^|\n)\s*>>>\s*(sooner\.[a-zA-Z0-9_]+\(.*\))/
+
constructor: (@path, @funcs) ->
if @path.trim() != ''
try
@@ -28,14 +31,25 @@ class Script extends EventEmitter2
if @realPath
input = JSON.stringify(data || {})
child = childProcess.spawn @realPath, [@sockPath, input], {}
+
@emit 'start', child.pid.toString()
+
child.stdout.on 'data', (data) =>
@emit 'data', data.toString()
- child.stderr.on 'data', (data) =>
- @emit 'data', data.toString()
+
+ # intercept side channel rpc
+ carrier.carry child.stderr, (line) =>
+ line = line.toString()
+ if m = line.match(@RPCRE)
+ sooner = @funcs
+ eval m[2]
+ else
+ @emit 'data', line
+
child.on 'exit', (code) =>
@closeSocket()
@emit 'end', code
+
else
@emit 'error', 'could not find path'
View
1  package.json
@@ -31,6 +31,7 @@
, "mongodb": "latest"
, "dnode": "0.9.x"
, "temp": "0.4.x"
+ , "carrier": "0.1.7"
}
, "devDependencies": {
"jasmine-node": "latest"
Please sign in to comment.
Something went wrong with that request. Please try again.