Skip to content
Browse files

Added:

- disk storage to reference section
- connecting to databases
- logging
- webhooks
- packaging / how to write client code
  • Loading branch information...
1 parent 2d4392f commit 61d3ddf67ae9437cf30715ecb7f5ad520c7caf90 @treeder treeder committed Apr 4, 2012
View
2 README.md
@@ -1,7 +1,7 @@
# Installation
1. Install jekyll gem: `sudo gem install jekyll`
-2. Install Pygments: `sudo easy_install Pygments`
+2. Install Pygments: `sudo easy_install pygments`
2. clone this repo.
Detailed instruction for different environments: https://github.com/mojombo/jekyll/wiki/install
View
1 _config.yml
@@ -3,3 +3,4 @@ auto: true
server: true
domain: "dev.iron.io"
google_cse_id: 012672112220120861281:u1mwwky5lri
+pygments: true
View
4 _includes/sidemenu_worker.html
@@ -8,6 +8,9 @@
<li><a {% if page.url == "/worker/reference/api/index.html" %}class="selected"{% endif %} href="/worker/reference/api">REST/HTTP API</a></li>
<li><a {% if page.url == "/worker/reference/environment/index.html" %}class="selected"{% endif %} href="/worker/reference/environment">System Environment</a></li>
<li><a {% if page.url == "/worker/reference/security/index.html" %}class="selected"{% endif %} href="/worker/reference/security">Securing Your Workers</a></li>
+ <li><a {% if page.url == "/worker/reference/disk-storage" %}class="selected"{% endif %} href="/worker/reference/disk-storage">Disk Storage</a></li>
+ <li><a {% if page.url == "/worker/reference/logging" %}class="selected"{% endif %} href="/worker/reference/logging">Logging</a></li>
+ <li><a {% if page.url == "/worker/reference/webhooks" %}class="selected"{% endif %} href="/worker/reference/webhooks">Webhooks</a></li>
</ul>
<h4>Client Libraries</h4>
@@ -23,5 +26,4 @@
<li><a {% if page.url == "/worker/articles/integrations/index.html" %}class="selected" {%else %}{% if page.categories contains "integrations" %}class="selected"{% endif %}{% endif %} href="/worker/articles/integrations">Integrations</a></li>
<li><a {% if page.url == "/worker/articles/data/index.html" %}class="selected"{% else %}{% if page.categories contains "data" %}class="selected"{% endif %}{% endif %} href="/worker/articles/data">Data &amp; Databases</a></li>
<li><a {% if page.url == "/worker/articles/platforms/index.html" %}class="selected"{% else %}{% if page.categories contains "platforms" %}class="selected"{% endif %}{% endif %} href="/worker/articles/platforms">Platforms</a></li>
- <li><a {% if page.url == "/worker/articles/languages/index.html" %}class="selected"{% else %}{% if page.categories contains "languages" %}class="selected"{% endif %}{% endif %} href="/worker/articles/languages">Languages</a></li>
</ul>
View
73 _posts/2012-03-11-worker-data-using-local-storage.md
@@ -13,75 +13,4 @@ categories:
- data
---
-# Using Local Disk Storage
-
-Workers can make use of a [large amount](/worker/reference/environment/) of local temporary storage space that's dedicated on a per-worker basis. You can perform almost any file operations with it that you could within a local environment.
-
-You access this storage by making use of the variable `user_dir` in the worker. This variable contains the path of the directory your worker has write access to.
-
-## Saving Files to Disk
-
-Here's an example that downloads a file from the web and saves it in local storage. The log snippet just logs the contents of `user_dir`.
-
-{% highlight ruby %}
-class S3Worker < IronWorker::Base
-
- filepath = user_dir + "ironman.jpg"
- File.open(filepath, 'wb') do |fo|
- fo.write open("http://www.iron.io/assets/banner-mq-ironio-robot.png").read
- end
-
- user_files = %x[ls #{user_dir.inspect}]
- log "\nLocal Temporary Storage ('user_dir')"
- log "#{user_files}"
-
-end
-{% endhighlight ruby %}
-
-## Location of Uploaded Files and Folders
-
-The `user_dir` directory also contains any uploaded files that you've included with your code. Note that any folders or nested files will appear at the top level.
-
-For example, let's say you upload a file with the following structure:
-
-{% highlight ruby %}
-merge "../site_stats/client.rb"
-{% endhighlight ruby %}
-
-This file will be placed in the `user_dir` directory. You can make use of it there, create local/remote path references (using the local/remote query switch in your worker), or replicate the path and move the file there. (We recommend one of the first two options.)
-
-<pre class="grey-box fixed-width">
-user_dir/
- ...
- client.rb
- ...
-</pre>
-
-In Ruby, to make use of the file (in the case of a code file), you would use a `require_relative` statement with the base path.
-
-{% highlight ruby %}
- require_relative './client'
-{% endhighlight ruby %}
-
-## Use Cases
-
-Typical use cases might include:
-
-* Downloading a large product catalog or log file, parsing it, processing the data, and inserting the new data into a database
-* Downloading an image from S3, modifying it, and re-uploading it,
-* Splitting up a large video file or the results of a website crawl in S3, then creating and queuing multiple workers to process each video or page slice.
-
-##Best Practices
-
-This is temporary storage and only available while the worker is running. You'll want to make use of databases and object stores to persist any data the worker produces.
-
-We recommend that you not pass any large data objects or data files in to workers, but instead use object storage solutions like [AWS S3](http://aws.amazon.com/s3/) or databases. To do this, just upload your data to S3 or store it in the database from your app, then pass the **identifier of the object** to the worker. The worker can then access the data from the data store. This is more efficient in terms of worker management and better for exception handling.
-
-## Examples
-
-You can find more examples of making use of local disk storage here:
-
-* [Image Processing Example](https://github.com/iron-io/iron_worker_examples/blob/master/ruby/Image_processing/image_processor.rb) on Github
-* [Image Processing Example with Carrierwave](https://github.com/iron-io/iron_worker_examples/tree/master/ruby/carrierwave) on Github
-* [S3 Example](https://github.com/iron-io/iron_worker_examples/blob/master/ruby/s3_tester/s3_worker.rb) on Github
-* [S3 Example 2](https://github.com/iron-io/iron_worker_examples/tree/master/ruby/download_to_s3) on Github
+This article has [moved to here](/worker/reference/disk-storage/).
View
30 _posts/2012-04-04-worker-databases.md
@@ -0,0 +1,30 @@
+---
+title: Connecting to Databases
+layout: default
+section: worker
+permalink: /worker/articles/databases
+breadcrumbs:
+ - ['Articles', '/articles']
+ - ['Working With Data & Databases', '/data']
+ - ['Connecting to Databases', '/databases']
+categories:
+ - worker
+ - articles
+ - data
+---
+
+# Connecting to your Database
+
+Connecting to your database from a worker is exactly the same as connecting to it from your application. The thing to
+be aware of is that you must make the connection to your database each time your worker runs.
+
+A typical example in pseudo code:
+
+{% highlight ruby %}
+# Connect to my database
+db = connect_to_db()
+# Now do some work
+results = db.find(...)
+{% endhighlight %}
+
+There's nothing really special to it, just make sure you create the connection explicitly in your worker.
View
104 worker/code/packaging/index.md
@@ -0,0 +1,104 @@
+---
+title: Packaging
+layout: default
+section: worker
+breadcrumbs:
+ - ['Code', '/code']
+ - ['Packaging', '/packaging']
+---
+
+# Packaging Code for IronWorker
+
+This section will explain how to develop client code to upload to this endpoint. The examples are in Ruby,
+but would apply to any language.
+
+Let's start with an extremely simple example, let's say all we wanted to do was run:
+
+{% highlight ruby %}
+puts "Hello World!"
+{% endhighlight %}
+
+Put that in a file called hello.rb.
+
+## Loading the Task Data Payload
+
+When your code is executed on IronWorker, it will be given six program arguments (3 keys and 3 values).
+
+### Program Arguments
+
+-id: The task id.
+-payload: the filename containing the data payload for this particular task.
+-d: the user writable directory that can be used while running your job.
+
+Here is a ruby example of obtaining this information (this is from the simple_worker ruby gem):
+
+{% highlight ruby %}
+payload_file = nil
+task_id = nil
+ARGV.each do |arg|
+ if arg == \"-d\"
+ dirname = ARGV[i+1]
+ end
+ if arg == \"-id\"
+ task_id = ARGV[i+1]
+ end
+ if arg == \"-payload\"
+ payload_file = ARGV[i+1]
+ end
+ i+=1
+end
+
+# Now load in job data
+require 'json'
+payload = JSON.load(File.open(payload_file))
+{% endhighlight %}
+
+## Packaging Code
+
+test
+
+Code must be submitted as a zip file containing all of the necessary files to execute.
+
+{% highlight ruby %}
+# this method will create a zip file given an array of file names.
+def package_code(files)
+ fname = "package.zip"
+ Zip::ZipFile.open(fname, 'w') do |f|
+ files.each do |file|
+ f.add(file, file)
+ end
+ end
+ fname
+end
+
+zip_filename = package_code(\['hello.rb'\])
+{% endhighlight %}
+
+So now we have a zip file containing the code we want to execute on IronWorker.
+
+## Uploading Code
+
+Now we need to upload this package to IronWorker so we can start using it for our tasks.
+For this, we use the Tasks Upload method of the API. In this example, we'll use the Ruby gem to make it easier.
+
+{% highlight ruby %}
+@sw.upload_code('hello', zip_filename, 'hello.rb', :runtime=>'ruby')
+{% endhighlight %}
+
+This is a method that exists in the iron_worker gem already that takes a name ('hello'), the zip filename from the
+packaging step above, the name of the file to execute ('hello.rb') and the runtime to execute it with.
+
+Now that it's uploaded, we can queue up a job to use it.
+
+## Running a Job Using Your Code
+
+Now we simply queue up a task that uses our 'hello' code.
+
+{% highlight ruby %}
+@sw.queue('hello')
+{% endhighlight %}
+
+## Conclusion
+
+That's the very basics, but should give you an idea of how to get started.
+
View
3 worker/reference/api/index.md
@@ -248,6 +248,9 @@ Sample:
### Upload a Code Package
+You will almost always want to use one of our [client libraries](/worker/code) in the language of your choice to make
+this easier. And if you want to make your own client, please read [Packaging Code for IronWorker](/worker/code/packaging).
+
#### Endpoint
POST /projects/<span class="variable project_id">{Project ID}</span>/codes
View
81 worker/reference/disk-storage/index.md
@@ -0,0 +1,81 @@
+---
+title: Disk Storage
+layout: default
+section: worker
+breadcrumbs:
+ - ['Reference', '/reference']
+ - ['Disk Storage', '/disk-storage']
+---
+
+# Using Local Disk Storage
+
+Workers can make use of a [large amount](/worker/reference/environment/) of local temporary storage space that's dedicated on a per-worker basis. You can perform almost any file operations with it that you could within a local environment.
+
+You access this storage by making use of the variable `user_dir` in the worker. This variable contains the path of the directory your worker has write access to.
+
+## Saving Files to Disk
+
+Here's an example that downloads a file from the web and saves it in local storage. The log snippet just logs the contents of `user_dir`.
+
+{% highlight ruby %}
+class S3Worker < IronWorker::Base
+
+ filepath = user_dir + "ironman.jpg"
+ File.open(filepath, 'wb') do |fo|
+ fo.write open("http://www.iron.io/assets/banner-mq-ironio-robot.png").read
+ end
+
+ user_files = %x[ls #{user_dir.inspect}]
+ log "\nLocal Temporary Storage ('user_dir')"
+ log "#{user_files}"
+
+end
+{% endhighlight ruby %}
+
+## Location of Uploaded Files and Folders
+
+The `user_dir` directory also contains any uploaded files that you've included with your code. Note that any folders or nested files will appear at the top level.
+
+For example, let's say you upload a file with the following structure:
+
+{% highlight ruby %}
+merge "../site_stats/client.rb"
+{% endhighlight ruby %}
+
+This file will be placed in the `user_dir` directory. You can make use of it there, create local/remote path references (using the local/remote query switch in your worker), or replicate the path and move the file there. (We recommend one of the first two options.)
+
+<pre class="grey-box fixed-width">
+user_dir/
+ ...
+ client.rb
+ ...
+</pre>
+
+In Ruby, to make use of the file (in the case of a code file), you would use a `require_relative` statement with the base path.
+
+{% highlight ruby %}
+ require_relative './client'
+{% endhighlight ruby %}
+
+## Use Cases
+
+Typical use cases might include:
+
+* Downloading a large product catalog or log file, parsing it, processing the data, and inserting the new data into a database
+* Downloading an image from S3, modifying it, and re-uploading it,
+* Splitting up a large video file or the results of a website crawl in S3, then creating and queuing multiple workers to process each video or page slice.
+
+##Best Practices
+
+This is temporary storage and only available while the worker is running. You'll want to make use of databases and object stores to persist any data the worker produces.
+
+We recommend that you not pass any large data objects or data files in to workers, but instead use object storage solutions like [AWS S3](http://aws.amazon.com/s3/) or databases. To do this, just upload your data to S3 or store it in the database from your app, then pass the **identifier of the object** to the worker. The worker can then access the data from the data store. This is more efficient in terms of worker management and better for exception handling.
+
+## Examples
+
+You can find more examples of making use of local disk storage here:
+
+* [Image Processing Example](https://github.com/iron-io/iron_worker_examples/blob/master/ruby/Image_processing/image_processor.rb) on Github
+* [Image Processing Example with Carrierwave](https://github.com/iron-io/iron_worker_examples/tree/master/ruby/carrierwave) on Github
+* [S3 Example](https://github.com/iron-io/iron_worker_examples/blob/master/ruby/s3_tester/s3_worker.rb) on Github
+* [S3 Example 2](https://github.com/iron-io/iron_worker_examples/tree/master/ruby/download_to_s3) on Github
View
22 worker/reference/logging/index.md
@@ -0,0 +1,22 @@
+---
+title: Logging
+layout: default
+section: worker
+breadcrumbs:
+ - ['Reference', '/reference']
+ - ['Logging', '/logging']
+---
+
+# Logging from your Workers
+
+Anything you write to STDOUT or STDERR will be logged and stored so you can view and/or retrieve it later.
+
+## Viewing Logs
+
+You can view your worker logs in the [HUD](https://hud.iron.io) by clicking on a task.
+
+## Retrieving Logs via the API
+
+You can also retrieve the logs via the API, [view the reference docs here](http://localhost:4000/worker/reference/api/#get_a_tasks_log).
+All of the client libraries have a simple method to retrieve the logs via the API too.
+
View
54 worker/reference/webhooks/index.md
@@ -0,0 +1,54 @@
+---
+title: Webhooks
+layout: default
+section: worker
+breadcrumbs:
+ - ['Reference', '/reference']
+ - ['Webhooks', '/webhooks']
+---
+
+# IronWorker Webhooks - The Ultimate Webhook
+
+
+Using IronWorker webhooks enables you to run pretty much anything you want whenever an event happens at a third
+party service that supports webhooks.
+
+## How to Use IronWorker Webhooks
+
+The best way to see how this works is via an example. The rest of this section will use a **Github to Hipchat** webhook
+where Github will hit the webhook and the worker will post to Hipchat. [The full code is here](https://github.com/iron-io/iron_worker_examples/tree/master/ruby_ng/github_to_hipchat_webhook_worker).
+
+### Step 1: Create a worker and upload it to IronWorker
+
+This is the same as you would create and upload a worker normally, the difference is in how the task
+is queued up.
+
+First let's create the worker:
+
+<script src="https://gist.github.com/2308369.js?file=github_to_hipchat_webhook_worker.rb"></script>
+
+Now let's upload it:
+
+<script src="https://gist.github.com/2308369.js?file=upload.rb"></script>
+
+### Step 2: Add your workers webhook URL to Github service hooks
+
+[Github service hooks](http://help.github.com/post-receive-hooks/) are where you can add webhooks for Github events. In
+your Github project, click Admin, Service Hooks, then Post-Receive URLs.
+
+![Admin button](http://img.skitch.com/20100620-r8st7468q7q5waf3y85hmpwtqs.png)
+
+![Post-receive](http://img.skitch.com/20100620-br6dw5iiyk2643fahkqbi54h36.png)
+
+In the text field, add the webhook url for
+your worker, it should look something like this:
+
+{% highlight %}
+https://worker-aws-us-east-1/projects/{Project ID}/tasks/webhook?code_name={Code Name}
+{% endhighlight %}
+
+The upload script above will print the exact URL to your console so you can just copy and paste it.
+
+### Step 3: Commit and push some code to your github project and watch the magic happen!
+
+That's it! It will post your github commit information to the Hipchat room you specified in the config file.

0 comments on commit 61d3ddf

Please sign in to comment.
Something went wrong with that request. Please try again.