Skip to content
Browse files

Removed old version of iron_worker

  • Loading branch information...
1 parent 271594a commit 96eac117261d59fd2c9a3d3c5edad8706ad505d6 @paddyforan paddyforan committed Oct 26, 2012
Showing with 0 additions and 1,770 deletions.
  1. +0 −22 LICENSE.txt
  2. +0 −134 README.md
  3. +0 −18 config_example.ini
  4. +0 −55 helper.py
  5. +0 −7 iron_worker/__init__.py
  6. +0 −671 iron_worker/iron_worker.py
  7. +0 −32 iron_worker/poster/__init__.py
  8. +0 −414 iron_worker/poster/encode.py
  9. +0 −199 iron_worker/poster/streaminghttp.py
  10. +0 −24 setup.py
  11. +0 −193 test.py
  12. +0 −1 testDir/hello.py
View
22 LICENSE.txt
@@ -1,22 +0,0 @@
-Copyright (c) 2012, Iron.io, Inc. All rights reserved.
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are met:
-
-* Redistributions of source code must retain the above copyright notice,
-this list of conditions and the following disclaimer.
-* Redistributions in binary form must reproduce the above copyright notice,
-this list of conditions and the following disclaimer in the documentation
-and/or other materials provided with the distribution.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
-FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
-PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
-OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
-WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
-OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
-ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
View
134 README.md
@@ -1,134 +0,0 @@
-iron_worker_python is Python language binding for IronWorker.
-
-IronWorker is a massively scalable background processing system.
-[See How It Works](http://www.iron.io/products/worker/how)
-
-# Getting Started
-
-
-## Get credentials
-To start using iron_worker_python, you need to sign up and get an oauth token.
-
-1. Go to http://iron.io/ and sign up.
-2. Get an OAuth Token at http://hud.iron.io/tokens
-
-## Install iron_worker_python
-Just copy ```iron_worker.py``` and ```poster``` and include iron_worker.py in your script:
-
-```python
-from iron_worker import *
-```
-## Configure
-Two ways to configure IronWorker:
-
-* Specifying options when constructing the binding:
-
-```python
-worker = IronWorker(token='XXXXXXXXXX', project_id='xxxxxxxxxxx')
-```
-
-* Passing an ini file name which stores your configuration options. Rename sample_config.ini to config.ini and include your Iron.io credentials (`token` and `project_id`):
-
-```python
-worker = IronWorker(config='config.ini')
-```
-
-## Creating a Worker
-
-Here's an example worker:
-
-```python
-print "Hello PYTHON World!\n"
-```
-
-## Upload code to server
-
-* Zip worker:
-
-```python
-# Zip single file:
-IronWorker.createZip(destination="worker.zip", files=['HelloWorld.py'], overwrite=True)
-# OR
-# Zip whole directory:
-IronWorker.zipDirectory(directory="hello_world/", destination='worker.zip', overwrite=True)
-```
-
-* Submit worker:
-
-```python
-res = worker.postCode(runFilename='HelloWorld.py', zipFilename='worker.zip', name='HelloWorld')
-```
-
-Where 'HelloWorld' is a worker name which should be used later for queueing and sheduling.
-
-## Queueing a Worker
-
-```python
-task = worker.postTask(name='HelloWorld')
-```
-
-Worker should start in a few seconds.
-
-## Scheduling a Worker
-If you want to run your code at a delay, you should schedule it:
-
-If you want to run your code repeatedly, you can create a Scheduled Task:
-
-```python
-# run every hour, five times in total
-run_every = 60*60 # 60 seconds in a minute, 60 minutes in an hour
-worker.postSchedule(name='HelloWorld', run_every=run_every, run_times=5)
-```
-
-## Status of a Worker
-To get the status of a worker, you can use the ```getTaskDetails()``` method.
-
-```python
-task = worker.postTask('HelloWorld')
-details = worker.getTaskDetails(task_id=task['tasks'][0]['id']);
-
-print details['status'] # prints 'queued', 'complete', 'error' etc.
-```
-
-## Get Worker Log
-
-Use any function that print text inside your worker to put messages to log.
-
-```python
-import time
-task = worker.postTask('HelloWorld')
-time.sleep(10)
-details = worker.getTaskDetails(task['tasks'][0]['id'])
-# Check log only if task is finished.
-if details['status'] != 'queued':
- log = worker.getLog(task_id);
- print log # prints "Hello PHP World!"
-```
-
-## Loading the Task Data Payload
-
-To provide Payload to your worker simply put a dict with any content you want.
-
-```python
-payload = {
- 'key_one': 'Helpful text',
- 'key_two': 2,
- 'options': ['option 1', 'option 2']
-}
-
-worker.postTask(name='HelloWorld', payload=payload)
-```
-
-When your code is executed, it will be passed three program arguments:
-
-* **-id** - The task id.
-* **-payload** - the filename containing the data payload for this particular task.
-* **-d** - the user writable directory that can be used while running your job.
-
-# Full Documentation
-
-You can find more documentation here:
-
-* [Iron.io Dev Center](http://dev.iron.io): Full documentation for Iron.io products.
-* [Example Workers](https://github.com/iron-io/iron_worker_examples)
-* [IronWorker Python Wiki pages](https://github.com/iron-io/iron_worker_python/wiki).
View
18 config_example.ini
@@ -1,18 +0,0 @@
-[CloudCache]
-token = YOUR_TOKEN_HERE
-host = 174.129.12.177
-port = 4567
-
-[IronWorker]
-project_id = YOUR_PROJECT_ID
-token = YOUR_TOKEN_HERE
-version = 2
-host = worker-aws-us-east-1.iron.io
-port = 80
-
-[HipChat]
-token = YOUR_TOKEN_HERE
-version = v1
-host = api.hipchat.com
-port = 80
-
View
55 helper.py
@@ -1,55 +0,0 @@
-import urllib2
-import urllib
-import sys
-try:
- import json
-except ImportError:
- import simplejson as json
-
-DEFAULT_HOST = "worker-aws-us-east-1.iron.io"
-USER_AGENT = "IronWorker Python v0.3"
-
-
-def getArgs():
- """Get the arguments that are passed to all IronWorkers as a dict"""
- args = {}
- for i in range(len(sys.argv)):
- if sys.argv[i].startswith("-") and (i + 1) < len(sys.argv):
- key = sys.argv[i][1:]
- i += 1
- args[key] = sys.argv[i]
- return args
-
-
-def getArg(key):
- args = getArgs()
- return args[key]
-
-
-def setTaskProgress(token, percent=None, message=None, project_id=None,
- task_id=None, host=DEFAULT_HOST, port=80, version=2,
- protocol="http"):
- if percent is None and message is None:
- return
- payload = {}
- if percent is not None:
- payload["percent"] = percent
- if message is not None:
- payload["msg"] = message
- if task_id is None:
- task_id = getArg("id")
- url = "%s://%s:%s/%s/projects/%s/tasks/%s/progress?oauth=%s" % (
- protocol, host, port, version, project_id, task_id,
- token)
- headers = {
- "Accept": "application/json",
- "Accept-Encoding": "gzip, deflate",
- "User-Agent": USER_AGENT
- }
- data = json.dumps(payload)
- dataLen = len(data)
- headers['Content-Type'] = "application/json"
- headers['Content-Length'] = str(dataLen)
- req = urllib2.Request(url, data, headers)
- ret = urllib2.urlopen(req)
- return ret.read()
View
7 iron_worker/__init__.py
@@ -1,7 +0,0 @@
-import sys
-import os
-sys.path.insert(0, os.path.split(__file__)[0])
-import iron_worker
-IronWorker = iron_worker.IronWorker
-
-
View
671 iron_worker/iron_worker.py
@@ -1,671 +0,0 @@
-# IronWorker For Python
-import os
-import sys
-import time
-import urllib2
-from poster.encode import multipart_encode
-from poster.streaminghttp import register_openers
-import zipfile
-import ConfigParser
-try:
- import json
-except ImportError:
- import simplejson as json
-
-
-def file_exists(file):
- """Check if a file exists."""
- try:
- open(file)
- except IOError:
- return False
- else:
- return True
-
-
-class IllegalArgumentException(Exception):
- def __init__(self, message):
- self.message = message
-
- def __str__(self):
- return repr(self.message)
-
-
-class RequestWithMethod(urllib2.Request):
-
- """Wrap urllib2 to make DELETE requests possible."""
-
- def __init__(self, url, method, data=None, headers={},
- origin_req_host=None, unverifiable=False):
- self._method = method
- return urllib2.Request.__init__(self, url, data, headers,
- origin_req_host, unverifiable)
-
- def get_method(self):
- if self._method:
- return self._method
- else:
- return urllib2.Request.get_method(self)
-
-
-class IronWorker:
- DEFAULT_HOST = "worker-aws-us-east-1.iron.io"
- USER_AGENT = "IronWorker Python v0.3"
-
- def __init__(self, token=None, project_id=None, host=DEFAULT_HOST, port=80,
- version=2, protocol='http', config=None, app_engine=False):
- """Prepare a configured instance of the API wrapper and return it.
-
- Keyword arguments:
- token -- An API token found on http://hud.iron.io. Defaults to None.
- project_id -- The ID for the project, found on http://hud.iron.io
- host -- The hostname of the API server.
- Defaults to worker-aws-us-east-1.iron.io.
- port -- The port of the API server. Defaults to 80.
- version -- The API version to use. Defaults to 2.
- protocol -- The protocol to use. Defaults to http.
- config -- The config file to draw config values from. Defaults to None.
- app_engine -- Whether to run in App Engine compatibility mode.
- Defaults to False.
- """
- self.token = token
- self.version = version
- self.project_id = project_id
- self.protocol = protocol
- self.host = host
- self.port = port
- self.version = version
- self.app_engine = app_engine
- if config is not None:
- config_file = ConfigParser.RawConfigParser()
- config_file.read(config)
- try:
- self.token = config_file.get("IronWorker", "token")
- except ConfigParser.NoOptionError:
- self.token = token
- try:
- self.project_id = config_file.get("IronWorker", "project_id")
- except ConfigParser.NoOptionError:
- self.project_id = project_id
- try:
- self.host = config_file.get("IronWorker", "host")
- except ConfigParser.NoOptionError:
- self.host = host
- try:
- self.port = config_file.get("IronWorker", "port")
- except ConfigParser.NoOptionError:
- self.port = port
- try:
- self.version = config_file.get("IronWorker", "version")
- except ConfigParser.NoOptionError:
- self.version = version
- try:
- self.protocol = config_file.get("IronWorker", "protocol")
- except ConfigParser.NoOptionError:
- self.protocol = protocol
- if self.token is None or self.project_id is None:
- raise IllegalArgumentException("Both token and project_id must \
- have a value")
- self.url = "%s://%s:%s/%s/" % (self.protocol, self.host, self.port,
- self.version)
- self.__setCommonHeaders()
-
- def __get(self, url, headers={}):
- """Execute an HTTP GET request and return the result.
-
- Keyword arguments:
- url -- The url to execute the request against. (Required)
- headers -- A dict of headers to merge with self.headers and send
- with the request.
- """
- headers = dict(headers.items() + self.headers.items())
- if not self.app_engine:
- req = urllib2.Request(url, None, headers)
- ret = urllib2.urlopen(req)
- return ret.read()
- else:
- from google.appengine.api import urlfetch
- return urlfetch.fetch(url=url, method=urlfetch.GET,
- headers=headers).content
-
- def __post(self, url, payload={}, headers={}):
- """Execute an HTTP POST request and return the result.
-
- Keyword arguments:
- url -- The url to execute the request against. (Required)
- payload -- A dict of key-value form data to send with the
- request. Will be urlencoded.
- headers -- A dict of headers to merge with self.headers and send
- with the request.
- """
- headers = dict(headers.items() + self.headers.items())
- if not self.app_engine:
- req = urllib2.Request(url, payload, headers)
- ret = urllib2.urlopen(req)
- return ret.read()
- else:
- from google.appengine.api import urlfetch
- if not isinstance(payload, basestring):
- import urllib
- payload = urllib.urlencode(payload)
- return urlfetch.fetch(url=url, payload=payload,
- method=urlfetch.POST, headers=headers).content
-
- def __delete(self, url, payload={}, headers={}):
- """Execute an HTTP DELETE request and return the result.
-
- Keyword arguments:
- url -- The url to execute the request against. (Required)
- payload -- A dict of key-value form data to send with the
- request. Will be urlencoded.
- headers -- A dict of headers to merge with self.headers and send
- with the request.
- """
- headers = dict(headers.items() + self.headers.items())
- if not self.app_engine:
- req = RequestWithMethod(url=url, method='DELETE', data=payload,
- headers=headers)
- ret = urllib2.urlopen(req)
- s = ret.read()
- else:
- from google.appengine.api import urlfetch
- if not isinstance(payload, basestring):
- import urllib
- payload = urllib.urlencode(payload)
- return urlfetch.fetch(url=url, payload=payload,
- method=urlfetch.DELETE, headers=headers).content
-
- def __setCommonHeaders(self):
- """Modify your headers to match the JSON default values."""
- self.headers = {
- "Accept": "application/json",
- "Accept-Encoding": "gzip, deflate",
- "User-Agent": "IronWorker Python v0.3"
- }
-
- @staticmethod
- def getArgs():
- """Get the arguments that are passed to all IronWorkers as a dict"""
- args = {}
- for i in range(len(sys.argv)):
- if sys.argv[i].startswith("-") and (i + 1) < len(sys.argv):
- key = sys.argv[i][1:]
- i += 1
- args[key] = sys.argv[i]
- return args
-
- @staticmethod
- def getPayload():
- """Get the payload that was sent to a worker."""
- args = IronWorker.getArgs()
- if 'payload' in args and file_exists(args['payload']):
- return json.loads(open(args['payload']).read())
-
- @staticmethod
- def Rfc3339(timestamp=None):
- if timestamp is None:
- timestamp = time.gmtime()
- base = time.strftime("%Y-%m-%dT%H:%M:%S", timestamp)
- timezone = time.strftime("%z", timestamp)
- if timezone is not None and timezone != "+00:00" and timezone != "":
- timezone = "%s:%s" % (timezone[:-2], timezone[-2:])
- elif timezone == "+00:00":
- timezone = "Z"
- else:
- timezone = "-00:00"
- return "%s%s" % (base, timezone)
-
- def getTasks(self, project_id=None):
- """Execute an HTTP request to get a list of tasks, and return it.
-
- Keyword arguments:
- project_id -- The project ID to get tasks from. Defaults to the
- project ID set when initialising the wrapper.
- """
- self.__setCommonHeaders()
- if project_id is None:
- project_id = self.project_id
- url = "%sprojects/%s/tasks?oauth=%s" % (self.url, project_id,
- self.token)
- body = self.__get(url)
- tasks = json.loads(body)
- return tasks['tasks']
-
- def getTaskDetails(self, task_id, project_id=""):
- """Execute an HTTP request to get details on a specific task, and
- return it.
-
- Keyword arguments:
- task_id -- The ID of the task to get the details of. (Required)
- project_id -- The ID of the project the task belongs to. Defaults to
- the project ID set when initialising the wrapper.
- """
- self.__setCommonHeaders()
- if project_id == "":
- project_id = self.project_id
- url = "%sprojects/%s/tasks/%s?oauth=%s" % (self.url, project_id,
- task_id, self.token)
- body = self.__get(url)
- task = json.loads(body)
- return task
-
- def getProjects(self):
- """Execute an HTTP request to get a list of projects and return it."""
- self.__setCommonHeaders()
- url = "%sprojects?oauth=%s" % (self.url, self.token)
- self.__setCommonHeaders()
- body = self.__get(url)
- projects = json.loads(body)
- return projects['projects']
-
- def setProject(self, project_id):
- """Set the default project ID for the wrapper.
-
- Keyword arguments:
- project_id -- The new project_id to use as the default. (Required)
- """
- self.project_id = project_id
-
- def getProjectDetails(self, project_id=None):
- """Execute an HTTP request to get the details of a project and return
- them.
-
- Keyword arguments:
- project_id -- The ID of the project to get details on. Defaults to the
- project ID set when initialising the wrapper.
- """
- self.__setCommonHeaders()
- if project_id is None:
- project_id = self.project_id
-
- url = "%sprojects/%s?oauth=%s" % (self.url, project_id, self.token)
- return json.loads(self.__get(url))
-
- def getCodes(self, project_id=None):
- """Execute an HTTP request to get a list of code packages associated
- with a project, and return them.
-
- Keyword arguments:
- project_id -- The ID of the project whose code packages are to be
- fetched. Defaults to the project ID set when initialising
- the wrapper.
- """
- self.__setCommonHeaders()
- if project_id is None:
- project_id = self.project_id
-
- url = "%sprojects/%s/codes?oauth=%s" % (self.url, project_id,
- self.token)
- body = self.__get(url)
- ret = json.loads(body)
- return ret['codes']
-
- def getCodeDetails(self, code_id):
- """Execute an HTTP request to get the details of a specific code
- package.
-
- Keyword arguments:
- code_id -- The ID of the code package to fetch details on. (Required)
- """
- self.__setCommonHeaders()
- project_id = self.project_id
- url = "%sprojects/%s/codes/%s?oauth=%s" % (self.url, project_id,
- code_id, self.token)
- return json.loads(self.__get(url))
-
- def postCode(self, name, runFilename, zipFilename, project_id=None):
-
- """Upload a code package to the IronWorker servers, to be executed
- against later.
-
- Keyword arguments:
- name -- A label for the code package, used to refer to it. (Required)
- runFilename -- The filename to be executed when the code
- is to be run. (Required)
- zipFilename -- The filename of a zip file containing the code
- to be uploaded. (Required)
- project_id -- The project to upload the code package to. Defaults to
- the project ID set when initialising the wrapper.
- """
- if project_id is None:
- project_id = self.project_id
-
- url = "%sprojects/%s/codes?oauth=%s" % (self.url, project_id,
- self.token)
-
- register_openers()
- ts = time.asctime()
- data = json.dumps({
- "code_name": name,
- "name": name,
- "standalone": True,
- "runtime": "python",
- "file_name": runFilename,
- "version": self.version,
- "timestamp": ts,
- "oauth": self.token,
- "class_name": name,
- "options": {},
- "access_key": name
- })
-
- datagen, headers = multipart_encode({
- "file": open(zipFilename, 'rb'),
- "data": data
- })
-
- body = self.__post(url=url, payload=str().join(datagen),
- headers=headers)
- return json.loads(body)
-
- def postProject(self, name):
- """Create a new project on the IronWorker servers. Returns the new
- project's ID.
-
- Keyword arguments:
- name -- The name of the new project. (Required)
- """
- self.__setCommonHeaders()
- url = "%sprojects?oauth=%s" % (self.url, self.token)
- payload = [{
- "name": name,
- "class_name": name,
- "access_key": name
- }]
-
- data = {"name": name}
- data = json.dumps(data)
- dataLen = len(data)
- headers = self.headers
- headers['Content-Type'] = "application/json"
- headers['Content-Length'] = str(dataLen)
-
- s = self.__post(url=url, payload=data, headers=headers)
- msg = json.loads(s)
- project_id = msg['id']
- self.__setCommonHeaders()
- return project_id
-
- def deleteCode(self, code_id, project_id=None):
- """Execute an HTTP request to delete a code package.
-
- Keyword arguments:
- code_id -- The ID of the code package to be deleted. (Required)
- project_id -- The ID of the project that contains the code package.
- Defaults to the project ID set when initialising the
- wrapper.
- """
- self.__setCommonHeaders()
- if project_id is None:
- project_id = self.project_id
- url = "%sprojects/%s/codes/%s?oauth=%s" % (self.url, project_id,
- code_id, self.token)
- req = RequestWithMethod(url, 'DELETE')
- ret = urllib2.urlopen(req)
- s = ret.read()
- return
-
- def cancelTask(self, task_id, project_id=None):
- """Execute an HTTP request to cancel a task.
-
- Keyword arguments:
- task_id -- The ID of the task to be cancelled. (Required)
- project_id -- The ID of the project that contains the task. Defaults
- to the project_id set when the wrapper was initialised.
- """
- self.__setCommonHeaders()
- if project_id is None:
- project_id = self.project_id
- url = "%sprojects/%s/tasks/%s/cancel?oauth=%s" % (self.url, project_id,
- task_id, self.token)
- data = json.dumps({})
- dataLen = len(data)
- headers = self.headers
- headers['Content-Type'] = "application/json"
- headers['Content-Length'] = str(dataLen)
-
- s = self.__post(url=url, payload=data, headers=headers)
-
- ret = json.loads(s)
- return ret
-
- def cancelSchedule(self, schedule_id, project_id=None):
- """Execute an HTTP request to cancel a task schedule.
-
- Keyword arguments:
- schedule_id -- The ID of the schedule to cancel. (Required)
- project_id -- The ID of the project that contains the schedule.
- Defaults to the project_id set when the wrapper was
- intialised.
- """
- self.__setCommonHeaders()
- if project_id is None:
- project_id = self.project_id
- url = "%sprojects/%s/schedules/%s/cancel?oauth=%s" % (self.url,
- project_id, schedule_id, self.token)
- data = json.dumps({})
- dataLen = len(data)
- headers = self.headers
- headers['Content-Type'] = "application/json"
- headers['Content-Length'] = str(dataLen)
-
- s = self.__post(url=url, payload=data, headers=headers)
-
- ret = json.loads(s)
- return ret
-
- def getSchedules(self, project_id=None):
- """Execute an HTTP request to get a list of task schedules and return
- them.
-
- Keyword arguments:
- project_id -- The ID of the project whose scheduled tasks are to be
- retrieved. Defaults to the project ID set when the
- wrapper was initialised.
- """
- self.__setCommonHeaders()
- if project_id is None:
- project_id = self.project_id
- url = "%sprojects/%s/schedules?oauth=%s" % (self.url, project_id,
- self.token)
- body = self.__get(url)
- schedules = json.loads(body)
- return schedules['schedules']
-
- def postSchedule(self, name, delay=None, payload={}, code_name=None,
- start_at=None, run_every=None, end_at=None, run_times=None,
- priority=0, project_id=None):
-
- """Execute an HTTP request to create a scheduled task that will be
- executed later.
-
- Keyword arguments:
- name -- A name for the schedule. (Required)
- delay -- The number of seconds to delay execution for.
- Defaults to None.
- payload -- The payload of arguments to execute the task with. Defaults
- to an empty dict.
- code_name -- The name of the code package to execute. Defaults to the
- name of the schedule.
- start_at -- A Time or DateTime object indicating when the schedule
- should start. Defaults to None.
- run_every -- The number of seconds between runs. If omitted, the
- schedule will be run once. Defaults to None.
- end_at -- A Time or DateTime object indicating when the schedule should
- end. Defaults to None.
- run_times -- The number of times to run the task. Defaults to None.
- priority -- The priority queue to run the job in (0, 1, 2). Run tasks
- at higher priority to decrease the time they may spend on
- queue once they come off the schedule. Defaults to 0.
- project_id -- The ID of the project to schedule the task under.
- Defaults to the project ID set when the wrapper was
- initialised.
- """
-
- self.__setCommonHeaders()
- if project_id is None:
- project_id = self.project_id
- if code_name is None:
- code_name = name
- url = "%sprojects/%s/schedules?oauth=%s" % (self.url, project_id,
- self.token)
-
- schedule = {
- "name": name
- }
- if delay is None and start_at is None:
- raise IllegalArgumentException("Either delay or start_at needs \
- to be set.")
- if delay is not None and start_at is None:
- schedule['delay'] = delay
- elif delay is None and start_at is not None:
- schedule['start_at'] = IronWorker.Rfc3339(start_at)
- elif delay is not None and start_at is not None:
- schedule['start_at'] = IronWorker.Rfc3339(start_at)
- schedule['code_name'] = code_name
- schedule['payload'] = json.dumps(payload)
- if run_every is not None:
- schedule['run_every'] = run_every
- if end_at is not None:
- schedule['end_at'] = IronWorker.Rfc3339(end_at)
- if run_times is not None:
- schedule['run_times'] = run_times
- schedule['priority'] = priority
-
- data = {"schedules": [schedule]}
- data = json.dumps(data)
- dataLen = len(data)
- headers = self.headers
- headers['Content-Type'] = "application/json"
- headers['Content-Length'] = str(dataLen)
- s = self.__post(url=url, payload=data, headers=headers)
-
- msg = json.loads(s)
- schedule_id = msg['schedules'][0]['id']
- self.__setCommonHeaders()
- return schedule_id
-
- def postTask(self, name, payload={}, project_id=None, priority=0, timeout=3600, delay=0):
- """Executes an HTTP request to create a task that will be executed by
- the worker.
-
- Keyword arguments:
- name -- The name of the code package to execute against. (Required)
- payload -- Arguments to be passed to the task. Defaults to {}.
- project_id -- The ID of the project the task is to be created under.
- Defaults to the project ID set when the wrapper was
- initialised.
- priority -- The priority queue to run the task in. Valid values are 0, 1, and 2. 0 is the default.
- timeout -- The maximum runtime of your task in seconds. No task can exceed 3600 seconds (60 minutes).
- The default is 3600 but can be set to a shorter duration.
- delay -- The number of seconds to delay before actually queuing the task. Default is 0.
- """
- self.__setCommonHeaders()
- if project_id is None:
- project_id = self.project_id
- url = "%sprojects/%s/tasks?oauth=%s" % (self.url, project_id,
- self.token)
- payload = json.dumps(payload)
- task = {
- "name": name,
- "code_name": name,
- "payload": payload,
- "priority": priority,
- "timeout" : timeout,
- "delay" : delay,
- }
- tasks = {"tasks": [task]}
- data = json.dumps(tasks)
- dataLen = len(data)
- headers = self.headers
- headers['Content-Type'] = "application/json"
- headers['Content-Length'] = str(dataLen)
-
- s = self.__post(url=url, payload=data, headers=headers)
-
- ret = json.loads(s)
- return ret
-
- def getLog(self, task_id, project_id=None):
- """Executes an HTTP request to fetch the log associated with a task.
-
- Keyword arguments:
- task_id -- The ID of the task whose log is being retrieved.
- project_id -- The ID of the project that contains the task whose log is
- being retrieved. Defaults to the project ID set when the
- wrapper was initialised.
- """
- self.__setCommonHeaders()
- if project_id is None:
- project_id = self.project_id
- url = "%sprojects/%s/tasks/%s/log?oauth=%s" % (self.url, project_id,
- task_id, self.token)
- self.headers['Accept'] = "text/plain"
- try:
- del self.headers['Content-Type']
- del self.headers['Content-Length']
- except:
- pass
- body = self.__get(url)
- return body
-
- @staticmethod
- def createZip(destination, base_dir='', files=[], overwrite=False):
- """Create a zip from an array of filenames.
-
- Keyword arguments:
- destination -- The filename or path the zip file is to be created
- in. (Required)
- base_dir -- A directory tree within the zip file that all files will be
- zipped under. Defaults to ''.
- files -- A list of files to included in the zip. Defaults to [].
- overwrite -- Whether the zip file should be overwritten if it exists.
- Defaults to False.
- """
- if file_exists(destination) and not overwrite:
- return False
- valid_files = []
- for file in files:
- if file_exists(file):
- valid_files.append(file)
- if len(valid_files) > 0:
- zip = zipfile.ZipFile(destination, "w")
- for file in valid_files:
- zip.write(file, os.path.join(base_dir, file))
- zip.close()
- return file_exists(destination)
-
- @staticmethod
- def zipDirectory(directory, destination, overwrite=False):
- """Create a zip from a directory. Passes through to createZip.
-
- Keyword arguments:
- directory -- The path to the directory to be zipped. (Required)
- destination -- The filename or path the zip file is to be created
- in. (Required)
- overwrite -- Whether the zip file should be overwritten if it exists.
- Defaults to False.
- """
- if not os.path.isdir(directory):
- return False
-
- files = IronWorker.getFilenames(directory)
- if len(files) < 1:
- return False
-
- return IronWorker.createZip(files=files, destination=destination,
- overwrite=overwrite)
-
- @staticmethod
- def getFilenames(directory):
- """Get a list of filenames and return it.
-
- Keyword arguments:
- directory -- The path to the directory whose filenames are to be
- retrieved. (Required)
- """
- names = []
- for dirname, dirnames, filenames in os.walk(directory):
- for filename in filenames:
- names.append(os.path.join(dirname, filename))
- return names
View
32 iron_worker/poster/__init__.py
@@ -1,32 +0,0 @@
-# Copyright (c) 2011 Chris AtLee
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-"""poster module
-
-Support for streaming HTTP uploads, and multipart/form-data encoding
-
-```poster.version``` is a 3-tuple of integers representing the version number.
-New releases of poster will always have a version number that compares greater
-than an older version of poster.
-New in version 0.6."""
-
-import poster.streaminghttp
-import poster.encode
-
-version = (0, 8, 1) # Thanks JP!
View
414 iron_worker/poster/encode.py
@@ -1,414 +0,0 @@
-"""multipart/form-data encoding module
-
-This module provides functions that faciliate encoding name/value pairs
-as multipart/form-data suitable for a HTTP POST or PUT request.
-
-multipart/form-data is the standard way to upload files over HTTP"""
-
-__all__ = ['gen_boundary', 'encode_and_quote', 'MultipartParam',
- 'encode_string', 'encode_file_header', 'get_body_size', 'get_headers',
- 'multipart_encode']
-
-try:
- import uuid
- def gen_boundary():
- """Returns a random string to use as the boundary for a message"""
- return uuid.uuid4().hex
-except ImportError:
- import random, sha
- def gen_boundary():
- """Returns a random string to use as the boundary for a message"""
- bits = random.getrandbits(160)
- return sha.new(str(bits)).hexdigest()
-
-import urllib, re, os, mimetypes
-try:
- from email.header import Header
-except ImportError:
- # Python 2.4
- from email.Header import Header
-
-def encode_and_quote(data):
- """If ``data`` is unicode, return urllib.quote_plus(data.encode("utf-8"))
- otherwise return urllib.quote_plus(data)"""
- if data is None:
- return None
-
- if isinstance(data, unicode):
- data = data.encode("utf-8")
- return urllib.quote_plus(data)
-
-def _strify(s):
- """If s is a unicode string, encode it to UTF-8 and return the results,
- otherwise return str(s), or None if s is None"""
- if s is None:
- return None
- if isinstance(s, unicode):
- return s.encode("utf-8")
- return str(s)
-
-class MultipartParam(object):
- """Represents a single parameter in a multipart/form-data request
-
- ``name`` is the name of this parameter.
-
- If ``value`` is set, it must be a string or unicode object to use as the
- data for this parameter.
-
- If ``filename`` is set, it is what to say that this parameter's filename
- is. Note that this does not have to be the actual filename any local file.
-
- If ``filetype`` is set, it is used as the Content-Type for this parameter.
- If unset it defaults to "text/plain; charset=utf8"
-
- If ``filesize`` is set, it specifies the length of the file ``fileobj``
-
- If ``fileobj`` is set, it must be a file-like object that supports
- .read().
-
- Both ``value`` and ``fileobj`` must not be set, doing so will
- raise a ValueError assertion.
-
- If ``fileobj`` is set, and ``filesize`` is not specified, then
- the file's size will be determined first by stat'ing ``fileobj``'s
- file descriptor, and if that fails, by seeking to the end of the file,
- recording the current position as the size, and then by seeking back to the
- beginning of the file.
-
- ``cb`` is a callable which will be called from iter_encode with (self,
- current, total), representing the current parameter, current amount
- transferred, and the total size.
- """
- def __init__(self, name, value=None, filename=None, filetype=None,
- filesize=None, fileobj=None, cb=None):
- self.name = Header(name).encode()
- self.value = _strify(value)
- if filename is None:
- self.filename = None
- else:
- if isinstance(filename, unicode):
- # Encode with XML entities
- self.filename = filename.encode("ascii", "xmlcharrefreplace")
- else:
- self.filename = str(filename)
- self.filename = self.filename.encode("string_escape").\
- replace('"', '\\"')
- self.filetype = _strify(filetype)
-
- self.filesize = filesize
- self.fileobj = fileobj
- self.cb = cb
-
- if self.value is not None and self.fileobj is not None:
- raise ValueError("Only one of value or fileobj may be specified")
-
- if fileobj is not None and filesize is None:
- # Try and determine the file size
- try:
- self.filesize = os.fstat(fileobj.fileno()).st_size
- except (OSError, AttributeError):
- try:
- fileobj.seek(0, 2)
- self.filesize = fileobj.tell()
- fileobj.seek(0)
- except:
- raise ValueError("Could not determine filesize")
-
- def __cmp__(self, other):
- attrs = ['name', 'value', 'filename', 'filetype', 'filesize', 'fileobj']
- myattrs = [getattr(self, a) for a in attrs]
- oattrs = [getattr(other, a) for a in attrs]
- return cmp(myattrs, oattrs)
-
- def reset(self):
- if self.fileobj is not None:
- self.fileobj.seek(0)
- elif self.value is None:
- raise ValueError("Don't know how to reset this parameter")
-
- @classmethod
- def from_file(cls, paramname, filename):
- """Returns a new MultipartParam object constructed from the local
- file at ``filename``.
-
- ``filesize`` is determined by os.path.getsize(``filename``)
-
- ``filetype`` is determined by mimetypes.guess_type(``filename``)[0]
-
- ``filename`` is set to os.path.basename(``filename``)
- """
-
- return cls(paramname, filename=os.path.basename(filename),
- filetype=mimetypes.guess_type(filename)[0],
- filesize=os.path.getsize(filename),
- fileobj=open(filename, "rb"))
-
- @classmethod
- def from_params(cls, params):
- """Returns a list of MultipartParam objects from a sequence of
- name, value pairs, MultipartParam instances,
- or from a mapping of names to values
-
- The values may be strings or file objects, or MultipartParam objects.
- MultipartParam object names must match the given names in the
- name,value pairs or mapping, if applicable."""
- if hasattr(params, 'items'):
- params = params.items()
-
- retval = []
- for item in params:
- if isinstance(item, cls):
- retval.append(item)
- continue
- name, value = item
- if isinstance(value, cls):
- assert value.name == name
- retval.append(value)
- continue
- if hasattr(value, 'read'):
- # Looks like a file object
- filename = getattr(value, 'name', None)
- if filename is not None:
- filetype = mimetypes.guess_type(filename)[0]
- else:
- filetype = None
-
- retval.append(cls(name=name, filename=filename,
- filetype=filetype, fileobj=value))
- else:
- retval.append(cls(name, value))
- return retval
-
- def encode_hdr(self, boundary):
- """Returns the header of the encoding of this parameter"""
- boundary = encode_and_quote(boundary)
-
- headers = ["--%s" % boundary]
-
- if self.filename:
- disposition = 'form-data; name="%s"; filename="%s"' % (self.name,
- self.filename)
- else:
- disposition = 'form-data; name="%s"' % self.name
-
- headers.append("Content-Disposition: %s" % disposition)
-
- if self.filetype:
- filetype = self.filetype
- else:
- filetype = "text/plain; charset=utf-8"
-
- headers.append("Content-Type: %s" % filetype)
-
- headers.append("")
- headers.append("")
-
- return "\r\n".join(headers)
-
- def encode(self, boundary):
- """Returns the string encoding of this parameter"""
- if self.value is None:
- value = self.fileobj.read()
- else:
- value = self.value
-
- if re.search("^--%s$" % re.escape(boundary), value, re.M):
- raise ValueError("boundary found in encoded string")
-
- return "%s%s\r\n" % (self.encode_hdr(boundary), value)
-
- def iter_encode(self, boundary, blocksize=4096):
- """Yields the encoding of this parameter
- If self.fileobj is set, then blocks of ``blocksize`` bytes are read and
- yielded."""
- total = self.get_size(boundary)
- current = 0
- if self.value is not None:
- block = self.encode(boundary)
- current += len(block)
- yield block
- if self.cb:
- self.cb(self, current, total)
- else:
- block = self.encode_hdr(boundary)
- current += len(block)
- yield block
- if self.cb:
- self.cb(self, current, total)
- last_block = ""
- encoded_boundary = "--%s" % encode_and_quote(boundary)
- boundary_exp = re.compile("^%s$" % re.escape(encoded_boundary),
- re.M)
- while True:
- block = self.fileobj.read(blocksize)
- if not block:
- current += 2
- yield "\r\n"
- if self.cb:
- self.cb(self, current, total)
- break
- last_block += block
- if boundary_exp.search(last_block):
- raise ValueError("boundary found in file data")
- last_block = last_block[-len(encoded_boundary)-2:]
- current += len(block)
- yield block
- if self.cb:
- self.cb(self, current, total)
-
- def get_size(self, boundary):
- """Returns the size in bytes that this param will be when encoded
- with the given boundary."""
- if self.filesize is not None:
- valuesize = self.filesize
- else:
- valuesize = len(self.value)
-
- return len(self.encode_hdr(boundary)) + 2 + valuesize
-
-def encode_string(boundary, name, value):
- """Returns ``name`` and ``value`` encoded as a multipart/form-data
- variable. ``boundary`` is the boundary string used throughout
- a single request to separate variables."""
-
- return MultipartParam(name, value).encode(boundary)
-
-def encode_file_header(boundary, paramname, filesize, filename=None,
- filetype=None):
- """Returns the leading data for a multipart/form-data field that contains
- file data.
-
- ``boundary`` is the boundary string used throughout a single request to
- separate variables.
-
- ``paramname`` is the name of the variable in this request.
-
- ``filesize`` is the size of the file data.
-
- ``filename`` if specified is the filename to give to this field. This
- field is only useful to the server for determining the original filename.
-
- ``filetype`` if specified is the MIME type of this file.
-
- The actual file data should be sent after this header has been sent.
- """
-
- return MultipartParam(paramname, filesize=filesize, filename=filename,
- filetype=filetype).encode_hdr(boundary)
-
-def get_body_size(params, boundary):
- """Returns the number of bytes that the multipart/form-data encoding
- of ``params`` will be."""
- size = sum(p.get_size(boundary) for p in MultipartParam.from_params(params))
- return size + len(boundary) + 6
-
-def get_headers(params, boundary):
- """Returns a dictionary with Content-Type and Content-Length headers
- for the multipart/form-data encoding of ``params``."""
- headers = {}
- boundary = urllib.quote_plus(boundary)
- headers['Content-Type'] = "multipart/form-data; boundary=%s" % boundary
- headers['Content-Length'] = str(get_body_size(params, boundary))
- return headers
-
-class multipart_yielder:
- def __init__(self, params, boundary, cb):
- self.params = params
- self.boundary = boundary
- self.cb = cb
-
- self.i = 0
- self.p = None
- self.param_iter = None
- self.current = 0
- self.total = get_body_size(params, boundary)
-
- def __iter__(self):
- return self
-
- def next(self):
- """generator function to yield multipart/form-data representation
- of parameters"""
- if self.param_iter is not None:
- try:
- block = self.param_iter.next()
- self.current += len(block)
- if self.cb:
- self.cb(self.p, self.current, self.total)
- return block
- except StopIteration:
- self.p = None
- self.param_iter = None
-
- if self.i is None:
- raise StopIteration
- elif self.i >= len(self.params):
- self.param_iter = None
- self.p = None
- self.i = None
- block = "--%s--\r\n" % self.boundary
- self.current += len(block)
- if self.cb:
- self.cb(self.p, self.current, self.total)
- return block
-
- self.p = self.params[self.i]
- self.param_iter = self.p.iter_encode(self.boundary)
- self.i += 1
- return self.next()
-
- def reset(self):
- self.i = 0
- self.current = 0
- for param in self.params:
- param.reset()
-
-def multipart_encode(params, boundary=None, cb=None):
- """Encode ``params`` as multipart/form-data.
-
- ``params`` should be a sequence of (name, value) pairs or MultipartParam
- objects, or a mapping of names to values.
- Values are either strings parameter values, or file-like objects to use as
- the parameter value. The file-like objects must support .read() and either
- .fileno() or both .seek() and .tell().
-
- If ``boundary`` is set, then it as used as the MIME boundary. Otherwise
- a randomly generated boundary will be used. In either case, if the
- boundary string appears in the parameter values a ValueError will be
- raised.
-
- If ``cb`` is set, it should be a callback which will get called as blocks
- of data are encoded. It will be called with (param, current, total),
- indicating the current parameter being encoded, the current amount encoded,
- and the total amount to encode.
-
- Returns a tuple of `datagen`, `headers`, where `datagen` is a
- generator that will yield blocks of data that make up the encoded
- parameters, and `headers` is a dictionary with the assoicated
- Content-Type and Content-Length headers.
-
- Examples:
-
- >>> datagen, headers = multipart_encode( [("key", "value1"), ("key", "value2")] )
- >>> s = "".join(datagen)
- >>> assert "value2" in s and "value1" in s
-
- >>> p = MultipartParam("key", "value2")
- >>> datagen, headers = multipart_encode( [("key", "value1"), p] )
- >>> s = "".join(datagen)
- >>> assert "value2" in s and "value1" in s
-
- >>> datagen, headers = multipart_encode( {"key": "value1"} )
- >>> s = "".join(datagen)
- >>> assert "value2" not in s and "value1" in s
-
- """
- if boundary is None:
- boundary = gen_boundary()
- else:
- boundary = urllib.quote_plus(boundary)
-
- headers = get_headers(params, boundary)
- params = MultipartParam.from_params(params)
-
- return multipart_yielder(params, boundary, cb), headers
View
199 iron_worker/poster/streaminghttp.py
@@ -1,199 +0,0 @@
-"""Streaming HTTP uploads module.
-
-This module extends the standard httplib and urllib2 objects so that
-iterable objects can be used in the body of HTTP requests.
-
-In most cases all one should have to do is call :func:`register_openers()`
-to register the new streaming http handlers which will take priority over
-the default handlers, and then you can use iterable objects in the body
-of HTTP requests.
-
-**N.B.** You must specify a Content-Length header if using an iterable object
-since there is no way to determine in advance the total size that will be
-yielded, and there is no way to reset an interator.
-
-Example usage:
-
->>> from StringIO import StringIO
->>> import urllib2, poster.streaminghttp
-
->>> opener = poster.streaminghttp.register_openers()
-
->>> s = "Test file data"
->>> f = StringIO(s)
-
->>> req = urllib2.Request("http://localhost:5000", f,
-... {'Content-Length': str(len(s))})
-"""
-
-import httplib, urllib2, socket
-from httplib import NotConnected
-
-__all__ = ['StreamingHTTPConnection', 'StreamingHTTPRedirectHandler',
- 'StreamingHTTPHandler', 'register_openers']
-
-if hasattr(httplib, 'HTTPS'):
- __all__.extend(['StreamingHTTPSHandler', 'StreamingHTTPSConnection'])
-
-class _StreamingHTTPMixin:
- """Mixin class for HTTP and HTTPS connections that implements a streaming
- send method."""
- def send(self, value):
- """Send ``value`` to the server.
-
- ``value`` can be a string object, a file-like object that supports
- a .read() method, or an iterable object that supports a .next()
- method.
- """
- # Based on python 2.6's httplib.HTTPConnection.send()
- if self.sock is None:
- if self.auto_open:
- self.connect()
- else:
- raise NotConnected()
-
- # send the data to the server. if we get a broken pipe, then close
- # the socket. we want to reconnect when somebody tries to send again.
- #
- # NOTE: we DO propagate the error, though, because we cannot simply
- # ignore the error... the caller will know if they can retry.
- if self.debuglevel > 0:
- print "send:", repr(value)
- try:
- blocksize = 8192
- if hasattr(value, 'read') :
- if hasattr(value, 'seek'):
- value.seek(0)
- if self.debuglevel > 0:
- print "sendIng a read()able"
- data = value.read(blocksize)
- while data:
- self.sock.sendall(data)
- data = value.read(blocksize)
- elif hasattr(value, 'next'):
- if hasattr(value, 'reset'):
- value.reset()
- if self.debuglevel > 0:
- print "sendIng an iterable"
- for data in value:
- self.sock.sendall(data)
- else:
- self.sock.sendall(value)
- except socket.error, v:
- if v[0] == 32: # Broken pipe
- self.close()
- raise
-
-class StreamingHTTPConnection(_StreamingHTTPMixin, httplib.HTTPConnection):
- """Subclass of `httplib.HTTPConnection` that overrides the `send()` method
- to support iterable body objects"""
-
-class StreamingHTTPRedirectHandler(urllib2.HTTPRedirectHandler):
- """Subclass of `urllib2.HTTPRedirectHandler` that overrides the
- `redirect_request` method to properly handle redirected POST requests
-
- This class is required because python 2.5's HTTPRedirectHandler does
- not remove the Content-Type or Content-Length headers when requesting
- the new resource, but the body of the original request is not preserved.
- """
-
- handler_order = urllib2.HTTPRedirectHandler.handler_order - 1
-
- # From python2.6 urllib2's HTTPRedirectHandler
- def redirect_request(self, req, fp, code, msg, headers, newurl):
- """Return a Request or None in response to a redirect.
-
- This is called by the http_error_30x methods when a
- redirection response is received. If a redirection should
- take place, return a new Request to allow http_error_30x to
- perform the redirect. Otherwise, raise HTTPError if no-one
- else should try to handle this url. Return None if you can't
- but another Handler might.
- """
- m = req.get_method()
- if (code in (301, 302, 303, 307) and m in ("GET", "HEAD")
- or code in (301, 302, 303) and m == "POST"):
- # Strictly (according to RFC 2616), 301 or 302 in response
- # to a POST MUST NOT cause a redirection without confirmation
- # from the user (of urllib2, in this case). In practice,
- # essentially all clients do redirect in this case, so we
- # do the same.
- # be conciliant with URIs containing a space
- newurl = newurl.replace(' ', '%20')
- newheaders = dict((k, v) for k, v in req.headers.items()
- if k.lower() not in (
- "content-length", "content-type")
- )
- return urllib2.Request(newurl,
- headers=newheaders,
- origin_req_host=req.get_origin_req_host(),
- unverifiable=True)
- else:
- raise urllib2.HTTPError(req.get_full_url(), code, msg, headers, fp)
-
-class StreamingHTTPHandler(urllib2.HTTPHandler):
- """Subclass of `urllib2.HTTPHandler` that uses
- StreamingHTTPConnection as its http connection class."""
-
- handler_order = urllib2.HTTPHandler.handler_order - 1
-
- def http_open(self, req):
- """Open a StreamingHTTPConnection for the given request"""
- return self.do_open(StreamingHTTPConnection, req)
-
- def http_request(self, req):
- """Handle a HTTP request. Make sure that Content-Length is specified
- if we're using an interable value"""
- # Make sure that if we're using an iterable object as the request
- # body, that we've also specified Content-Length
- if req.has_data():
- data = req.get_data()
- if hasattr(data, 'read') or hasattr(data, 'next'):
- if not req.has_header('Content-length'):
- raise ValueError(
- "No Content-Length specified for iterable body")
- return urllib2.HTTPHandler.do_request_(self, req)
-
-if hasattr(httplib, 'HTTPS'):
- class StreamingHTTPSConnection(_StreamingHTTPMixin,
- httplib.HTTPSConnection):
- """Subclass of `httplib.HTTSConnection` that overrides the `send()`
- method to support iterable body objects"""
-
- class StreamingHTTPSHandler(urllib2.HTTPSHandler):
- """Subclass of `urllib2.HTTPSHandler` that uses
- StreamingHTTPSConnection as its http connection class."""
-
- handler_order = urllib2.HTTPSHandler.handler_order - 1
-
- def https_open(self, req):
- return self.do_open(StreamingHTTPSConnection, req)
-
- def https_request(self, req):
- # Make sure that if we're using an iterable object as the request
- # body, that we've also specified Content-Length
- if req.has_data():
- data = req.get_data()
- if hasattr(data, 'read') or hasattr(data, 'next'):
- if not req.has_header('Content-length'):
- raise ValueError(
- "No Content-Length specified for iterable body")
- return urllib2.HTTPSHandler.do_request_(self, req)
-
-
-def get_handlers():
- handlers = [StreamingHTTPHandler, StreamingHTTPRedirectHandler]
- if hasattr(httplib, "HTTPS"):
- handlers.append(StreamingHTTPSHandler)
- return handlers
-
-def register_openers():
- """Register the streaming http handlers in the global urllib2 default
- opener object.
-
- Returns the created OpenerDirector object."""
- opener = urllib2.build_opener(*get_handlers())
-
- urllib2.install_opener(opener)
-
- return opener
View
24 setup.py
@@ -1,24 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
-try:
- from setuptools import setup
-except ImportError:
- from distutils.core import setup
-
-
-required = ["poster"]
-
-
-setup(
- name='iron-worker',
- version='0.0.13',
- description='The Python client for IronWorker, a cloud service for background processing.',
- long_description='IronWorker is a background processing and task queuing system that lets your applications use the cloud to do their heavy lifting. This is the Python library to interface with the API.',
- keywords=['iron', 'ironio', 'iron.io', 'iron-io', 'ironworker', 'iron-worker', 'iron_worker', 'worker', 'cloud', 'task queue', 'background processing'],
- author='Iron.io',
- author_email="support@iron.io",
- url='http://iron.io/products/worker',
- install_requires=required,
- packages=[ 'iron_worker' ],
-)
View
193 test.py
@@ -1,193 +0,0 @@
-from iron_worker import *
-import unittest
-import ConfigParser
-import time
-
-
-class TestIronWorker(unittest.TestCase):
-
- def setUp(self):
- config = ConfigParser.RawConfigParser()
- config.read('config.ini')
- self.token = config.get("IronWorker", "token")
- self.project_id = config.get("IronWorker", "project_id")
- self.new_project_id = "New Project Id"
-
- self.code_name = "test%d" % time.time()
-
- self.worker = IronWorker(token=self.token, project_id=self.project_id)
-
- IronWorker.zipDirectory(destination="test.zip", overwrite=True,
- directory="testDir")
- response = self.worker.postCode(name=self.code_name,
- runFilename="testDir/hello.py", zipFilename="test.zip")
-
- def test_setProject(self):
- self.assertNotEqual(self.worker.project_id, self.new_project_id)
-
- self.worker.setProject(self.new_project_id)
- self.assertEqual(self.worker.project_id, self.new_project_id)
-
- self.worker.setProject(self.project_id)
- self.assertEqual(self.worker.project_id, self.project_id)
-
- def test_headers(self):
- self.assertEqual(self.worker.headers['Accept'], "application/json")
- self.assertEqual(self.worker.headers['Accept-Encoding'],
- "gzip, deflate")
- self.assertEqual(self.worker.headers['User-Agent'],
- "IronWorker Python v0.3")
-
- def test_postCode(self):
- IronWorker.createZip(destination="test.zip", overwrite=True,
- files=["testDir/hello.py"])
- response = self.worker.postCode(name=self.code_name,
- runFilename="testDir/hello.py", zipFilename="test.zip")
-
- codes = self.worker.getCodes()
- code_names = []
- for code in codes:
- code_names.append(code["name"])
- self.assertIn(self.code_name, code_names)
-
- def test_postZip(self):
- IronWorker.zipDirectory(directory="testDir", destination="test.zip",
- overwrite=True)
- response = self.worker.postCode(name=self.code_name,
- runFilename="testDir/hello.py", zipFilename="test.zip")
-
- codes = self.worker.getCodes()
-
- code = self.worker.getCodeDetails(code_id=codes[0]['id'])
- self.assertEqual(codes[0]['id'], code['id'])
-
- def test_getCodeDetails(self):
- IronWorker.createZip(destination="test.zip", files=["test.py"],
- overwrite=True)
- response = self.worker.postCode(name=self.code_name,
- runFilename="test.py", zipFilename="test.zip")
-
- codes = self.worker.getCodes()
-
- code = self.worker.getCodeDetails(code_id=codes[0]['id'])
- self.assertEqual(codes[0]['id'], code['id'])
-
- def test_postTask(self):
- payload = {
- "dict": {"a": 1, "b": 2},
- "var": "alpha",
- "list": ['apples', 'oranges', 'bananas']
- }
- resp = self.worker.postTask(name=self.code_name, payload=payload)
-
- self.assertEqual(len(resp['tasks']), 1)
-
- task_id = resp['tasks'][0]['id']
-
- tasks = self.worker.getTasks()
- task_ids = []
- for task in tasks:
- task_ids.append(task['id'])
-
- self.assertIn(task_id, task_ids)
-
- def test_getTaskDetails(self):
- payload = {
- "dict": {"a": 1, "b": 2},
- "var": "alpha",
- "list": ['apples', 'oranges', 'bananas']
- }
- resp = self.worker.postTask(name=self.code_name, payload=payload)
-
- self.assertEqual(len(resp['tasks']), 1)
-
- task_id = resp['tasks'][0]['id']
-
- tasks = self.worker.getTasks()
- task_ids = []
- for task in tasks:
- task_ids.append(task['id'])
-
- self.assertIn(task_id, task_ids)
-
- tasks = self.worker.getTasks()
- task_id = tasks[0]['id']
-
- task = self.worker.getTaskDetails(task_id=task_id)
-
- self.assertEqual(task_id, task['id'])
-
- def test_cancelTask(self):
- tasks = self.worker.getTasks()
-
- for task in tasks:
- self.worker.cancelTask(task_id=task['id'])
-
- new_tasks = self.worker.getTasks()
- real_tasks = []
- for task in new_tasks:
- if task['status'] not in ['cancelled', 'error']:
- real_tasks.append(task)
- self.assertEqual(len(real_tasks), 0)
-
- def test_postSchedule(self):
- schedule_id = self.worker.postSchedule(name=self.code_name, delay=120)
-
- schedules = self.worker.getSchedules()
- schedule_ids = []
- for schedule in schedules:
- schedule_ids.append(schedule['id'])
-
- self.assertIn(schedule_id, schedule_ids)
-
- def test_postScheduleAndPayload(self):
- schedule_id = self.worker.postSchedule(name=self.code_name, delay=120,
- payload={"foo": "bar"})
-
- schedules = self.worker.getSchedules()
- schedule_ids = []
- for schedule in schedules:
- schedule_ids.append(schedule['id'])
-
- self.assertIn(schedule_id, schedule_ids)
-
- def test_postAdvancedSchedule(self):
- start_at = time.gmtime(time.time() + 3600) # one hour from now
- schedule_id = self.worker.postSchedule(
- name="advanced_%s" % self.code_name,
- payload={"schedule": "AWESOME SCHEDULE!"},
- code_name=self.code_name, start_at=start_at, run_every=3600,
- run_times=8)
-
- schedules = self.worker.getSchedules()
- schedule_ids = []
- for schedule in schedules:
- schedule_ids.append(schedule['id'])
-
- self.assertIn(schedule_id, schedule_ids)
-
- def test_cancelSchedule(self):
- schedules = self.worker.getSchedules()
-
- for schedule in schedules:
- self.worker.cancelSchedule(schedule_id=schedule['id'])
-
- new_schedules = self.worker.getSchedules()
- real_schedules = []
- for schedule in new_schedules:
- if schedule['status'] not in ['cancelled', 'error']:
- real_schedules.append(schedule)
-
- self.assertEqual(len(real_schedules), 0)
-
- def test_deleteCode(self):
- codes = self.worker.getCodes()
-
- for code in codes:
- self.worker.deleteCode(code_id=code['id'])
-
- new_codes = self.worker.getCodes()
- self.assertEqual(len(new_codes), 0)
-
-if __name__ == '__main__':
- unittest.main()
View
1 testDir/hello.py
@@ -1 +0,0 @@
-print "HELLO WORLD"

0 comments on commit 96eac11

Please sign in to comment.
Something went wrong with that request. Please try again.