Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100755 88 lines (59 sloc) 3.704 kb
5528529 Chris O'Hara Updated README
authored
1 [node.io](http://node.io/) is a distributed data scraping and processing framework
5fab6c3 Chris O'Hara Updated README
authored
2
bcab815 Chris O'Hara Updated README
authored
3 - Jobs are written in Javascript or Coffeescript and run in Node.js - jobs are concise, asynchronous and _FAST_
60b3894 Chris O'Hara Updated README
authored
4 - Includes a robust framework for scraping, selecting and traversing data from the web
5 - Includes support for a variety of proxies
9eb941a Chris O'Hara Updated README
authored
6 - Includes a data validation and sanitization framework
5528529 Chris O'Hara Updated README
authored
7 - Easily handle a variety of input / output - files, databases, streams, stdin/stdout, etc.
8 - Speed up execution by distributing work across multiple processes and (soon) other servers
60b3894 Chris O'Hara Updated README
authored
9 - Provides support for retries, timeouts, dynamically adding input, etc.
10 - Manage & run jobs through a web interface
11 - Run encrypted jobs made with [packnode](https://github.com/chriso/packnode)
bc05fa1 Chris O'Hara Updated README
authored
12
7552dac Chris O'Hara Updated README
authored
13 ## Installation
14
60b3894 Chris O'Hara Updated README
authored
15 Note: if you do not have [Node.JS](http://nodejs.org/) or [npm](http://github.com/isaacs/npm) installed, [see this page](https://github.com/chriso/node.io/wiki/Installation).
16
17 To install node.io, run
7552dac Chris O'Hara Updated README
authored
18
19 $ npm install node.io
20
9fb3af5 Chris O'Hara Updated README
authored
21 To run tests
22
23 $ make test
60b3894 Chris O'Hara Updated README
authored
24
9fb3af5 Chris O'Hara Updated README
authored
25 For usage details
7552dac Chris O'Hara Updated README
authored
26
27 $ node.io --help
60b3894 Chris O'Hara Updated README
authored
28
b31d561 Chris O'Hara Updated README
authored
29 ## Getting started
bc05fa1 Chris O'Hara Updated README
authored
30
b3309be Chris O'Hara Updated README and API
authored
31 To get started, see the [documentation](https://github.com/chriso/node.io/blob/master/docs/README.md), [API](https://github.com/chriso/node.io/blob/master/docs/api.md), and [examples](https://github.com/chriso/node.io/tree/master/examples/).
bc05fa1 Chris O'Hara Updated README
authored
32
6af0c26 Chris O'Hara Added async complete() and loading jobs from ~/.node_modules
authored
33 *Note: node.io is a _BETA_ release. There will no doubt be some bugs and oddities.*
b31d561 Chris O'Hara Updated README
authored
34
35 Check [@nodeio](http://twitter.com/nodeio) or [http://node.io/](http://node.io/) for updates.
e7281e2 Chris O'Hara Updated README
authored
36
22f8535 Chris O'Hara Added Roadmap
authored
37 ## Roadmap
38
9eb941a Chris O'Hara Updated README
authored
39 - More tests & better coverage
40 - Improve documentation and examples
41 - Installation without NPM
5fab6c3 Chris O'Hara Updated README
authored
42 - Fix up the [http://node.io/](http://node.io/) site
e7281e2 Chris O'Hara Updated README
authored
43 - Add more DOM [selector](http://api.jquery.com/category/selectors/) / [traversal](http://api.jquery.com/category/traversing/) methods
22f8535 Chris O'Hara Added Roadmap
authored
44 - Add distributed processing
ddaff54 Chris O'Hara Updated roadmap
authored
45 - Speed improvements
22f8535 Chris O'Hara Added Roadmap
authored
46
4df5c5a Chris O'Hara 0.2.0 release
authored
47 [history.md](https://github.com/chriso/node.io/blob/master/HISTORY.md) lists recent changes.
48
8d30aba Chris O'Hara Updated README
authored
49 ## Contributing
50
51 If you find a bug, please report the issue [here](https://github.com/chriso/node.io/issues).
52
4df5c5a Chris O'Hara 0.2.0 release
authored
53 If you want to contribute, please [fork/pull](https://github.com/chriso/node.io/fork).
8d30aba Chris O'Hara Updated README
authored
54
d9da9dc Chris O'Hara Updated README
authored
55 ## Credits
56
5aa225e Chris O'Hara Updated README
authored
57 node.io wouldn't be possible without
d9da9dc Chris O'Hara Updated README
authored
58
459525b Chris O'Hara Updated README
authored
59 - [ry's](https://github.com/ry) [node.js](http://nodejs.org/)
1f8d7e7 Chris O'Hara Updated README
authored
60 - [tautologistics'](https://github.com/tautologistics) [node-htmlparser](https://github.com/tautologistics/node-htmlparser)
61 - [harryf's](https://github.com/harryf) [soupselect](https://github.com/harryf/node-soupselect)
62 - [kriszyp's](https://github.com/kriszyp) [multi-node](https://github.com/kriszyp/multi-node)
d9da9dc Chris O'Hara Updated README
authored
63
64 ## License
65
f11a5bb Chris O'Hara Updated README
authored
66 (MIT License)
67
68 Copyright (c) 2010 Chris O'Hara <cohara87@gmail.com>
69
70 Permission is hereby granted, free of charge, to any person obtaining
71 a copy of this software and associated documentation files (the
72 "Software"), to deal in the Software without restriction, including
73 without limitation the rights to use, copy, modify, merge, publish,
74 distribute, sublicense, and/or sell copies of the Software, and to
75 permit persons to whom the Software is furnished to do so, subject to
76 the following conditions:
77
78 The above copyright notice and this permission notice shall be
79 included in all copies or substantial portions of the Software.
80
81 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
82 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
83 MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
84 NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
85 LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
86 OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
87 WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Something went wrong with that request. Please try again.