Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 91 lines (67 sloc) 3.781 kb
2a78533 Chris O'Hara Fixed IO cases where job is to run once or forever
authored
1 # What is [node.io](http://node.io/)?
8f797fb Chris O'Hara Updated README and documentation
authored
2
bcab815 Chris O'Hara Updated README
authored
3 node.io can streamline the process of:
77811bb Chris O'Hara Updated README
authored
4
bcab815 Chris O'Hara Updated README
authored
5 - Parsing / filtering / sanitizing large amounts of data
8f797fb Chris O'Hara Updated README and documentation
authored
6 - MapReduce
bcab815 Chris O'Hara Updated README
authored
7 - Scraping data from the web using familiar CSS selectors and traversal methods
b3309be Chris O'Hara Updated README and API
authored
8 - Scraping web data through a proxy
bcab815 Chris O'Hara Updated README
authored
9 - Transforming data from one format to another, e.g. from CSV => a database
b3309be Chris O'Hara Updated README and API
authored
10 - Recursively load all files in a directory and its subdirs and execute a command on each
bcab815 Chris O'Hara Updated README
authored
11 - + much more..
5fab6c3 Chris O'Hara Updated README
authored
12
e535de9 Chris O'Hara Updated README
authored
13 ## Why node.io?
58de877 Chris O'Hara Updated README
authored
14
a3e56c1 Chris O'Hara Updated README and made some JSLint fixes
authored
15 - Create modular and extensible jobs for scraping and processing data
bcab815 Chris O'Hara Updated README
authored
16 - Jobs are written in Javascript or Coffeescript and run in Node.js - jobs are concise, asynchronous and _FAST_
17 - Seamlessly speed up execution by distributing work among child processes and other servers (soon)
8f797fb Chris O'Hara Updated README and documentation
authored
18 - Easily handle a variety of input / output situations
bc05fa1 Chris O'Hara Updated README
authored
19 * Reading / writing lines to and from files
b3309be Chris O'Hara Updated README and API
authored
20 * Traversing files in a directory
8f797fb Chris O'Hara Updated README and documentation
authored
21 * Reading / writing rows to and from a database
b3309be Chris O'Hara Updated README and API
authored
22 * STDIN / STDOUT / Custom streams
23 * Piping between other node.io jobs
24ee8df Chris O'Hara Updated documentation
authored
24 * Any combination of the above, or your own IO
e43509d Chris O'Hara Updated README
authored
25 - Includes a robust framework for scraping and selecting web data
b3309be Chris O'Hara Updated README and API
authored
26 - Support for a variety of proxies when scraping web data
a3e56c1 Chris O'Hara Updated README and made some JSLint fixes
authored
27 - Includes a data validation and sanitization framework
28 - Provides support for retries, timeouts, dynamically adding input, etc.
bc05fa1 Chris O'Hara Updated README
authored
29
7552dac Chris O'Hara Updated README
authored
30 ## Installation
31
32 To install node.io, use [npm](http://github.com/isaacs/npm):
33
34 $ npm install node.io
35
36 For usage details, run
37
38 $ node.io --help
39
01eb509 Chris O'Hara Updated README
authored
40 ## Documentation
bc05fa1 Chris O'Hara Updated README
authored
41
b3309be Chris O'Hara Updated README and API
authored
42 To get started, see the [documentation](https://github.com/chriso/node.io/blob/master/docs/README.md), [API](https://github.com/chriso/node.io/blob/master/docs/api.md), and [examples](https://github.com/chriso/node.io/tree/master/examples/).
bc05fa1 Chris O'Hara Updated README
authored
43
5fab6c3 Chris O'Hara Updated README
authored
44 Better documentation will be available once I have time to write it.
bc05fa1 Chris O'Hara Updated README
authored
45
22f8535 Chris O'Hara Added Roadmap
authored
46 ## Roadmap
47
5fab6c3 Chris O'Hara Updated README
authored
48 - Fix up the [http://node.io/](http://node.io/) site
1080b2f Chris O'Hara Fixed README bug with markdown
authored
49 - Automatically handle HTTP codes, e.g. redirect on 3xx or call fail() on 4xx/5xx
22f8535 Chris O'Hara Added Roadmap
authored
50 - Nested requests inherit referrer / cookies if to the same domain
51 - Add more DOM selector / traversal methods
b3309be Chris O'Hara Updated README and API
authored
52 - Test proxy callbacks and write proxy documentation
22f8535 Chris O'Hara Added Roadmap
authored
53 - Add distributed processing
c1dc9da Chris O'Hara Added initial documentation
authored
54 - Installation without NPM (install.sh)
55 - Refactoring
fab397c Chris O'Hara Updated README
authored
56 - More tests / better test coverage
22f8535 Chris O'Hara Added Roadmap
authored
57
d9da9dc Chris O'Hara Updated README
authored
58 ## Credits
59
5aa225e Chris O'Hara Updated README
authored
60 node.io wouldn't be possible without
d9da9dc Chris O'Hara Updated README
authored
61
459525b Chris O'Hara Updated README
authored
62 - [ry's](https://github.com/ry) [node.js](http://nodejs.org/)
1f8d7e7 Chris O'Hara Updated README
authored
63 - [tautologistics'](https://github.com/tautologistics) [node-htmlparser](https://github.com/tautologistics/node-htmlparser)
64 - [harryf's](https://github.com/harryf) [soupselect](https://github.com/harryf/node-soupselect)
65 - [kriszyp's](https://github.com/kriszyp) [multi-node](https://github.com/kriszyp/multi-node)
d9da9dc Chris O'Hara Updated README
authored
66
67 ## License
68
f11a5bb Chris O'Hara Updated README
authored
69 (MIT License)
70
71 Copyright (c) 2010 Chris O'Hara <cohara87@gmail.com>
72
73 Permission is hereby granted, free of charge, to any person obtaining
74 a copy of this software and associated documentation files (the
75 "Software"), to deal in the Software without restriction, including
76 without limitation the rights to use, copy, modify, merge, publish,
77 distribute, sublicense, and/or sell copies of the Software, and to
78 permit persons to whom the Software is furnished to do so, subject to
79 the following conditions:
80
81 The above copyright notice and this permission notice shall be
82 included in all copies or substantial portions of the Software.
83
84 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
85 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
86 MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
87 NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
88 LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
89 OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
90 WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Something went wrong with that request. Please try again.