Permalink
Browse files

Removed the duplicates builtin

  • Loading branch information...
1 parent 4139a99 commit 08c57bc05c790d545412d675a3a048a6f22b2889 @chriso chriso committed Mar 8, 2011
Showing with 12 additions and 68 deletions.
  1. +12 −25 builtin/README.md
  2. +0 −43 builtin/duplicates.coffee
View
37 builtin/README.md
@@ -5,17 +5,17 @@ Node.io comes with several built-in modules which can be accessed through the co
To run a built-in module, run
$ node.io [MODULE] [ARGS]
-
+
To see usage details, run
-
+
$ node.io [MODULE] help
### digest
This module calculates the hash/checksum of each element of input. Available hashes are [md5, crc32, sha1, sha256, sha512, ...]
Example 1 - find the MD5 hash of a string
-
+
$ echo "this is a string" | node.io digest md5
=> b37e16c620c055cf8207b999e3270e9b
@@ -25,15 +25,15 @@ This module checks a URL's Google pagerank (rate limits apply)
Example 1 - find the pagerank of mastercard.com
- $ echo "mastercard.com" | node.io pagerank
+ $ echo "mastercard.com" | node.io pagerank
=> mastercard.com,7
-
+
### resolve
This module provides DNS resolution utilities
Example 1 - resolve domains and output "domain,ip"
-
+
$ node.io resolve < domains.txt
Example 2 - return domains that do not resolve (potentially available)
@@ -45,9 +45,9 @@ Example 3 - return domains that do resolve
$ node.io resolve found < domains.txt
Example 4 - return unique IPs
-
+
$ node.io resolve ips < domains.txt
-
+
### statuscode
Makes a HEAD request to each URL of input and returns the status code
@@ -87,34 +87,21 @@ Example 1 - remove lines that **do not** match a filter
Example 2 - output lines that do not match a filter (remove valid lines)
$ node.io validate not [FILTER] < list.txt
-
+
### eval
This module evaluates an expression on each line of input and emits the result (unless the result is null)
Example 1 - remove empty lines
-
+
$ node.io -s eval "input.trim() != '' ? input : null" < input.txt > modified.txt
-
+
Example 2 - convert a TSV (tab separated file) to CSV
-
+
$ node.io -s eval "input.split('\t').join(',')" < data.tsv > data.csv
### word_count
This module uses map/reduce to count word occurrences in a file
$ node.io word_count < input.txt
-
-### duplicates
-
-This module can find or remove duplicates from a list
-
-Example 1 - remove duplicates from a list and output unique lines
-
- $ node.io duplicates < list.txt
-
-Example 2 - to output duplicate lines
-
- $ node.io duplicates find < list.txt
-
View
43 builtin/duplicates.coffee
@@ -1,43 +0,0 @@
-usage = '''
-This module can find/remove duplicates in a list
-
- 1. To remove duplicates from a list and output unique lines:
- $ node.io duplicates < list.txt
-
- 2. To output lines that appear more than once:
- $ node.io duplicates find < list.txt
-'''
-
-nodeio = require 'node.io'
-
-seen_lines = []
-emitted_lines = []
-
-class RemoveDuplicates extends nodeio.JobClass
- reduce: (lines) ->
- for line in lines
- if not line in seen_lines
- @emit line
- seen_lines.push line
-
-class FindDuplicates extends nodeio.JobClass
- reduce: (lines) ->
- for line in lines
- if line in seen_lines
- if not line in emitted_lines
- @emit line
- emitted_lines.push line
- else
- seen_lines.push line
-
-class UsageDetails extends nodeio.JobClass
- input: ->
- @status usage
- @exit()
-
-@class = RemoveDuplicates
-@job = {
- remove: new RemoveDuplicates()
- find: new FindDuplicates()
- help: new UsageDetails()
-}

0 comments on commit 08c57bc

Please sign in to comment.