FAQ.rst

riko FAQ

Index

What pipes are available | What file types are supported | What protocols are supported

What pipes are available?

Overview

riko's available pipes are outlined below [1]:

Pipe name	Pipe type	Pipe sub-type	Pipe description
count	operator	aggregator	counts the number of items in a feed
csv	processor	source	parses a csv file to yield items
currencyformat	processor	transformer	formats a number to a given currency string
dateformat	processor	transformer	formats a date
exchangerate	processor	transformer	retrieves the current exchange rate for a given currency pair
feedautodiscovery	processor	source	fetches and parses the first feed found on a site
fetch	processor	source	fetches and parses a feed to return the entries
fetchdata	processor	source	fetches and parses an XML or JSON file to return the feed entries
fetchpage	processor	source	fetches the content of a given web site as a string
fetchsitefeed	processor	source	fetches and parses the first feed found on a site
fetchtext	processor	source	fetches and parses a text file
filter	operator	composer	extracts items matching the given rules
hash	processor	transformer	hashes the field of a feed item
input	processor	source	prompts for text and parses it into a variety of different types, e.g., int, bool, date, etc
itembuilder	processor	source	builds an item
join	operator	aggregator	perform a SQL like join on two feeds
regex	processor	transformer	replaces text in fields of a feed item using regexes
refind	processor	transformer	finds text located before, after, or between substrings using regular expressions
rename	processor	transformer	renames or copies fields in a feed item
`reverse`_	operator	composer	reverses the order of source items in a feed
rssitembuilder	processor	source	builds an rss item
simplemath	processor	transformer	performs basic arithmetic, such as addition and subtraction
slugify	operator	transformer	slugifies text
sort	operator	composer	sorts a feed according to a specified key
split	operator	composer	splits a feed into identical copies
strconcat	processor	transformer	concatenates strings
strfind	processor	transformer	finds text located before, after, or between substrings
strreplace	processor	transformer	replaces the text of a field of a feed item
strtransform	processor	transformer	performs string transformations on the field of a feed item
subelement	processor	transformer	extracts sub-elements for the item of a feed
substr	processor	transformer	returns a substring of a field of a feed item
sum	operator	aggregator	sums a field of items in a feed
tail	operator	composer	truncates a feed to the last N items
timeout	operator	composer	returns items from a stream until a certain amount of time has passed
tokenizer	processor	transformer	splits a string by a delimiter
truncate	operator	composer	returns a specified number of items from a feed
union	operator	composer	merges multiple feeds together
uniq	operator	composer	filters out non unique items according to a specified field
urlbuilder	processor	transformer	builds a url
urlparse	processor	transformer	parses a URL into its six components
xpathfetchpage	processor	source	fetches the content of a given website as DOM nodes or a string
yql	processor	source	fetches the result of a given YQL query

Args

riko pipes come in two flavors; operator and processor [2]. operator``s operate on an entire ``stream at once. Example operator``s include ``pipecount, pipefilter, and pipereverse.

>>> from riko.modules.reverse import pipe
>>>
>>> stream = [{'title': 'riko pt. 1'}, {'title': 'riko pt. 2'}]
>>> next(pipe(stream))
{'title': 'riko pt. 2'}

processor``s process individual ``items. Example processor``s include ``pipefetchsitefeed, pipehash, pipeitembuilder, and piperegex.

>>> from riko.modules.hash import pipe
>>>
>>> item = {'title': 'riko pt. 1'}
>>> stream = pipe(item, field='title')
>>> next(stream)
{'title': 'riko pt. 1', 'hash': 2853617420L}

Kwargs

The following table outlines the available kwargs.

kwarg	type	description	default
conf	dict	The pipe configuration	varies
extract	str	The key with which to get a value from conf	None
listize	bool	Ensure that an extract value is list-like	False
pdictize	bool	Convert conf / extract to a DotDict instance	varies
objectify	bool	Convert conf to an Objectify instance	varies
ptype	str	Used to convert conf items to a specific type.	pass
dictize	bool	Convert the input item to a DotDict instance	True
field	str	The key with which to get a value from the input	None
ftype	str	Converts the input item to a specific type	pass
count	str	The output count	all
assign	str	Attribute used to assign output	varies
emit	bool	Return the output as is (don't assign)	varies
skip_if	func	Determines if processing should be skipped	None
inputs	dict	Values to be used in place of prompting the user	None

Notes

[1]	See Design Principles for explanation on pipe types and sub-types

[2]	See Alternate workflow creation for pipe composition examples

What file types are supported?

File types that riko supports are outlined below:

File type	Recognized extension(s)	Supported pipes
HTML	html	feedautodiscovery, fetchpage, fetchsitefeed
XML	xml	fetch, fetchdata
JSON	json	fetchdata
Comma separated file	csv, tsv	csv

What protocols are supported?

Protocols that riko supports are outlined below:

Protocol	example
http	http://google.com
https	https://github.com/reubano/feed
file	file:///Users/reubano/Downloads/feed.xml