Web data mining for Lua
=======================

Wdm is a collection of simple utility functions for web page retrieval and processing
that I use for data mining purposes, like gathering the history of prices of my
favorite products. It provides very simple interface over other libraries, mainly
cURL and HTML parsing. Maybe these will be useful to someone else...

Copyright 2010(c) Michal Kottman
Licensed under the MIT License.

Dependencies
------------

Wdm depends on cURL for http retrieval, all other dependencies are optional,
in case a library is not found, the functions from it will simply not be avaliable.

* cURL binding - either luacurl [1] or Lua-cURL [2] should work
* luasql [3] - uses sqlite3 for simple database
* iconv [4] - for character conversion into utf8
* tidy [5] - bindings for HTML Tidy, to be able to parse those badly-written sites
* bz2 [6] - bindings to bzip2 for compression of cached pages

[1] http://luacurl.luaforge.net/
[2] http://luaforge.net/projects/lua-curl/
[3] http://www.keplerproject.org/luasql/
[4] http://luaforge.net/projects/lua-iconv/
[5] http://github.com/mkottman/tidy
[6] http://github.com/mkottman/lua-bz2

Installation
------------

Just copy wdm.lua into your package.path

Example of use
--------------

This example retrieves the first 10 pages of Google search for "lua"
and prints the links and text. Shows how to use the functions get(), toTidy(),
text() and getElements().

	require 'wdm'

	-- start page
	local page = 'http://www.google.sk/search?q=lua'
	for i=1,10 do
		-- retrieve the page
		local src = get(page)
		-- convert it to table
		local t = toTidy(src)
		
		-- get links with class "l"
		local res = getElements(t, 'tag=="a" and class=="l"')
		for _,link in ipairs(res) do
			-- print the link url and link text
			print(link.href, text(link))
		end
		
		-- get the element, whose first child is the 'next' image
		local nxt = getElements(t, 'self[1].tag=="img" and self[1].src=="nav_next.gif"')[1]
		-- construct the next url
		page = 'http://www.google.com' .. nxt.href
	end

Documentation
-------------

TODO later