AOPPA -- A program for extracting the AnarchyOnline resource database
$Id: README 1112 2009-03-04 08:06:08Z os $

Copyright (C) 2002-2009 Oskari Saarenmaa <auno@auno.org>.
All rights reserved.

Licensed under the GNU General Public License version 3.

============================================================================

Background,

'aoppa' is the current database parser I use to parse Funcom's CTree
resource database and store it in a PostgreSQL database for use in my
website (auno.org).  This is the second, or perhaps third, generation of the
parser, the first one was written entirely in PHP and had tons of problems.
Including the fact that it took several hours to parse the database, in
comparison it takes only about 5 seconds for aoppa to parse the current
shadowlands database.

As you can probably tell, this is mostly Unix stuff -- I don't do any
development on Windows.  The parser can only read a special data format that
is produced by a Windows program that actually dumps the Anarchy Online
CTree database. The program is called 'adbd' and the source code for it is
available at http://auno.org/dev/adbd/ .  I produced the program on a Linux
system with the MinGW32 cross-compiler and run it under Wine.

It would probably be trivial for someone familiar with Windows to rewrite
such a program, the only thing it has to do is fetch records from the CTree
database and then store them in a file in the format
AOID: <id> LEN: <length>\n==========\n<data>\n==========\n


Plugins,

Aoppa supports plugins, in fact it can't do anything at all without at least
one plugin loaded.  Currently there exist two kinds of plugins, 'input' and
'output' ones.  Input plugins are used to feed data to the parser, and
output plugins are used to store the data structures populated by the
parser.

Aoppa ships with four default plugins, input-mmap and input-gzip for reading
data from files generated by ADBD (see above), and output-xml and output-ign
for writing XML and IGN-bot format output files.  There's also the
aunoorg-output plugin that I will discuss a bit later.

The plugin system is quite simple to use if you wish to create new plugins,
you should just add a public structure of the type aoppa_plugin_t called
'aoppa_plugin' in your code and then compile it as a shared object.  The
aoppa_plugin structure contains the following elements (in order):
  api version       - for specifying the aoppa API version
  plugin type       - can be AOPPA_INPUT, AOPPA_OUTPUT or both
  init function     - function to be called when plugin is loaded
  finish function   - function to be called when plugin is unloaded
  store function    - function to store item_t structures
  flush function    - function to be called when done with input file
  parse function    - function to be called with a file to parse
  description       - a const char * with a short description of the plugin
All of these, except the version, are optional and can be NULL if the
specified functionality is not required or wanted in the plugin.
See aoppa.h or the supplied plugins for more information.

The obvious plugin that people would probably want, and the one I would like
to see developed is one for reading data directly from the AO CTree
database.  That would of course require someone to port aoppa to Windows.


Auno.org,

There are several files in the repository that are directly related to my
Auno.org website.  These include a special PostgreSQL output plugin, the sql
schemas and some tools for creating additional tables out of the data in
aodb_* tables.  Some of them might be interesting for others who want to
develop an AO database website, most are probably uninteresting, though.


Getting started,

Get the latest .tar.gz package from http://auno.org/dev/aoppa/ and gunzip
and untar it in your favourite Unix system.  Then run 'make' to generate
aoppa and the plugins and run:
./aoppa -p input-gzip.so -p output-xml.so <input files>
Where the <input files> are your database dumps generated by ADBD.  If you
don't want or are unable to run ADBD you can also fetch 18.0.0 item and nano
dumps from http://auno.org/dev/aoppa/items-18000000ep1.dat.gz and
http://auno.org/dev/aoppa/nanos-18000000ep1.dat.gz

After running the command you can just watch the XML output scroll by...


Development,

I have been meaning to release this code for public for as long as I have
been working on it, but there's always been one or another thing that has
blocked the release.  I have wanted to fix some of the most obvious and
embarrassing bugs in the code before a release, but the past has proved that
the chances are that it'll never happen unless I just go ahead and give out
the current version of the code.

As you can see, I have chosen the GNU General Public License for the
project.  What it means is that you can freely copy, modify, sell, and
distribute aoppa, but you have to give the full source code with any
binaries you ship.  This includes the source code for any derivative works,
or anything that is linked with Aoppa.  This includes all plugins.

I would also like anyone who modifies the code to send diffs of their
modifications back to me, so I can update my version and post any
improvements on my website.  If there is a lot of interest in this we can
probably also setup a Sourceforge project.


Anything and everything else,

If you have any questions about this feel free to send me an email at
auno@auno.org or write in the forums at http://auno.org/

Cheers,
Auno