Update the readme.txt somewhat by deleting old and outdated content

This likely needs more improvements to make the readme actually useful.
But for now this mostly removes parts of the readme that were totally
out of date and confusing.

readme.txt

@@ -3,92 +3,94 @@ mod_tile
 A program to efficiently render and serve map tiles for
 www.openstreetmap.org map using Apache and Mapnik.
 
-Note: This program is very much still in development
-it has numerous hard coded paths and options which need
-to be made user configurable options. You may need to
-change these to fit your local environment.
-
 Requirements
 ============
 OSM map data imported into PostgreSQL using osm2pgsql
 Mapnik renderer along with the OSM.xml file and map
 symbols, world_boundaries shapefiles. Apache with
 development headers for APR module development.
 
-iniParser
-=========
-renderd uses the ini config file parser from
-http://ndevilla.free.fr/iniparser/ . To make it easier to
-compile, the full distribution is included in the
-subdirectory "iniparser3.0b". Go there and type "make"
-to generate the library before compiling renderd.
-
-Updating From Previous Version
-==============================
-The new version of mod_tile uses a slightly different
-directory hierarchy from the previous version.  In order
-to preserve the previously built tiles you will need to
-move them to:
-
-/var/lib/mod_tile/Default/[Z]/nnn/nnn/nnn/nnn/nnn.png
-
-You will also need to reconfigure your http configuration
-to use the new apache directives and to create an 
-/etc/renderd.conf file. See the example renderd.conf and
-mod_tile.conf for details.
-
 Tile Rendering
 ==============
+
 The rendering is implemented in a multithreaded process
-called renderd which opens a unix socket and listens for
-requests to render tiles. It uses Mapnik to render tiles
-using the rendering rules defined in the configuration file
-/etc/renderd.conf
-
-The render daemon implements a queueing mechanism which
-can render foreground requests (for new tiles being viewed)
-and background requests (updating tiles which have expired).
-The size of the queue and the number of threads is determined
+called renderd which opens either a unix or tcp socket
+and listens for requests to render tiles. It uses Mapnik
+to render tiles using the rendering rules defined in
+the configuration file /etc/renderd.conf. Its configuration
+also allows to specify the number of rendering
+threads.
+
+The render daemon implements a queuing mechanism with multiple
+priority levels to provide an as up-to-date viewing experience
+given the available rendering resources. The highest priority
+is for on the fly rendering of tiles not yet in the tile cache,
+two priority levels for re-rendering out of date tiles on the fly
+and two background batch rendering queues. The on the fly rendering
+queues are limited to a short 32 metatile size to minimize latency.
+The size of the main background queue is determined
 at compile time, see: render_config.h
 
 Tile serving
 ============
-To avoid problems with directories becoming too large the
-files are stored in a different layout to that presented
-by the web server. The tiles are now stored under
-/var/lib/mod_tile/[TileSetName]/[Z]/nnn/nnn/nnn/nnn/nnn.png
-
-Where nnn is derived from a combination of the X and Y
-OSM tile co-ordinates.
-
-Apache serves the files as if they were present
-under "/osm_tiles2/Z/X/Y.png" with the path being
-converted automatically.
 
 An Apache module called mod_tile enhances the regular
 Apache file serving mechanisms to provide:
 
-1) Tile expiry. It estimates when the tile is next
+1) When tiles have expired it requests the rendering
+daemon to render (or re-render) the tile.
+
+2) Remapping of the file path to the hashed layout
+
+3) Prioritizes rendering requests depending on the available
+resources on the server and how out of date they are.
+
+4) Use tile storage other than a plain posix file system.
+e.g it can store tiles in a ceph object store, or proxy them
+from another tile server.
+
+5) Tile expiry. It estimates when the tile is next
 likely to be rendered and adds the appropriate HTTP
-cache expiry headers
+cache expiry headers. This is a configurable heuristic.
+
+To avoid problems with directories becoming too large and to avoid
+too many tiny files.  Mod_tile / renderd store the rendered tiles
+in "meta tiles" in a special hashed directory structure. These combine
+8x8 actual tiles into a single metatile file.  This is a more efficient
+use of disk space and inodes. For example, many sea tiles are 103 bytes
+long. In the old scheme a meta tile of blank sea tiles would take
+64 inodes of 4kB each, a total of 256kB. In this optimized scheme it
+needs a single file of about 7kB. The metatiles are then stored
+in the following directory structure:
+/[base_dir]/[TileSetName]/[Z]/[xxxxyyyy]/[xxxxyyyy]/[xxxxyyyy]/[xxxxyyyy]/[xxxxyyyy].png
+Where base_dir is a configurable base path for all tiles. TileSetName
+is the name of the style sheet rendered. Z is the zoom level.
+[xxxxyyyy] is an 8 bit number, with the first 4 bits taken from the x
+coordinate and the second 4 bits taken from the y coordinate. This
+attempts to cluster 16x16 square of tiles together into a single sub
+directory for more efficient access patterns.
 
-2) When tiles have expired it requests the rendering
-daemon to render (or re-render) the tile.
+Apache serves the files as if they were present
+under "/[TileSetName]/Z/X/Y.png" with the path being
+converted automatically.
+
+Compiling
+=========
 
-3) Remapping of the file path to the hashed layout
+mod_tile and renderd utilize a number of third party, some of which it
+depends on and some provide optional features and are compiled in
+if the respective libraries are installed. Once the dependencies are installed
+you can compile and install mod_tile / renderd the usual way:
+
+./autogen.sh
+./configure
+make
+sudo make install
+sudo make install-mod_tile
 
-There is an attempt to make the mod_tile code aware of
-the load on the server so that it backs off the rendering
-if the machine is under heavy load.
 
 Setup
 =====
-Make sure you've read and implemented the things in the
-requirements section. Edit the paths in the source to
-match your local setup. Compile the code with make, and
-then make install (as root, to copy the mod_tile to the
-apache module directory).
-
 Create a new apache config file to load the module,
 e.g.
 
@@ -99,10 +101,10 @@ See the sample mod_tile.conf for details
 Edit /etc/renderd.conf to indicate the location of your
 mapnik style sheet and the uri you wish to use to access
 it.  You may configure up to 10 (by default) mapnik
-stylesheets - simply give each section a unique name and
+style sheets - simply give each section a unique name and
 enter the uri and style sheet path.
 
-Make sure the /var/lib/mod_tile directory is writeable by 
+Make sure the /var/lib/mod_tile directory is writable by 
 the user running the renderd process and create a file an
 empty file planet-import-complete in this folder.
 
@@ -114,7 +116,7 @@ Note: SELinux will prevent the mod_tile code from opening
 the unix-socket to the render daemon so must be disabled.
 
 Try loading a tile in your browser, e.g.
-http://localhost/osm_tiles2/0/0/0.png
+http://localhost/osm_tiles/0/0/0.png
 
 The render daemon should have produce a message like:
 
@@ -128,71 +130,23 @@ cease and then reload the tile. With a bit of luck you
 should see a tile of the world in your browser window.
 
 If this fails to happen check the http error log.  You can 
-increate the level of debugging using the LogLevel apache
+increase the level of debugging using the LogLevel apache
 directive.  If no log messages are shown check that you
 are accessing the correct virtual host - the new version
 of mod_tile is only installed on a single host by default.
 To install on multiple hosts either use ServerAlias or
 use the LoadTileConfigFile in each virtual host.
 
-To get a complete slippy map you should install a copy
-of the OpenLayers based OSM slippy map and point this to
-fetch tiles from http://localhost/osm_tiles2
 
-mysql2file
-==========
-This was written to export the existing OSM tiles from
-the Mysql database to the filesystem.
+Performance
+===========
+mod_tile is designed for high performance tile serving. If the
+underlying disk system allows it, it can easily provide > 10k tiles/s
+on a single serve.
 
+Rendering performance is mostly dependent on mapnik and postgis performance,
+however renderd tries to make sure it uses underlying hardware as efficiently
+as possible and scales well on multi core systems. Renderd also provides
+built-in features to scale to multi server rendering set-ups.
 
-Bugs
-====
-Too many hard coded options (need to be come module options or command
-line options to renderd).
-mod_tile uses many non-APR routines. It probably only works in Linux.
-If rendering daemon dies then all queued rendering requests are lost.
-Code has not been thoroughly tested.
 
-Performance
-===========
-The existing tile serving based on Apache + mod_ruby + cat_tile.rb
-+ Mysql manages to serve something in the region of 250 - 500 requests
-per second. Apache + mod_tile manages 2000+ per second. Both these
-figures are for tiles which have already been rendered.
-
-Filesystem Issues
-=================
-The average tile size is currently somewhere in the region of 2.5kB.
-(Based on a 20GB MySQL DB which contains 8M tiles). Typically
-filesystems are not particularly efficient at storing large numbers
-of small files. They often take a minimum of 4kB on the disk.
-
-Unfortunately if you reduce the block size to 1 or 2kB then this also
-has a significant impact on the maximum file system size and number of
-inodes available.
-
-
-**  Note: The issues below have been worked around in the current
-code by using the hashed directory path. 
-
-The simple z/x/y.png filesystem layout means that at high zoom levels
-there can be large numbers of files in a single directory
-
-  Zoom 18 = 2^18 = 256k files in a single directory.
-
-If ext2/3 is being used then you really need to have directory indexing
-enabled.
-
-New ".meta" tile storage
-========================
-The latest code stores each metatile in a single .meta file instead of
-lots of small .png files. This is a more efficient use of disk space
-and inodes. For example, many sea tiles are 103 bytes long. In the old
-scheme a meta tile of blank sea tiles would take 64 inodes of 4kB each,
-a total of 256kB. In the new scheme it needs a single file of about 7kB.
-
-The utility convert_meta can be used to convert a tree of .png files to
-.meta (or back again).
-
-mod_tile has been reworked to integrate more closely with Apache and
-deliver tiles from the .meta files.