Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Fetching contributors…

Cannot retrieve contributors at this time

136 lines (92 sloc) 3.784 kB
= SkyTools - tools for PostgreSQL =
This is a package of tools in use in Skype for replication and failover.
It also includes a generic queuing mechanism PgQ and utility library for
Python scripts, as well as a script for setting up and managing WAL based
standby servers.
== Overview ==
It contains following modules:
=== PgQ ===
PgQ is a queuing system written in PL/pgsql, Python and C code. It is
based on snapshot based event handling ideas from Slony-I, written for
general usage.
PgQ provides an efficient, transactionnal, queueing system with
multi-nodes support (including work sharing and splitting, failover and
switchover, for queues and for consumers).
Rules:
- There can be several queues in database.
- There can be several producers than can insert into any queue.
- There can be several consumers on one queue.
- There can be several subconsumers on a consumer.
PgQ is splited into 3 layers: Producers, Ticker and Consumers.
*Producers* and *Consumers* respectively push and read events into a
queue. Producers just need to call PostgreSQL stored procedures (like a
trigger on a table or a PostgreSQL call from the application). And
consumers are frequently written in Python, the prefered language as it
has a powerful SKytools Framework but it is not limitative and any
language able to run PostgreSQL stored procedures can be used.
*Ticker* is a daemon which splits the queues in batch of events and
handle the maintenance of the system. The Ticker is provided with the
Skytools.
Documentation:
- PgQ ticker daemon (pgqd) usage: link:doc/pgqd.html[]
- PgQ admin tool (qadm) usage: link:doc/qadmin.html[]
- PgQ SQL API overview: link:doc/pgq-sql.html[]
- PgQ SQL reference: link:pgq/[]
=== Londiste ===
Replication tool written in Python, using PgQ as event transport.
Features:
- Tables can be added one-by-one into set.
- Initial COPY for one table does not block event replay for other
tables.
- Can compare tables on both sides.
Documentation:
- Londiste script usage: doc/londiste3.txt
(also available as `man 1 londiste`)
- Londiste HOWTOS: doc/howto/
=== walmgr ===
This script will setup WAL archiving, does initial backup and
runtime WAL archive and restore.
It can also be used for up-to-last-second partial file copying,
so less than whole file is lost in case of loss of master database server.
== Source tree contents ==
doc/::
Documentation in asciidoc format. Source for both html
and man pages.
python/::
Python modules and primary executables - walmgr, londiste, qadmin, pgqadm.
python/pgq/::
Python framework for PgQ.
python/londiste/::
Londiste replication.
python/skytools/::
Low-level utilities for writing database scripts in Python.
sql/::
Database modules.
sql/pgq/::
Table definitions and functions for PgQ queueing.
sql/pgq_node/::
Framework for cascaded consuming.
sql/pgq_coop/::
Functions for cooperative consuming.
sql/londiste/::
Table definitions and functions for Londiste replication.
sql/ticker/::
PgQ ticker written in C.
scripts/::
Python scripts with lesser priority.
lib/::
libusual C libary, for pgqd.
debian/::
Debian packaging. This is for creating private packages,
official Debian packages uses it's own packagin code.
misc/::
Random scripts used for building.
== Upgrade from 2.1 ==
Assuming PgQ + Londiste setup. This will upgrade PgQ to 3.0 and install
Londiste 3 in parallel with Londiste 2.
1. Install Postgres modules. They are backwards compatible with 2.1.
2. Stop `pgqadm.py ticker` processes.
3. Apply ./upgrade/final/v3.0_pgq_core.sql
4. Apply installed pgq_node.sql
5. Apply installed londiste.sql - this will throw error on CREATE SCHEMA, but should otherwise apply fine.
6. Start pgqd.
Jump to Line
Something went wrong with that request. Please try again.