Use Greenplum's parallel loader to batch load messages in RabbitMQ into Greenplum
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


Use Greenplum's parallel loader to batch load messages from RabbitMQ into Greenplum



  • Python - tested with version 2.7.3
  • Running RabbitMQ instance
  • Running Greenplum DB cluster
  • GPFdist started and pointed to the correct data directory

Required Python packages:

Add to your path

$ export PATH=$PATH:bin/


Create a configuration file based on sample.ini. Be sure Greenplum and RabbitMQ connectivity options are correct.

Start, use a process control system like Supervisor to manage if desired.

$ python -c config.ini

Sample expected output:

$ python -c test.ini
2013-08-07 14:53:41,789:24125-[INFO]:-Connecting to
2013-08-07 14:53:41,839:24125-[INFO]:-Established RabbitMQ connection
2013-08-07 14:53:41,840:24125-[INFO]:-[c]: Greenplum consumer started
2013-08-07 14:53:52,598:24125-[INFO]:-[c]: Starting Greenplum loader process
2013-08-07 14:53:52,635:24228-[INFO]:-[3a044674-7744-48af-ba1a-771f6302f953]: Starting load
2013-08-07 14:53:52,638:24125-[INFO]:-[c]: Starting Greenplum loader process
2013-08-07 14:53:52,653:24229-[INFO]:-[88770bd9-4bb9-40e6-9ee9-072174089a78]: Starting load
2013-08-07 14:53:52,821:24228-[INFO]:-[3a044674-7744-48af-ba1a-771f6302f953]: Load complete
2013-08-07 14:53:52,822:24229-[INFO]:-[88770bd9-4bb9-40e6-9ee9-072174089a78]: Load complete
2013-08-07 14:53:54,741:24125-[INFO]:-Shutting down...
2013-08-07 14:53:54,742:24125-[INFO]:-[c]: Flushing in process loads
2013-08-07 14:53:54,742:24125-[INFO]:-[c]: Starting Greenplum loader process
2013-08-07 14:53:54,743:24125-[INFO]:-[c]: Waiting for loads to finish
2013-08-07 14:53:54,745:24231-[INFO]:-[655174cd-621e-472d-a965-1cb8f7624dd3]: Starting load
2013-08-07 14:53:54,746:24231-[INFO]:-[655174cd-621e-472d-a965-1cb8f7624dd3]: Load complete


The following known limitations will be addressed in a future release:

  • Use FIFO pipes instead of files for intermediate data files