LINZ Bulk Data Extract Uploader
linz_bde_uploader is a programme for loading LINZ BDE files into a PostgreSQL
linz_bde_uploader has the ability to load full and incremental table
Landonline BDE loads, as well as manage versioning information.
Refer to LINZ::Bde documentation for further information about LINZ BDE files and repository format.
- PostgreSQL 9.3 or greater
- Perl 5.12 or greater, plus
Tested on Ubuntu 14.04
Use the Debian packaging within the debian/ directory to install this software on Ubuntu.
The main components to install are the perl packages, PostgreSQL database
setup and config script and the actual
linz_bde_uploader programme and
configuration. The build install process handles this, however setup of the
PostgreSQL user account, database, logging directory and configuration setup need
to be done as manual task afterward.
perl Build.PL ./Build install
Advanced install options
The build system is using Perl Module::Build. A full list of the building options are available run:
A more complex example involving specific install directories could something like:
perl Build.PL --prefix=/usr ./Build install
perl Build.PL \ --prefix=/usr \ --install_path conf=/my/conf/dir \ --install_path sql=/my/sql/dir
setup under UNIX
For a database to be usable as the target of a
run, it needs to be prepared in multiple steps.
First step is creating the BDE schema. This can be done following the instructions in linz-bde-schema. The BDE schema creation scripts will take care of ensuring required cluster-global roles exist and database-local objects (schemas, tables, extensions, functions etc) are loaded.
table_version extension is an optional component for
the BDE schema, if you haven't already installed it you'll need
to do it as the second step:
psql $DB_NAME -c "CREATE EXTENSION IF NOT EXISTS table_version"
Third step is executing the installed linz-bde-uploader SQL support files with something like the following:
for file in /usr/share/linz-bde-uploader/sql/*.sql do psql $DB_NAME -f $file -v ON_ERROR_STOP=1 done
linz_bde_uploader software should be run under a UNIX account that
also has PostgreSQL
bde_dba database access. It is best to first create a
system user account as well so ident authentication can be used.
adduser --system --gecos "BDE Maintainer" bde
The bde PostgreSQL user account needs to have
bde_dba rights, but does not
need to have superuser rights by default. An example SQL create user script
could look like:
DO $$ BEGIN IF NOT EXISTS (SELECT * FROM pg_roles where rolname = 'bde') THEN CREATE ROLE bde LOGIN NOSUPERUSER INHERIT NOCREATEDB NOCREATEROLE; ALTER ROLE bde SET search_path=bde, bde_control, lds, public; GRANT bde_dba TO bde; END IF; END $$;
Creating a standard system log directory in /var/log would be a good idea:
BDE_LOG_DIR=/var/log/linz-bde-uploader BDE_USER=bde sudo mkdir -p $BDE_LOG_DIR sudo chown -R $BDE_USER:adm $BDE_LOG_DIR; sudo chmod 0755 $BDE_LOG_DIR;
All parameters to setup and running of
linz_bde_uploader can be found within the
template conf file in
conf/linz_bde_uploader.conf. Another important file is
tables.conf which lists the BDE tables and their associated loading parameters.
On Ubuntu using the debian packaging these files are installed into
Note: You can create .conf.test configuration file (e.g
which can override any of the parameters in the main config file. This can be used as a
convenient way to set parameters without having to change the installed default config
Key parameters to change in the
linz_bde_uploader.conf config are:
db_connection: The PostgreSQL connection string for setting the database connection string. e.g
db_error_level: The PostgreSQL error level, can be 0 for TERSE, 1 for DEFAULT (the default value), 2 for VERBOSE. See https://www.postgresql.org/docs/current/static/runtime-config-logging.html
bde_repository: Set the path to the directory of BDE unload files. This directory should have a
level_5subdirectory with child folders in each with the naming convention of YYYYMMDDhhmmss for each dataset
tmp_base_dir: This temp processing directory for uncompressing and pre-processing BDE datafile. This directory should have at 15GB of free space for large production tables such as
crs_adj_obs_change. Defaults to
include_tables: A list of table to load for the run. These table must exist in the in the file as defined by
log4perl.logger: Setting the loggers to use. By default the
smtpsender: The mail host and sender email address to send completion or error reports to if email logging is enabled.
log_email_address: The email address that the mail server will send the completion or error reports to. multiple emails can be listing with a "," separator.
A simple example to upload latest available full set of bde files into the database:
linz_bde_uploader -full -verbose -config-path /etc/linz-bde-uploader/linz_bde_uploader.conf -listing /var/log/linz-bde-uploader/linz_bde_uploader.log
There is also the wrapper shell script
run_bde_upload that runs
linz_bde_uploader with the config and log directory already defined:
run_bde_upload -full -verbose
The normal mode to run the software is to load any available level 0 or 5 dataset files available. This can be done via:
run_bde_upload -full-incremental -incremental
Note: In this mode the
-full-incremental option is used rather than the
options which updates the current dataset instead of a full replace.
For more help about options for running
linz_bde_uploader can be made available
through the following command:
Copyright 2016 Crown copyright (c) Land Information New Zealand and the New Zealand Government.
This project is under 3-clause BSD License, except where otherwise specified. See the LICENSE file for more details.