See NixOS#38616. This is a minor issue, and for now we keep around the
old attribute name with a warning to tell people how to switch.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

This simply makes it easier to browse the PostgreSQL package directory. More to
come.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

This is a backwards compatible change; it mostly puts all the extensions
for postgresql in a common directory to keep them isolated.

It also moves a few things that /were not/ extensions out into other parts of
the filesystem namespace; namely the postgresql_jdbc and psqlodbc libraries
were moved under development/java-modules and development/libraries,
respectively. Because these libraries use the libpq postgresql client drivers,
they're less sensitive to underlying version changes anyway (since the protocol
is relatively stable).

No attributes were renamed or harmed in the creation of this patch.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

This makes pgjwt take a dummy 'postgresql' argument, which it does not *need*
in the buildInputs (it is purely a SQL extension with no C code). However, this
argument will be necessary for an upcoming change that will parameterize the
extensions over a particular PostgreSQL version.

It also does some tiny cleanup, setting a null build phase.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

This adds five new top-level attrsets:

    pkgs.postgresql93Packages
    pkgs.postgresql94Packages
    pkgs.postgresql95Packages
    pkgs.postgresql96Packages
    pkgs.postgresql10Packages

Each of these contains a set of PostgreSQL extensions, built against the
particular specified postgresql package. This makes it far easier for specific
PostgreSQL extensions to be built and used in NixOS modules, and provides an
easier way to test extension builds against multiple versions. For example, we
can now use either:

    pkgs.postgresql96Packages.timescaledb

or

    pkgs.postgresql10Packages.timescaledb

when building and specifying extensions.

By default, the prior top-level attribute names such as 'timescaledb' HAVE NOT
been removed. They will be deprecated in a future commit to encourage people to
move to explicitly-versioned attrsets.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

… checking PostgreSQL compatibility

Not every PostgreSQL extension builds with every version we have available; for
instance, timescaledb is only available for PostgreSQL 9.6 and later.

To do this, extensions may use a 'passthru.versionCheck' attribute, which
should return 'true' if the extension can be built and used against the given
postgresql version. This avoids needless build failures and helps indicate
to the user that an extension is not available.

Unfortunately the PostgreSQL expression did not previously offer a 'version'
attribute to check against, so we must add it (as expected). This implies a
rebuild of the PostgreSQL derivations, unfortunately.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

…ttrs

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

This is needed because some PostgreSQL plugins don't have a bin
directory. If only these plugins are listed in cfg.extraPlugins buildEnv
will turn $out/bin into a symbolic link to ${pg}/bin. Lateron we try to
rm $out/bin/{pg_config,postgres,pg_ctl} which will then fail because
$out/bin will be read-only.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

This results in a marginal closure size increase of ~137mb -> ~141mb.
This only affects the 'out' output, not the 'lib' output, so
libpq-dependent expressions should not bloat as a result of this.

The type of the systemd service is changed to 'notify' on 9.6+, which
will help identify when Postgres is starting/shutting down.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Extensions need to compare against the version of PostgreSQL they're
building against, so providing a partially applied function is easier
and less verbose for these cases.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

This also bars all extensions which currently do not build.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

This adds preliminary support for JIT'ing SQL queries via LLVM to
PostgreSQL, enabled on versions 11+, on Linux. The default stdenv of
PostgreSQL itself and all exposed sub-packages are overridden to use a
particular major version of llvmPackages.stdenv so that 'clang' is
chosen as the default compiler. Currently, we use llvmPackages_6. To do
this, we expose another component from postgresqlXXPackages -- a new
.stdenv attribute, which exposes which stdenv PostgreSQL was built with.

PostgreSQL JIT support works by compiling all of the source code for the
system into LLVM bitcode, and shipping this as part of the binary
distribution (under the .lib output, in our case, $lib/lib/bitcode/). At
runtime, Postgres uses the LLVM API in order to load this bitcode and
JIT queries directly against the source code. This allows inlining
database code into queries, in particular inlining operator and
expression definitions for custom types, in any extensions. This feature
is enabled via the '--with-llvm' flag during configurePhase.

This feature is integrated with 'PGXS', the Makefile infrastructure for
writing and distributing Postgres extensions. If an extension is being
compiled against a version of PostgreSQL that has LLVM JIT support, then
the extension code is *also* compiled to bitcode and distributed
transparently in the binary output, alongside the extension shared
object. In order to ensure consistent bitcode is emitted, when Postgres
is compiled with LLVM support, it *must* be compiled with Clang (hence
the stdenv choice), and it also *must* embed the copy/path of Clang into
the resulting binaries, so PGXS can use the exact same compiler later on
during 3rd party extension builds. This design decision is likely
because on systems such as Debian, multiple versions of Clang can exist,
so any compiled Postgres code/extensions must use a consistent compiler
for all future builds.

However, this decision has extremely negative consequences for Nix-based
packages, because it inserts LLVM and Clang into the closure of the
PostgreSQL derivations as a hard runtime-dependency. This bloats the
closure size by over a gigabyte (~140MB -> 1.4GB), which is fairly
unwieldly and unlikely to be permissible by default. Currently this
bloat only applies to the .out (binary) outputs.

But this is made worse by the fact that the .lib output is also bloated
by having a hard runtime dependency on llvmPackages.llvm.lib. This is
because postgresql.lib now ships libllvmjit.so which talks to
libLLVM.so, but postgresql.lib also contains client libraries like
libpq.so. This effectively bloats every libpq client expression as well
by about 200MB.

Finally, because of the way PGXS's default installation logic works, it
wants to install binary artifacts into the postgresql lib/ directory,
which obviously isn't possible in the Nix store as it's read-only.
Hence, we create environments composed of all extensions outputs and
patch postgresql to load that. But this means the installPhase for every
extension is currently a custom hand-written script, and *that* means
every extension must now contain logic to install LLVM .bc files on top
of .sql and .so files.

Instead, we should probably patch PGXS to install to a proper external
directory so its install logic can take over and we can remove custom
installPhase scripts for most extensions. Postgres' LLVM support logic
in its configure script and PGXS code will need to be patched to remove
hard-coded references to clang-wrapper, since we always control the
exact version of clang used and can remove it as a run-time dependency.
Finally, bitcode should probably be moved to separate .bitcode
derivations for all server versions and extension outputs, so that libpq
clients aren't bloated by indirect dependencies on libLLVM.so (by way of
libllvmjit.so).

Oh, and PostGIS fails to build with JIT support/clang as the compiler,
for some reason.

As a result of these significant complications, this support is disabled
by default, and should only be considered supported for vanilla
PostgreSQL with no third-party extensions.

Darwin may also be supported in the future; it may even build, but can't
be tested.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

When JIT support was enabled, PGXS embedded a copy of clang-wrapper's
path into its $out, resulting in a large increase in closure size due
to dragging the whole compiler along. A similar problem existed
previously by way of binutils 'ld', which was worked around. Perform
that same workaround for clang-wrapper, too.

With JIT support enabled, this reduces the closure size of
postgresql11.out from ~1.4GB to ~410MB.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

With JIT support enabled, the Postgres build system keeps a copy of
bitcode for the server, and a new llvmjit.so extension, in the $lib
output. However, this bloats the closure of any libpq-dependent
libraries by requiring a hard dependency on libLLVM.so, as well as the
actual bitcode files (which aren't useful at all to clients).

Moving both of these into $out does not hurt binary output size (since
it already was dependent on $lib anyway), but does help all clients.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

withPackages is like a local version of the services.postgresql.plugins
option: it allows you to build an environment containing PostgreSQL and
all the extensions you desire, by way of attrsets.

For example, the following command will drop you in a Nix shell
containing all the binaries from PostgreSQL 10, including the
TimescaleDB extension as a loadable library:

    $ nix-shell -p \
      'postgresql10Packages.withPackages (p: with p; [ timescaledb ])'

This is primarily useful for running demonstrations or local
experiments, but may also be useful for performing database upgrades (as
you may want pg_dumpall, etc from _newer_ PostgreSQL versions, to run
against your current server).

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

By fixing the (non-overrideable) PGXS installation directory variables
in the Postgres expression, any further extensions can be installed
using traditional pg_config-based PGXS Makefile build systems. This that
overriding 'installPhase' is no longer necessary; an ordinary PREFIX=...
will work, just like most Makefile build systems.

Aside from being cleaner, less-error prone, and easier to write, this is
also another piece necessary for useable JIT support.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

…extension is built against

GDAL's Postgres support had a weird cyclic dependency: GDAL depends on
Postgres (for possible spatial information querying for GDAL clients),
and PostGIS depends on GDAL. This sounds fine, except it has a dirty
follow up: GDAL is *only* built against the default version of
PostgreSQL. This means if you try to use a non-default version of
Postgres with PostGIS, it pulls in GDAL, which pulls in some other
version of Postgres. This results in weird situations like PostgreSQL
9.4 + PostGIS depending on PostgreSQL 9.6 in their closure.

In a previous experiment I tried making this optional, but there's a
better way: instead of making it optional, we simply let the default
'gdal' expression depend on 'postgresql' (whatever version the default
might be), and then we simply override the version of GDAL used by
postgis to be the *same* postgres that postgis is built against. This
effectively results in a private copy of gdal that only is used by the
postgis extension builds.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

…ts, for testing

Occasionally, it may be useful to enumerate _all_ supported package
sets, for example, in the NixOS tests. This refactoring supports that
while keeping the level of redundancy the same, for the most part.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

This test doesn't work, but it suppresses a needless warning, anyway.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

This was a bug that was fixed upstream; the applied patch is slightly
different but with the same result. Should be removed when a new version
of PostGIS is out.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

This allows to merge configurations that both define plugins as in:

  mkMerge [
    { services.postgresql.plugins = p: [ p.pg_journal ]; }
    { services.postgresql.plugins = p: [ p.postgis ]; }
  ]

selectorFunction should be moved to lib/types.nix since options like
"plugins" occur in at least 6 other places in NixOS.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

For certain SQL operations such as

    COPY (...) [TO|FROM] PROGRAM '...' WITH (FORMAT CSV)

the designated PostgreSQL server user must be able to execute the
specified program and pipe the data into it; however, Postgres executes
these commands *under a shell* in order to support features such as
traditional piping, for commands like split or gzip. With no homedir and
no shell assigned, Postgres fails to execute the shell, resulting in a
hang that then later results in the query failing. Depending on the
program and query executed, this may happen at seemingly random times
due to buffering/piping issues.

While it isn't traditional to assign a shell to the database user (at
least on NixOS), doing so restores this functionality, which is
extremely useful for e.g. bulk ETL pipelines that copy to/from external
data files.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Otherwise, features like 'man psql' don't work, which is weird for a
machine that has PostgreSQL running as a service.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

This turns on JIT support by default for PostgreSQL 11-compatible
packages, now that most of the problems are fixed:

  1) Closure sizes don't include Clang, only LLVM,
  2) libpq clients don't suffer bloat from libLLVM.so dependencies,
  3) PGXS support is improved, so hand-written installPhases are
     mostly not needed, making it much more viable to ship bitcode
     for 3rd party extensions.

As a side note, this change also should not impact the nixos-small
channel builds very much (I hope), because those already have LLVM 6.0
in their closure, by way of QEMU/mesa packages.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

…qlPackage option

This option is used in modules that need to refer to the selected
PostgreSQL package. Without this option these modules need to
duplicate some conditional logic which is error-prone.

This option should be removed when the deprecated
config.services.postgresql.package is removed.