New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overhaul PostgreSQL packaging, native extensions, and NixOS support #38698

Open
wants to merge 61 commits into
base: master
from

Conversation

@thoughtpolice
Member

thoughtpolice commented Apr 10, 2018

Motivation for this change

I'm brief on time now, but this isn't done yet. See #38616. In short, this overhauls the namespacing and extension management of PostgreSQL packages, greatly easing the ability to test, build, and package PostgreSQL extensions properly. It also overhauls the NixOS module to support the proposed API in #38616, making it far easier to add custom extensions to your server in a correct way, no matter what version you choose.

I've been running these changes with some light testing for a few days, but I'm not done yet. I'll be amending and adding more comments to the commits and maybe squash one or two. These changes otherwise seem solid and shouldn't break anything.

Note that the NixOS module remains backwards compatible. Nothing should break, and you're unlikely to even notice if you don't use extensions. I consider this a good thing and ideally supporting it to 18.09 should not be terrible. However, warnings will be thrown when using the old APIs, and assertions are thrown if both are used at once in incompatible ways. (In fact you can even use a mix of the old and new APIs, in theory -- e.g. use the new .packages attribute with the old .extraPlugins, instead of the new .plugins.) I'm open to dropping backwards compatibility if other people think we can give it up for 18.09, though.

Finally, this also updates Nixpkgs and NixOS to use PostgreSQL 10 and libpq 10 by default. Although this isn't necessary, it seems like a reasonable time to make the push, since 9.6 has been the default since 17.09. PostgreSQL 11 will ideally be out around September, near the 18.09 release, but it's unlikely we'll have time to fit it in by then (depending on how things go for us, and them.)

I have also stepped up and added myself as the maintainer of the PostgreSQL module, and will be finishing the manual section.

I suggest reviewing the commits sequentially, in order, rather than at once. They are all individually small, work, and should be much easier to follow.

Things done
  • Actually finish writing the PostgreSQL section of the manual.

  • Look into deprecation warnings for old top-level attrs; using builtins.trace is annoying because e.g. it triggers on nix search -u ...

  • Look into #22653 and #38469 as well -- these should be fixed while I'm at it, IMO.

  • Bikeshed the naming of the new options? .packages is really close to .package, but I'm not sure of a better name. On the other hand, I think .plugin is a good name vs .extraPlugins

  • Migrate all tests and NixOS modules to new API

  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)

  • Built on platform(s)

    • NixOS
    • macOS
    • other Linux distributions (Fedora 25, sandboxing enabled)
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)

  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"

  • Tested execution of all binary files (usually in ./result/bin/)

  • Fits CONTRIBUTING.md.


@thoughtpolice

This comment has been minimized.

Show comment
Hide comment
@thoughtpolice

thoughtpolice Apr 11, 2018

Member

Please note that GitHub hates actually showing you the linear stream of commits as it will appear in the history at merge-time, and instead always prefers showing things sorted-by-date, even when this is completely, totally, 100% factually wrong. I rebased and re-arranged a few commits... Ugh.

Member

thoughtpolice commented Apr 11, 2018

Please note that GitHub hates actually showing you the linear stream of commits as it will appear in the history at merge-time, and instead always prefers showing things sorted-by-date, even when this is completely, totally, 100% factually wrong. I rebased and re-arranged a few commits... Ugh.

@thoughtpolice

This comment has been minimized.

Show comment
Hide comment
@thoughtpolice

thoughtpolice Apr 16, 2018

Member

Note to future archaeologists who will later discover my corpse here: this was the magical incantation needed to make GitHub display commits in order, as it should, by resetting the commit/author date of every individual commit to be 2-3 seconds apart.

for x in $(git log --format=oneline master..pgsql-fixes | tac | awk '{print $1}'); do
  git cherry-pick $x; git commit --amend --reset-author --no-edit;
  sleep 2;
done

Surely this will be the solution to an untold future terror and save the world, one day.

Member

thoughtpolice commented Apr 16, 2018

Note to future archaeologists who will later discover my corpse here: this was the magical incantation needed to make GitHub display commits in order, as it should, by resetting the commit/author date of every individual commit to be 2-3 seconds apart.

for x in $(git log --format=oneline master..pgsql-fixes | tac | awk '{print $1}'); do
  git cherry-pick $x; git commit --amend --reset-author --no-edit;
  sleep 2;
done

Surely this will be the solution to an untold future terror and save the world, one day.

@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Apr 16, 2018

Member

Really good work so far @thoughtpolice 👍

Member

domenkozar commented Apr 16, 2018

Really good work so far @thoughtpolice 👍

@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar May 11, 2018

Member

@thoughtpolice can this go as-is and then we add further improvements?

Member

domenkozar commented May 11, 2018

@thoughtpolice can this go as-is and then we add further improvements?

@thoughtpolice

This comment has been minimized.

Show comment
Hide comment
@thoughtpolice

thoughtpolice May 11, 2018

Member

Yes, I've been running these changes in a stable manner for at least a few weeks on my development machine, though I was hoping @ocharles could have given a review

Member

thoughtpolice commented May 11, 2018

Yes, I've been running these changes in a stable manner for at least a few weeks on my development machine, though I was hoping @ocharles could have given a review

@thoughtpolice

This comment has been minimized.

Show comment
Hide comment
@thoughtpolice

thoughtpolice May 11, 2018

Member

I was also hoping to add a postgresqlWithPackages function, so you could use postgres without NixOS, but with NIX_LIBDIR properly set up -- unfortunately this either requires some unsavory duplication of code between Nixpkgs and NixOS to retain backwards compatibility.

I believe I have a WIP patch for this, so I may add it as well.

Member

thoughtpolice commented May 11, 2018

I was also hoping to add a postgresqlWithPackages function, so you could use postgres without NixOS, but with NIX_LIBDIR properly set up -- unfortunately this either requires some unsavory duplication of code between Nixpkgs and NixOS to retain backwards compatibility.

I believe I have a WIP patch for this, so I may add it as well.

mkDefault (if versionAtLeast config.system.stateVersion "18.09" then pkgs.postgresql10Packages
else if versionAtLeast config.system.stateVersion "17.09" then pkgs.postgresql96Packages
else if versionAtLeast config.system.stateVersion "16.03" then pkgs.postgresql95Packages
else pkgs.postgresql94Packages);

This comment has been minimized.

@danbst

danbst May 12, 2018

Contributor

Can this be made dependent on cfg.package.version instead of config.system.stateVersion?

@danbst

danbst May 12, 2018

Contributor

Can this be made dependent on cfg.package.version instead of config.system.stateVersion?

This comment has been minimized.

@thoughtpolice

thoughtpolice May 13, 2018

Member

The intent is that e.g. if you were on NixOS 16.03, and you upgrade to NixOS 17.09, you should keep using PostgreSQL 9.5, because 9.6 is not on-disk compatible, and breaking your database across an upgrade isnt friendly. This is because when you installed NixOS, the (16.03) installer explicitly sets stateVersion = "16.03" in configuration.nix by default, and that should not be changed even beyond upgrades.

I might be misunderstanding what you mean, of course (a diff would be helpful if that's the case!). But otherwise, this logic must remain to not break already-installed databases.

@thoughtpolice

thoughtpolice May 13, 2018

Member

The intent is that e.g. if you were on NixOS 16.03, and you upgrade to NixOS 17.09, you should keep using PostgreSQL 9.5, because 9.6 is not on-disk compatible, and breaking your database across an upgrade isnt friendly. This is because when you installed NixOS, the (16.03) installer explicitly sets stateVersion = "16.03" in configuration.nix by default, and that should not be changed even beyond upgrades.

I might be misunderstanding what you mean, of course (a diff would be helpful if that's the case!). But otherwise, this logic must remain to not break already-installed databases.

This comment has been minimized.

@danbst

danbst May 13, 2018

Contributor

This isn't a direct problem by itself, but rather a simplification for those who use PG older than NixOS default. When you set services.postgresql.package to some old version, you now have to set services.postgresql.packages to an older pacakgeset too, which, I think, can be detected by NixOS automatically.

@danbst

danbst May 13, 2018

Contributor

This isn't a direct problem by itself, but rather a simplification for those who use PG older than NixOS default. When you set services.postgresql.package to some old version, you now have to set services.postgresql.packages to an older pacakgeset too, which, I think, can be detected by NixOS automatically.

@thoughtpolice

This comment has been minimized.

Show comment
Hide comment
@thoughtpolice

thoughtpolice May 24, 2018

Member

@ocharles has given me his blessing/trust on this change in lieu of the fact he doesn't have much time to review it, but I'd love to start merging some of this in soon. I will post a message to the mailing list soon about this for any final objections.

Member

thoughtpolice commented May 24, 2018

@ocharles has given me his blessing/trust on this change in lieu of the fact he doesn't have much time to review it, but I'd love to start merging some of this in soon. I will post a message to the mailing list soon about this for any final objections.

@thoughtpolice

This comment has been minimized.

Show comment
Hide comment
@thoughtpolice

thoughtpolice May 29, 2018

Member

I'm back from vacation and plan on looking at seeing this through this week. I'll post to Discourse shortly for feedback.

Member

thoughtpolice commented May 29, 2018

I'm back from vacation and plan on looking at seeing this through this week. I'll post to Discourse shortly for feedback.

@thoughtpolice thoughtpolice changed the title from [WIP] Overhaul PostgreSQL packaging, native extensions, and NixOS support to Overhaul PostgreSQL packaging, native extensions, and NixOS support May 31, 2018

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr May 31, 2018

Member

This will need to be documented in the release notes (nixos/doc/manual/release-notes/rl-1809.xml) I presume.

As a user, I really much prefer this newly presented interface for PostgreSQL stuff. 👍

Member

samueldr commented May 31, 2018

This will need to be documented in the release notes (nixos/doc/manual/release-notes/rl-1809.xml) I presume.

As a user, I really much prefer this newly presented interface for PostgreSQL stuff. 👍

@thoughtpolice

This comment has been minimized.

Show comment
Hide comment
@thoughtpolice

thoughtpolice Aug 6, 2018

Member

I'm still working on this, sorry for all the delays all. If it's any consolation, this has been running quite stable for me. :)

I've updated the release notes, issues like #22653 and #38469 should be fixed now (thanks to @basvandijk), and there is a beta of PostgreSQL 11 now available[1]. I'd like to actually get this done this week. Remaining things:

  • Finish manual (particularly backup/restore/upgrade cycle)
  • Finish migrating tests.

I expect this will probably take another few hours of focused work but I plan on landing this before the 18.09 branch.


[1] I am currently using this primarily to explore integration of LLVM/JIT support into the Postgres 11 betas, to get a jump on this feature, since it's tricky to support correctly. It's quite possible that I will remove the top-level attributes for postgres 11 before merging this, so people don't get any funny ideas, but in the mean time the support will be there and worked out, hopefully.

Member

thoughtpolice commented Aug 6, 2018

I'm still working on this, sorry for all the delays all. If it's any consolation, this has been running quite stable for me. :)

I've updated the release notes, issues like #22653 and #38469 should be fixed now (thanks to @basvandijk), and there is a beta of PostgreSQL 11 now available[1]. I'd like to actually get this done this week. Remaining things:

  • Finish manual (particularly backup/restore/upgrade cycle)
  • Finish migrating tests.

I expect this will probably take another few hours of focused work but I plan on landing this before the 18.09 branch.


[1] I am currently using this primarily to explore integration of LLVM/JIT support into the Postgres 11 betas, to get a jump on this feature, since it's tricky to support correctly. It's quite possible that I will remove the top-level attributes for postgres 11 before merging this, so people don't get any funny ideas, but in the mean time the support will be there and worked out, hopefully.

@thoughtpolice

This comment has been minimized.

Show comment
Hide comment
@thoughtpolice

thoughtpolice Aug 6, 2018

Member

Some more changes:

  • There's a new .withPackages combinator exposed from every package set, which allows you to drop into a Nix shell with postgres+plugins inside a buildEnv. This is mostly useful for doing upgrades or demonstrations, but in the future could be useful for other things, too (for example: packing PostgreSQL Docker images along with extensions, using Nix).
  • PGXS previously did not respect PREFIX during builds, meaning every Postgres extension had to have a custom, hackily written installPhase. I have fixed PGXS' to obey PREFIX, meaning make install PREFIX=/nix/store/... now works for the majority of extensions, so their installPhases can be removed. This is also necessary for JIT support, which requires shipping bitcode files during installPhase, and would be tedious to write correctly over and over again; it just doesn't scale. Some packages may still require minor patches to work correctly (such as CitusDB)
  • JIT support for PostgreSQL 11 has been extended and should be "Ready to Ship" -- I fixed most glaring problems with this feature. Please see the corresponding postgres11 commit messages for details. The net result though is that postgresql11.out itself will bloat from about ~130MB to ~400MB due to needing libLLVM.so, but everything else should be fine. (In particular, libpq-based clients will not bloat at all). This feature is not enabled by default however, as I'd like to at least make sure this bloat/LLVM dependency is acceptable (especially because postgresql is part of the -small channels.)
  • There's now a section for doing simple, pg_dumpall-based upgrades in the manual. It uses the .withPackage technique for showing a demonstration upgrade, since you want newer pg_dumpall tools.

I think this branch now fixes everything I set out to tackle, and then-some.

Member

thoughtpolice commented Aug 6, 2018

Some more changes:

  • There's a new .withPackages combinator exposed from every package set, which allows you to drop into a Nix shell with postgres+plugins inside a buildEnv. This is mostly useful for doing upgrades or demonstrations, but in the future could be useful for other things, too (for example: packing PostgreSQL Docker images along with extensions, using Nix).
  • PGXS previously did not respect PREFIX during builds, meaning every Postgres extension had to have a custom, hackily written installPhase. I have fixed PGXS' to obey PREFIX, meaning make install PREFIX=/nix/store/... now works for the majority of extensions, so their installPhases can be removed. This is also necessary for JIT support, which requires shipping bitcode files during installPhase, and would be tedious to write correctly over and over again; it just doesn't scale. Some packages may still require minor patches to work correctly (such as CitusDB)
  • JIT support for PostgreSQL 11 has been extended and should be "Ready to Ship" -- I fixed most glaring problems with this feature. Please see the corresponding postgres11 commit messages for details. The net result though is that postgresql11.out itself will bloat from about ~130MB to ~400MB due to needing libLLVM.so, but everything else should be fine. (In particular, libpq-based clients will not bloat at all). This feature is not enabled by default however, as I'd like to at least make sure this bloat/LLVM dependency is acceptable (especially because postgresql is part of the -small channels.)
  • There's now a section for doing simple, pg_dumpall-based upgrades in the manual. It uses the .withPackage technique for showing a demonstration upgrade, since you want newer pg_dumpall tools.

I think this branch now fixes everything I set out to tackle, and then-some.

@basvandijk

Impressive work! 18.09 is getting better and better.

@basvandijk

This comment has been minimized.

Show comment
Hide comment
@basvandijk

basvandijk Aug 7, 2018

Member

@thoughtpolice I added extra tests for some extensions in:

thoughtpolice/nixpkgs@pgsql-fixes...LumiGuide:pgsql-fixes-extra-tests

I leave it up to you to decide to merge this into your PR.

I made it so the extra tests are easily disableable which is handy if you want to debug the tests and don't want the extra complexity of certain extensions.

@nbp in the process I discovered that merging options like services.postgresql.plugins produce the evaluation error error: value is a list while a set was expected, at nixpkgs/lib/options.nix:77:23. These options should be assigned a function that takes an attribute set and returns a list with a selection of the values in the input set. Merging these options should create a new function that applies the functions of all the options to the input set and concatenates the resulting lists.

For this I created the selectorFunction type. It's currently defined in the postgresql module but there are at least 6 other places in nixpkgs that use these type of options:

What do you think about moving the selectorFunction type to lib/types.nix and patching these module to use it?

If you agree I'll make a PR to do this.

Member

basvandijk commented Aug 7, 2018

@thoughtpolice I added extra tests for some extensions in:

thoughtpolice/nixpkgs@pgsql-fixes...LumiGuide:pgsql-fixes-extra-tests

I leave it up to you to decide to merge this into your PR.

I made it so the extra tests are easily disableable which is handy if you want to debug the tests and don't want the extra complexity of certain extensions.

@nbp in the process I discovered that merging options like services.postgresql.plugins produce the evaluation error error: value is a list while a set was expected, at nixpkgs/lib/options.nix:77:23. These options should be assigned a function that takes an attribute set and returns a list with a selection of the values in the input set. Merging these options should create a new function that applies the functions of all the options to the input set and concatenates the resulting lists.

For this I created the selectorFunction type. It's currently defined in the postgresql module but there are at least 6 other places in nixpkgs that use these type of options:

What do you think about moving the selectorFunction type to lib/types.nix and patching these module to use it?

If you agree I'll make a PR to do this.

@basvandijk basvandijk referenced this pull request Aug 7, 2018

Closed

lib/types: add the selectorFunction type #44601

3 of 9 tasks complete

thoughtpolice and others added some commits Aug 6, 2018

nixos/gitea: update to new PostgreSQL interface
Signed-off-by: Austin Seipp <aseipp@pobox.com>
nixos/gitlab: use new postgres module API
Signed-off-by: Austin Seipp <aseipp@pobox.com>
nixos/matrix-synapse: use new postgres module API
Signed-off-by: Austin Seipp <aseipp@pobox.com>
postgresqlPackages.postgis: enable PostgreSQL 11 JIT support
This was a bug that was fixed upstream; the applied patch is slightly
different but with the same result. Should be removed when a new version
of PostGIS is out.

Signed-off-by: Austin Seipp <aseipp@pobox.com>
postgresqlPackages: remove outdated comments
Signed-off-by: Austin Seipp <aseipp@pobox.com>
nixos/manual: add more postgresql modules to 18.09 release notes
Signed-off-by: Austin Seipp <aseipp@pobox.com>
nixos/postgresql: give the plugins option the selectorFunction type
This allows to merge configurations that both define plugins as in:

  mkMerge [
    { services.postgresql.plugins = p: [ p.pg_journal ]; }
    { services.postgresql.plugins = p: [ p.postgis ]; }
  ]

selectorFunction should be moved to lib/types.nix since options like
"plugins" occur in at least 6 other places in NixOS.

Signed-off-by: Austin Seipp <aseipp@pobox.com>
nixos/postgresql: give the postgres user a homedir and shell
For certain SQL operations such as

    COPY (...) [TO|FROM] PROGRAM '...' WITH (FORMAT CSV)

the designated PostgreSQL server user must be able to execute the
specified program and pipe the data into it; however, Postgres executes
these commands *under a shell* in order to support features such as
traditional piping, for commands like split or gzip. With no homedir and
no shell assigned, Postgres fails to execute the shell, resulting in a
hang that then later results in the query failing. Depending on the
program and query executed, this may happen at seemingly random times
due to buffering/piping issues.

While it isn't traditional to assign a shell to the database user (at
least on NixOS), doing so restores this functionality, which is
extremely useful for e.g. bulk ETL pipelines that copy to/from external
data files.

Signed-off-by: Austin Seipp <aseipp@pobox.com>
timescaledb-parallel-copy: init at 2018-05-14
Signed-off-by: Austin Seipp <aseipp@pobox.com>
postgresqlPackages: add .man to all withPackage/NixOS configurations
Otherwise, features like 'man psql' don't work, which is weird for a
machine that has PostgreSQL running as a service.

Signed-off-by: Austin Seipp <aseipp@pobox.com>
postgresqlPackages: enable LLVM JIT by default on PostgreSQL 11+
This turns on JIT support by default for PostgreSQL 11-compatible
packages, now that most of the problems are fixed:

  1) Closure sizes don't include Clang, only LLVM,
  2) libpq clients don't suffer bloat from libLLVM.so dependencies,
  3) PGXS support is improved, so hand-written installPhases are
     mostly not needed, making it much more viable to ship bitcode
     for 3rd party extensions.

As a side note, this change also should not impact the nixos-small
channel builds very much (I hope), because those already have LLVM 6.0
in their closure, by way of QEMU/mesa packages.

Signed-off-by: Austin Seipp <aseipp@pobox.com>
@basvandijk

This comment has been minimized.

Show comment
Hide comment
@basvandijk

basvandijk Sep 9, 2018

Member

I rebased on master to fix the eval errors.

@GrahamcOfBorg test postgresql

Member

basvandijk commented Sep 9, 2018

I rebased on master to fix the eval errors.

@GrahamcOfBorg test postgresql

@GrahamcOfBorg

This comment has been minimized.

Show comment
Hide comment
@GrahamcOfBorg

GrahamcOfBorg Sep 9, 2018

Success on x86_64-linux

Attempted: tests.postgresql

No partial log is available.

GrahamcOfBorg commented Sep 9, 2018

Success on x86_64-linux

Attempted: tests.postgresql

No partial log is available.

@GrahamcOfBorg

This comment has been minimized.

Show comment
Hide comment
@GrahamcOfBorg

GrahamcOfBorg Sep 9, 2018

Success on aarch64-linux

Attempted: tests.postgresql

No partial log is available.

GrahamcOfBorg commented Sep 9, 2018

Success on aarch64-linux

Attempted: tests.postgresql

No partial log is available.

@basvandijk

This comment has been minimized.

Show comment
Hide comment
@basvandijk

basvandijk Sep 9, 2018

Member

Cherry-pick of this PR on release-18.09: release-18.09...LumiGuide:pgsql-fixes-release-18.09.

The postgresql tests pass. The manual incl. release notes build successfully and look correct.

Member

basvandijk commented Sep 9, 2018

Cherry-pick of this PR on release-18.09: release-18.09...LumiGuide:pgsql-fixes-release-18.09.

The postgresql tests pass. The manual incl. release notes build successfully and look correct.

@basvandijk

This comment has been minimized.

Show comment
Hide comment
@basvandijk

basvandijk Sep 9, 2018

Member

I think this is ready to merge now. If nobody objects I'll merge it after Monday 20:00 CEST if nobody beats me to it. I'll also create a cherry-pick PR for release-18.09.

Member

basvandijk commented Sep 9, 2018

I think this is ready to merge now. If nobody objects I'll merge it after Monday 20:00 CEST if nobody beats me to it. I'll also create a cherry-pick PR for release-18.09.

@basvandijk

This comment has been minimized.

Show comment
Hide comment
@basvandijk

basvandijk Sep 10, 2018

Member

I missed that we still have an unchecked TODO item: "Migrate all tests and NixOS modules to new API". Indeed modules like hydron haven't been migrated yet.

Every module that revers to config.services.postgresql.package needs to replace that with:

if (config.services.postgresql.package != null) 
then config.services.postgresql.package 
else config.services.postgresql.packages.postgresql

This is quite cumbersome and error-prone to repeat everywhere.

@thoughtpolice may I propose that we turn the local variable postgresqlPackage into a readOnly (and possibly invisible) option of the postgresql module? Then other modules can refer to that option abstracting over the if..then..else.

Member

basvandijk commented Sep 10, 2018

I missed that we still have an unchecked TODO item: "Migrate all tests and NixOS modules to new API". Indeed modules like hydron haven't been migrated yet.

Every module that revers to config.services.postgresql.package needs to replace that with:

if (config.services.postgresql.package != null) 
then config.services.postgresql.package 
else config.services.postgresql.packages.postgresql

This is quite cumbersome and error-prone to repeat everywhere.

@thoughtpolice may I propose that we turn the local variable postgresqlPackage into a readOnly (and possibly invisible) option of the postgresql module? Then other modules can refer to that option abstracting over the if..then..else.

postgresql: introduce the deprecated, readOnly and invisible postgres…
…qlPackage option

This option is used in modules that need to refer to the selected
PostgreSQL package. Without this option these modules need to
duplicate some conditional logic which is error-prone.

This option should be removed when the deprecated
config.services.postgresql.package is removed.
@basvandijk

This comment has been minimized.

Show comment
Hide comment
@basvandijk

basvandijk Sep 10, 2018

Member

@thoughtpolice I've now introduced the read-only and invisible config.services.postgresql.postgresqlPackage option in the last commit of master...LumiGuide:pgsql-fixes-postgresqlPackage (a1d18b7). That commit also migrates all remaining modules to the new scheme. What do you say? Merge it here?

Member

basvandijk commented Sep 10, 2018

@thoughtpolice I've now introduced the read-only and invisible config.services.postgresql.postgresqlPackage option in the last commit of master...LumiGuide:pgsql-fixes-postgresqlPackage (a1d18b7). That commit also migrates all remaining modules to the new scheme. What do you say? Merge it here?

@basvandijk

This comment has been minimized.

Show comment
Hide comment
@basvandijk

basvandijk Sep 10, 2018

Member

@vcunat nox-review mentions that this PR rebuilds over 1200 packages so I guess this first has to go through the staging branch. I'm unfamiliar with that procedure so I would appreciate some guidance and I see that you merge into staging a lot. So how does it work?

  • Do I just merge this PR into staging and will it then automatically merge into master when everything has build successfully?

  • Or does the latter need to happen manually?

  • What's the difference between staging and staging-next?

Member

basvandijk commented Sep 10, 2018

@vcunat nox-review mentions that this PR rebuilds over 1200 packages so I guess this first has to go through the staging branch. I'm unfamiliar with that procedure so I would appreciate some guidance and I see that you merge into staging a lot. So how does it work?

  • Do I just merge this PR into staging and will it then automatically merge into master when everything has build successfully?

  • Or does the latter need to happen manually?

  • What's the difference between staging and staging-next?

@vcunat

This comment has been minimized.

Show comment
Hide comment
@vcunat

vcunat Sep 11, 2018

Member

It's better, unless there's something that's urgent (e.g. security fixes). You just merge to the staging branch.

Everything is manual. Someone chooses some version of staging to stabilize – merges it to staging-next to stop inflow of other large changes, then repeatedly regressions are fixed (if any) and then it's merged to master. We recently added that intermediate step NixOS/rfcs#26 (not merged yet :)

Member

vcunat commented Sep 11, 2018

It's better, unless there's something that's urgent (e.g. security fixes). You just merge to the staging branch.

Everything is manual. Someone chooses some version of staging to stabilize – merges it to staging-next to stop inflow of other large changes, then repeatedly regressions are fixed (if any) and then it's merged to master. We recently added that intermediate step NixOS/rfcs#26 (not merged yet :)

@basvandijk

This comment has been minimized.

Show comment
Hide comment
@basvandijk

basvandijk Sep 11, 2018

Member

@vcunat thanks for the excellent explanation and RFC. Then I think I will be merging this into staging soon.

@vcunat as 18.09 release manager, is there a chance that this can still be cherry-picked in release-18.09 once this is merged in master? Or is that out of the question?

@GrahamcOfBorg test postgresql.postgresql10 postgresql.postgresql11 postgresql.postgresql93 postgresql.postgresql94 postgresql.postgresql95 postgresql.postgresql96

Member

basvandijk commented Sep 11, 2018

@vcunat thanks for the excellent explanation and RFC. Then I think I will be merging this into staging soon.

@vcunat as 18.09 release manager, is there a chance that this can still be cherry-picked in release-18.09 once this is merged in master? Or is that out of the question?

@GrahamcOfBorg test postgresql.postgresql10 postgresql.postgresql11 postgresql.postgresql93 postgresql.postgresql94 postgresql.postgresql95 postgresql.postgresql96

@GrahamcOfBorg

This comment has been minimized.

Show comment
Hide comment
@GrahamcOfBorg

GrahamcOfBorg Sep 11, 2018

Success on aarch64-linux (full log)

Attempted: tests.postgresql.postgresql10, tests.postgresql.postgresql11, tests.postgresql.postgresql93, tests.postgresql.postgresql94, tests.postgresql.postgresql95, tests.postgresql.postgresql96

Partial log (click to expand)

syncing
test script finished in 65.41s
vde_switch: EOF on stdin, cleaning up and exiting
cleaning up
/nix/store/mk98qvgs2w57i2fxc2g1c3amydp9ky9g-vm-test-run-postgresql10
/nix/store/j80bwkkqibwhrvy6rprlvpm1114f12xf-vm-test-run-postgresql11
/nix/store/l7fr93vxk247903k5iq04pq21s1zvn2x-vm-test-run-postgresql93
/nix/store/mh9p9njljlkr31saqxip7crpz3y0dn9w-vm-test-run-postgresql94
/nix/store/dp3n1hhydxxnazjwxph4qws0zyv8k59n-vm-test-run-postgresql95
/nix/store/zhi50awz9lhiq057kpvrfh6k08yniy71-vm-test-run-postgresql96

GrahamcOfBorg commented Sep 11, 2018

Success on aarch64-linux (full log)

Attempted: tests.postgresql.postgresql10, tests.postgresql.postgresql11, tests.postgresql.postgresql93, tests.postgresql.postgresql94, tests.postgresql.postgresql95, tests.postgresql.postgresql96

Partial log (click to expand)

syncing
test script finished in 65.41s
vde_switch: EOF on stdin, cleaning up and exiting
cleaning up
/nix/store/mk98qvgs2w57i2fxc2g1c3amydp9ky9g-vm-test-run-postgresql10
/nix/store/j80bwkkqibwhrvy6rprlvpm1114f12xf-vm-test-run-postgresql11
/nix/store/l7fr93vxk247903k5iq04pq21s1zvn2x-vm-test-run-postgresql93
/nix/store/mh9p9njljlkr31saqxip7crpz3y0dn9w-vm-test-run-postgresql94
/nix/store/dp3n1hhydxxnazjwxph4qws0zyv8k59n-vm-test-run-postgresql95
/nix/store/zhi50awz9lhiq057kpvrfh6k08yniy71-vm-test-run-postgresql96

@GrahamcOfBorg

This comment has been minimized.

Show comment
Hide comment
@GrahamcOfBorg

GrahamcOfBorg Sep 11, 2018

Success on x86_64-linux (full log)

Attempted: tests.postgresql.postgresql10, tests.postgresql.postgresql11, tests.postgresql.postgresql93, tests.postgresql.postgresql94, tests.postgresql.postgresql95, tests.postgresql.postgresql96

Partial log (click to expand)

syncing
test script finished in 121.01s
vde_switch: EOF on stdin, cleaning up and exiting
cleaning up
/nix/store/r6shs002pfg4fvw0avs2znz2fwky2v3f-vm-test-run-postgresql10
/nix/store/l41aj23bdi3wkya4wi1rvvznj3zrjvwr-vm-test-run-postgresql11
/nix/store/4r0nwlm32cjz2nwr3afs25dx2niyyhv0-vm-test-run-postgresql93
/nix/store/mpp9kzkaaffl33v450yf79ybjg64hwsf-vm-test-run-postgresql94
/nix/store/9qxywinb9kn5wnh7dh3mpnsqfzxkh26q-vm-test-run-postgresql95
/nix/store/wxxrnxsvrx8p8ghw2hjki0gwz2mh4hsm-vm-test-run-postgresql96

GrahamcOfBorg commented Sep 11, 2018

Success on x86_64-linux (full log)

Attempted: tests.postgresql.postgresql10, tests.postgresql.postgresql11, tests.postgresql.postgresql93, tests.postgresql.postgresql94, tests.postgresql.postgresql95, tests.postgresql.postgresql96

Partial log (click to expand)

syncing
test script finished in 121.01s
vde_switch: EOF on stdin, cleaning up and exiting
cleaning up
/nix/store/r6shs002pfg4fvw0avs2znz2fwky2v3f-vm-test-run-postgresql10
/nix/store/l41aj23bdi3wkya4wi1rvvznj3zrjvwr-vm-test-run-postgresql11
/nix/store/4r0nwlm32cjz2nwr3afs25dx2niyyhv0-vm-test-run-postgresql93
/nix/store/mpp9kzkaaffl33v450yf79ybjg64hwsf-vm-test-run-postgresql94
/nix/store/9qxywinb9kn5wnh7dh3mpnsqfzxkh26q-vm-test-run-postgresql95
/nix/store/wxxrnxsvrx8p8ghw2hjki0gwz2mh4hsm-vm-test-run-postgresql96

@thoughtpolice

This comment has been minimized.

Show comment
Hide comment
@thoughtpolice

thoughtpolice Sep 11, 2018

Member

Sorry about slacking on this folks, I was sick during the beginning of September which meant I missed the fork deadline. :( I'm willing to shepard these changes in still, providing we have enough time.

I'll review the rest of the info in this thread shortly.

Member

thoughtpolice commented Sep 11, 2018

Sorry about slacking on this folks, I was sick during the beginning of September which meant I missed the fork deadline. :( I'm willing to shepard these changes in still, providing we have enough time.

I'll review the rest of the info in this thread shortly.

@basvandijk

This comment has been minimized.

Show comment
Hide comment
@basvandijk

basvandijk Sep 11, 2018

Member

@thoughtpolice cool to have you back. Let me know how I can help.

Member

basvandijk commented Sep 11, 2018

@thoughtpolice cool to have you back. Let me know how I can help.

@thoughtpolice

This comment has been minimized.

Show comment
Hide comment
@thoughtpolice

thoughtpolice Sep 11, 2018

Member

So I think the immediate thing we should do is merge this to staging ASAP, in order to go ahead and get this in the pipeline to hit master. What I will probably do is rebase things again in order to get them in a clean state on top of staging, since it's still active right now, and it will trickle in through staging-next. I think most of the downstream consumers (the 1,200 packages) are mostly just due to PostgreSQL 10 becoming the default, requiring all libpq clients to be rebuilt.

Doing this I think will mostly just require fixing up the remaining tests and hitting The Big Button. (Unfortunately there was also a regression in the PostgreSQL 11 JIT support, meaning the closure size is bloated again, but I think I can fix it easily, and the user has to opt into 11beta2/11beta3 anyway...)

The NixOS 18.09 question is trickier. I have asked @samueldr about an exception on IRC, and I would also like @vcunat's opinion, so I await their judgement. @basvandijk, it looks like you're already doing this in your LumiGuide fork[1] -- so if we get an exception, I think we can use that as the basis for a cherry pick into staging-18.09 nicely as well, which will hopefully get brought in one final time before the release.

In the event this is barred, I'd suggest we can keep maintaining an independent fork that we merge with release-18.09 regularly, for people who want it, although it would be unfortunate (but understandable).

Realistically, my biggest concern about merging this to 18.09 is if the API needs to change again in any substantial way for 19.03, due to feedback from users. That would cause some annoying churn between two releases. But, I think even if I had merged this a month ago, we'd still be in the same place -- the vast majority of users would not be updating or testing this new API until after the release anyway. Meaning the end result would have been the same: we probably would only have reliable feedback well after "The point of no return". I think that's probably just part of the game...

Alternatively, if this hit 18.09 -- we could take any feedback from this new API and roll it into 19.09, in order to ensure we have adequate time for cleanup of prior code, and any future amendments, leaving the gap of 19.03 for one release. This would also be livable, in my opinion. I think that, given an exception, this would probably be the right thing to do, provided we had to rework anything substantially.

@vcunat, @samueldr -- please let us know what you think, and @basvandijk and I can certainly handle it.


[1] Just curious, are you already using this work at LumiGuide? I've been using it successfully for several months on my server, so I'm curious.

Member

thoughtpolice commented Sep 11, 2018

So I think the immediate thing we should do is merge this to staging ASAP, in order to go ahead and get this in the pipeline to hit master. What I will probably do is rebase things again in order to get them in a clean state on top of staging, since it's still active right now, and it will trickle in through staging-next. I think most of the downstream consumers (the 1,200 packages) are mostly just due to PostgreSQL 10 becoming the default, requiring all libpq clients to be rebuilt.

Doing this I think will mostly just require fixing up the remaining tests and hitting The Big Button. (Unfortunately there was also a regression in the PostgreSQL 11 JIT support, meaning the closure size is bloated again, but I think I can fix it easily, and the user has to opt into 11beta2/11beta3 anyway...)

The NixOS 18.09 question is trickier. I have asked @samueldr about an exception on IRC, and I would also like @vcunat's opinion, so I await their judgement. @basvandijk, it looks like you're already doing this in your LumiGuide fork[1] -- so if we get an exception, I think we can use that as the basis for a cherry pick into staging-18.09 nicely as well, which will hopefully get brought in one final time before the release.

In the event this is barred, I'd suggest we can keep maintaining an independent fork that we merge with release-18.09 regularly, for people who want it, although it would be unfortunate (but understandable).

Realistically, my biggest concern about merging this to 18.09 is if the API needs to change again in any substantial way for 19.03, due to feedback from users. That would cause some annoying churn between two releases. But, I think even if I had merged this a month ago, we'd still be in the same place -- the vast majority of users would not be updating or testing this new API until after the release anyway. Meaning the end result would have been the same: we probably would only have reliable feedback well after "The point of no return". I think that's probably just part of the game...

Alternatively, if this hit 18.09 -- we could take any feedback from this new API and roll it into 19.09, in order to ensure we have adequate time for cleanup of prior code, and any future amendments, leaving the gap of 19.03 for one release. This would also be livable, in my opinion. I think that, given an exception, this would probably be the right thing to do, provided we had to rework anything substantially.

@vcunat, @samueldr -- please let us know what you think, and @basvandijk and I can certainly handle it.


[1] Just curious, are you already using this work at LumiGuide? I've been using it successfully for several months on my server, so I'm curious.

@basvandijk

This comment has been minimized.

Show comment
Hide comment
@basvandijk

basvandijk Sep 11, 2018

Member

@thoughtpolice sounds like a plan.

Just curious, are you already using this work at LumiGuide?

No, not yet but today I started the migration to 18.09 so hopefully I will be able to use it soon.

@jchildren I see you're giving a talk at NixCon 2018 about building and testing PostgreSQL extensions with Nix. So you're probably interested in this work as well.

Member

basvandijk commented Sep 11, 2018

@thoughtpolice sounds like a plan.

Just curious, are you already using this work at LumiGuide?

No, not yet but today I started the migration to 18.09 so hopefully I will be able to use it soon.

@jchildren I see you're giving a talk at NixCon 2018 about building and testing PostgreSQL extensions with Nix. So you're probably interested in this work as well.

@thoughtpolice

This comment has been minimized.

Show comment
Hide comment
@thoughtpolice

thoughtpolice Sep 11, 2018

Member

Wait, @basvandijk -- how did you push to my fork exactly? I see this change on https://github.com/thoughtpolice/nixpkgs/tree/pgsql-fixes, but how can you push there? I don't have any collaborators, so this seems really weird!

Member

thoughtpolice commented Sep 11, 2018

Wait, @basvandijk -- how did you push to my fork exactly? I see this change on https://github.com/thoughtpolice/nixpkgs/tree/pgsql-fixes, but how can you push there? I don't have any collaborators, so this seems really weird!

@samueldr

This comment has been minimized.

Show comment
Hide comment
@samueldr

samueldr Sep 12, 2018

Member

The NixOS 18.09 question is trickier. I have asked @samueldr about an exception on IRC, and I would also like @vcunat's opinion, so I await their judgement.

For the record the conversation we had.

I'm not too hot on changing the semantics of stateVersion == 18.09 so close to the release, but not totally entirely against it. Though, combined with the idea that this is a huge change post-freeze, it's a bit spooky IMHO. My main thinking was seeing how anybody else concerned would react to the backport news, and seeing only agreement I would go for it, since we're still at the beta stage. Though, the roadmap makes me think it'll be late beta, and possibly hard to go for it.

Anyway, let's see other opinions on backporting post feature-freeze.

Member

samueldr commented Sep 12, 2018

The NixOS 18.09 question is trickier. I have asked @samueldr about an exception on IRC, and I would also like @vcunat's opinion, so I await their judgement.

For the record the conversation we had.

I'm not too hot on changing the semantics of stateVersion == 18.09 so close to the release, but not totally entirely against it. Though, combined with the idea that this is a huge change post-freeze, it's a bit spooky IMHO. My main thinking was seeing how anybody else concerned would react to the backport news, and seeing only agreement I would go for it, since we're still at the beta stage. Though, the roadmap makes me think it'll be late beta, and possibly hard to go for it.

Anyway, let's see other opinions on backporting post feature-freeze.

@danbst

This comment has been minimized.

Show comment
Hide comment
@danbst

danbst Sep 12, 2018

Contributor

I guess those who rely on Postgresql will appreciate the fix of packaging situation, even if it requires postfixups in release branch.

Contributor

danbst commented Sep 12, 2018

I guess those who rely on Postgresql will appreciate the fix of packaging situation, even if it requires postfixups in release branch.

@basvandijk

This comment has been minimized.

Show comment
Hide comment
@basvandijk

basvandijk Sep 12, 2018

Member

Since this PR also touches the vital hydra module:
@GrahamcOfBorg test hydra

Member

basvandijk commented Sep 12, 2018

Since this PR also touches the vital hydra module:
@GrahamcOfBorg test hydra

@GrahamcOfBorg

This comment has been minimized.

Show comment
Hide comment
@GrahamcOfBorg

GrahamcOfBorg Sep 12, 2018

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.hydra

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.


GrahamcOfBorg commented Sep 12, 2018

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.hydra

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.


@GrahamcOfBorg

This comment has been minimized.

Show comment
Hide comment
@GrahamcOfBorg

GrahamcOfBorg Sep 12, 2018

Failure on x86_64-linux (full log)

Attempted: tests.hydra

Partial log (click to expand)

Sep 12 07:15:29 machine systemd[1]: Failed to start PostgreSQL Server.
Hint: Some lines were ellipsized, use -l to show in full.
error: command `systemctl status postgresql.service' did not succeed (exit code 3)
command `systemctl status postgresql.service' did not succeed (exit code 3)
cleaning up
killing machine (pid 600)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
builder for '/nix/store/nr0dlxd6s49a4w27vpw2an79gh0xlb70-vm-test-run-hydra-init-localdb.drv' failed with exit code 255
error: build of '/nix/store/nr0dlxd6s49a4w27vpw2an79gh0xlb70-vm-test-run-hydra-init-localdb.drv' failed

GrahamcOfBorg commented Sep 12, 2018

Failure on x86_64-linux (full log)

Attempted: tests.hydra

Partial log (click to expand)

Sep 12 07:15:29 machine systemd[1]: Failed to start PostgreSQL Server.
Hint: Some lines were ellipsized, use -l to show in full.
error: command `systemctl status postgresql.service' did not succeed (exit code 3)
command `systemctl status postgresql.service' did not succeed (exit code 3)
cleaning up
killing machine (pid 600)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
builder for '/nix/store/nr0dlxd6s49a4w27vpw2an79gh0xlb70-vm-test-run-hydra-init-localdb.drv' failed with exit code 255
error: build of '/nix/store/nr0dlxd6s49a4w27vpw2an79gh0xlb70-vm-test-run-hydra-init-localdb.drv' failed

@vcunat

This comment has been minimized.

Show comment
Hide comment
@vcunat

vcunat Sep 12, 2018

Member

My sentiment is similar to Samuel's. If this gets enough testing, no significant regressions and consensus among pgsql users in nixpkgs that it's suitable for such a fast inclusion, then it seems fine. It's typically not easy to achieve these so fast, but it seems possible (at least in theory).

Member

vcunat commented Sep 12, 2018

My sentiment is similar to Samuel's. If this gets enough testing, no significant regressions and consensus among pgsql users in nixpkgs that it's suitable for such a fast inclusion, then it seems fine. It's typically not easy to achieve these so fast, but it seems possible (at least in theory).

@basvandijk

This comment has been minimized.

Show comment
Hide comment
@basvandijk

basvandijk Sep 12, 2018

Member

The hydra test failure is due to the postgresql systemd service taking more than 2m to start up causing a timeout. Probably because ofborg is under heavy load. The test succeeds on my system.

Member

basvandijk commented Sep 12, 2018

The hydra test failure is due to the postgresql systemd service taking more than 2m to start up causing a timeout. Probably because ofborg is under heavy load. The test succeeds on my system.

@basvandijk

This comment has been minimized.

Show comment
Hide comment
@basvandijk

basvandijk Sep 16, 2018

Member

@thoughtpolice is this already merged with staging? (I don't have my laptop with me so I it's hard to check myself).

Member

basvandijk commented Sep 16, 2018

@thoughtpolice is this already merged with staging? (I don't have my laptop with me so I it's hard to check myself).

@thoughtpolice

This comment has been minimized.

Show comment
Hide comment
@thoughtpolice

thoughtpolice Sep 17, 2018

Member

No, I have some WIP branch to merge/rebase it on staging myself; meant to get to it this weekend but was distracted the whole time. Should come later tonight.

Member

thoughtpolice commented Sep 17, 2018

No, I have some WIP branch to merge/rebase it on staging myself; meant to get to it this weekend but was distracted the whole time. Should come later tonight.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment