Support for upstream postgresql.org zypp repo #176

noelmcloughlin · 2018-01-14T22:05:07Z

This PR is an enhancement to allow the upstream repo to be used for SLES and OpenSUSE.

This rebased commit replaces PR #169

Tested successfully on version '9.6' and '10'. Failed with 9.5 due to upstream repo problem.

postgres:
  use_upstream_repo: True
  version: '10'
  pkgs_extra:
  - postgresql10-contrib
  - postgresql10-plpython

base:
  '*':
    - postgres
    - postgres.client
    - postgres.manage
    - postgres.python
    - postgres.server
    - postgres.server.image
    - postgres.upstream

---------
          ID: postgresql-repo
    Function: pkgrepo.managed
        Name: pgdg-sles-10
      Result: True
     Comment: Configured package repo 'pgdg-sles-10'
     Started: 14:32:38.642363
    Duration: 47.584 ms
     Changes:   
----------
          ID: postgresql-repo
    Function: cmd.run
        Name: zypper --gpg-auto-import-keys refresh --force
      Result: True
     Comment: Command "zypper --gpg-auto-import-keys refresh --force" run
     Started: 14:32:38.691449
    Duration: 8691.113 ms
     Changes:   
              ----------
              pid:
                  13510
              retcode:
                  0
              stderr:
              stdout:
                  Forcing raw metadata refresh
                  Retrieving repository 'pgdg-sles-10' metadata [...done]
                  Forcing building of repository cache
                  Building repository 'pgdg-sles-10' cache [....done]
                  Forcing raw metadata refresh
                  Retrieving repository 'openSUSE-Leap-42.3-Non-Oss' metadata [.done]
                  Forcing building of repository cache
                  Building repository 'openSUSE-Leap-42.3-Non-Oss' cache [....done]
                  Forcing raw metadata refresh
                  Retrieving repository 'openSUSE-Leap-42.3-Oss' metadata [..done]
                  Forcing building of repository cache
                  Building repository 'openSUSE-Leap-42.3-Oss' cache [....done]
                  Forcing raw metadata refresh
                  Retrieving repository 'openSUSE-Leap-42.3-Update' metadata [..done]
                  Forcing building of repository cache
                  Building repository 'openSUSE-Leap-42.3-Update' cache [....done]
                  Forcing raw metadata refresh
                  Retrieving repository 'openSUSE-Leap-42.3-Update-Non-Oss' metadata [..done]
                  Forcing building of repository cache
                  Building repository 'openSUSE-Leap-42.3-Update-Non-Oss' cache [....done]
                  Forcing raw metadata refresh
                  Retrieving repository 'Salt releases for SLE-based SUSE products (openSUSE_Leap_42.3)' metadata [..done]
                  Forcing building of repository cache
                  Building repository 'Salt releases for SLE-based SUSE products (openSUSE_Leap_42.3)' cache [....done]
                  All repositories have been refreshed.
----------
          ID: postgresql-server
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 14:32:47.389718
    Duration: 314.426 ms
     Changes:   
----------
          ID: postgresql-cluster-prepared
    Function: cmd.run
        Name: initdb --pgdata='/var/lib/pgsql/10/data'
      Result: True
     Comment: unless execution succeeded
     Started: 14:32:47.711825
    Duration: 45.167 ms
     Changes:   
----------
          ID: postgresql-config-dir
    Function: file.directory
        Name: /var/lib/pgsql/10/data
      Result: True
     Comment: Directory /var/lib/pgsql/10/data is in the correct state
              Directory /var/lib/pgsql/10/data updated
     Started: 14:32:47.758883
    Duration: 0.474 ms
     Changes:   
----------
          ID: postgresql-pg_hba
    Function: file.managed
        Name: /var/lib/pgsql/10/data/pg_hba.conf
      Result: True
     Comment: File /var/lib/pgsql/10/data/pg_hba.conf is in the correct state
     Started: 14:32:47.759573
    Duration: 162.051 ms
     Changes:   
----------
          ID: postgresql-running
    Function: service.running
        Name: postgresql-10
      Result: True
     Comment: The service postgresql-10 is already running
     Started: 14:32:47.914996
    Duration: 22.87 ms
     Changes:   
----------
          ID: postgresql-client-libs
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 14:32:47.945155
    Duration: 19.896 ms
     Changes:   
----------
          ID: postgres-reload-modules
    Function: test.nop
      Result: True
     Comment: Success!
     Started: 14:32:47.965238
    Duration: 0.398 ms
     Changes:   
----------
          ID: postgresql-python
    Function: pkg.installed
        Name: python-psycopg2
      Result: True
     Comment: All specified packages are already installed
     Started: 14:32:47.965724
    Duration: 22.854 ms
     Changes:   
----------
          ID: postgresql-start
    Function: test.show_notification
      Result: True
     Comment: The 'postgres:bake_image' Pillar is disabled (set to 'False').
     Started: 14:32:47.988690
    Duration: 0.228 ms
     Changes:   

Summary for local
-------------
Succeeded: 11 (changed=1)
Failed:     0
-------------
Total states run:     11
Total run time:    9.334 s

noelmcloughlin · 2018-01-15T14:51:40Z

Hi @vutny Can you merge this if you are happy please?

noelmcloughlin · 2018-01-19T13:07:29Z

Is anyone able to look at this?

vutny

Thanks @noelmcloughlin !

This looks pretty good. I have few suggestions to make it little bit more robust, so will post in-line comments by one on a time. Feel free to discuss. This should be merged eventually!

vutny · 2018-01-24T09:12:48Z

pillar.example

-  pkg: 'postgresql-9.3'
-  pkg_client: 'postgresql-client-9.3'
+  pkg: 'postgresql-9.6'
+  pkg_client: 'postgresql-client-9.6'


As the comment above said, these values are specific to Ubuntu, 14.04 to be precise.
This is because the pillar.example is used in test-kitchen within the formula.
That's kind of legacy, but still serves as smoke test for regressions.

Since now we have good support for package names mapping for various distros, I think it could be safely commented out.

Can we go further and remove all package names from pillar.example?

Executing the formula with pillars use_upstream_repo: True with pkg_extras: ['postgresql-contrib', 'postgresql-plpython'] corrupts the postgres installation for example. Package names as pillars are not needed imho.

And removeservice pillar too. And pkgs_extra example pillar implies postgres-contrib is optional "extra package", but the postgres.prepare_cluster.command fails (cannot find initdb) if the contrib package is not installed, implying its needed.

You're right, the package and service definitions in Pillar are mostly unneeded.
But they could be useful when trying to install something on a latest-greatest and still unsupported distro.
I think it would be better to add a comment that would mention they are unnecessary in most cases.

I'll think about this, thanks.

Sure, just for now I think we should comment them out to make test-kitchen work, but add a proper description for these parameter later.
Or use generic ones like postgresql and postgresql-client. It will make transition to newer distro version much smoother as well.

vutny · 2018-01-25T07:59:47Z

pillar.example

-    - postgresql-contrib
-    - postgresql-plpython
+  #pkgs_extra:
+  #  - postgresql-contrib


Regarding your comment above. If we're talking about Ubuntu, postgresql-contrib is definitely optional package, it does not contain initdb binary.
https://packages.ubuntu.com/trusty/amd64/postgresql-contrib-9.3/filelist
https://packages.ubuntu.com/xenial/amd64/postgresql-contrib-9.5/filelist

I think the issue with missing initdb lies somewhere else, maybe in our beloved alternatives?
So commenting out these extras is a good thing.

Yeah, postgres-contrib creates alternatives. The formula requires initdb in PATH (see below) or state failures occur.

./postgres/defaults.yaml: command: initdb --pgdata=/var/lib/pgsql/data ./postgres/osmap.yaml: command: initdb -D /var/lib/postgres/data ./postgres/osmap.yaml: command: initdb --pgdata='{{ data_dir }}' ./postgres/osmap.yaml: - initdb ./postgres/osmap.yaml: command: initdb --pgdata='{{ lib_dir }}' ./postgres/osmap.yaml: command: initdb -D /Users/{{ repo.user }}/Library/AppSupport/postgres_{{ repo.use_upstream_repo }}

Do you know how initdb gets added to PATH without postgres-contrib package in this formula? The redhat os mapping invokes saltstack to create alternatives, so the package must have been missing when that feature was added.

Back to the specific comment - I would prefer to remove pkgs_extra pillar example, because it corrupted my installation once or twice, this is bad onboarding experience for new users. Otherwise commenting is definitely a good thing - I agree and will implement either solution.

This is because of the postgres-contrib is just a metapackage that also installs a full PostgreSQL server + client of particular version available in the distro by dependencies. And there we have all the alternatives installation code in the downstream.
So if you have a wrong or empty value in the pkg Pillar value, the postgres-contrib just does the right thing.

This is just a very handy coincidence, that does not make postgres-contrib package mandatory.

Okay. In this PR I explicitly fix the INITDB problem (#160) via Full Path. As documented in #160, upstream packages do not add initdb to PATH by design.

Here is my verified solution - I am unaware of any postgres-defined solution.

... + prepare_cluster: + #Full path needed as initdb is NOT 'cross version compatible' binary + command: /usr/pgsql-{{ release }}/bin/initdb --pgdata='{{ lib_dir }}' + test: test -f '{{ lib_dir }}/PG_VERSION' + + # Handles etc alternatives for 'cross version compatible' binaries + pkgs_extra: ['postgresql{{ release }}-contrib',] + +{% else %}

If this solution is acceptable to you, let me know.

Note: For upstream repos, pillars values which override verified OS mapping defaults usually breaks things. Setting pkgs_extra: postgresql-contrib is particularly dangerous because it pulls distro dependencies unto upstream packages resulting in a mess.

Yeah. That is a separate issue appearing exclusively when installing from upstream.
We will handle this. Thank you.

vutny · 2018-01-29T08:51:16Z

postgres/client.sls

@@ -23,11 +23,14 @@ postgresql-client-libs:
    - refresh: True
    - require:
      - pkgrepo: postgresql-repo
+       {% if grains.os_family == 'Suse' %}
+      - cmd: postgresql-repo
+       {% endif %}


I think here would be better to depend on the whole postgres.upstream SLS to reduce verbose Jinja statements:

... - require: - sls: postgres.upstream

vutny · 2018-01-29T08:58:17Z

postgres/client.sls

 {%- endif %}

 {%- if 'bin_dir' in postgres %}

-# Make client binaries available in $PATH
+  # Make client binaries available in $PATH


No need for indenting these comments, since they're part of rendered YAML, not Jinja.
It would look better in debug log when they are aligned with states they describe.

vutny · 2018-01-29T09:01:50Z

postgres/client.sls

@@ -23,6 +23,9 @@ postgresql-client-libs:
    - refresh: True
    - require:
      - pkgrepo: postgresql-repo
+       {% if grains.os_family == 'Suse' %}


I meant, no need for {% if grains... %} statements and pkgrepo requisite above as well 😃

no problem. check now.

vutny · 2018-01-29T12:33:22Z

postgres/upstream.sls

 postgresql-repo:
  pkgrepo.managed:
    {{- format_kwargs(postgres.pkg_repo) }}
+    {% if grains.os_family == 'Suse' %}
+  cmd.run:
+    - name: zypper --gpg-auto-import-keys refresh --force


This is in connection with what we've discussed about requisites for the pkg state.
I know it is not obvious from the Salt docs, but pkgrepo.managed state supports special option gpgautoimport when target package manager is Zypper.
No need for messing with grains here. I'll show how simple it could be done above at osmap.yaml file.

How did you know that ;-) I did check the docs.

vutny · 2018-01-29T12:35:08Z

postgres/osmap.yaml

+    #Using sles-12 upstream repo for opensuse
+    baseurl: 'https://download.postgresql.org/pub/repos/zypp/{{ repo.version }}/suse/sles-12-$basearch'
+    key_url: 'https://download.postgresql.org/pub/repos/zypp/{{ repo.version }}/suse/sles-12-$basearch/repodata/repomd.xml.key'
+    gpgcheck: 1


Just add

gpgautoimport: True

bellow, and importing GPG key from the upstream will start to magically work. No need for additional cmd.run state in the upstream.sls.

Thanks - done - let me know when your PR comments are done and I'll do some retesting on Suse to make sure there is no regression.

vutny · 2018-01-30T12:45:23Z

postgres/osmap.yaml

+    test: test -f '{{ lib_dir }}/PG_VERSION'
+
+  # Handles etc alternatives for 'cross version compatible' binaries
+  pkgs_extra: ['postgresql{{ release }}-contrib',]


I've checked out the postgresql10-contrib-10.1-1PGDG.sles12.x86_64.rpm and appears it has nothing to do with alternatives at all.
I think we should disable extra packages installation by default, but may add a comment to pillar.example file mentioning how typically they called if having use_upstream_repo enabled.

Thank you for checking. I fully agree extra package installation can be disabled by default for general OS mapping/defaults/pillar.example.

I will retest with Zypp upstream and if not needed will also ensure its disabled for Suse.

vutny · 2018-01-31T09:48:27Z

postgres/osmap.yaml

+  pkg_client: postgresql{{ release }}
+  conf_dir: {{ lib_dir }}
+   {% if release|int == 10 %}
+  service: postgresql-{{ release }}


I've checked out postgresql96-server-9.6.6-1PGDG.sles12.x86_64.rpm package, and there the service name follows the same pattern. I think we should omit release version check above.

noelmcloughlin · 2018-02-05T14:46:51Z

Hi @vutny, I have done testing and few updates were needed to consistently pass verification. Please review again when you have time. Thanks.

noelmcloughlin · 2018-02-12T10:39:37Z

Hi @vutny can I merge this?

vutny

@noelmcloughlin This PR looks better and better. Good job!

But I doubt about introducing postgres.pkg_contrib property...

vutny · 2018-02-12T14:11:54Z

postgres/server/init.sls

@@ -15,7 +15,11 @@ include:

 {%- endif %}

-{%- set pkgs = [postgres.pkg] + postgres.pkgs_extra %}
+{%- if postgres.extensions %}
+  {%- set pkgs = [postgres.pkg] + [postgres.pkg_contrib] + postgres.pkgs_extra %}


This statement looks very verbose and unnecessary...
It declares strait dependency between extensions setting in Pillar and newly introduced pkg_contrib value, which in turn seems to be applicable only on SUSE distros.

What if desired extensions are not shipped with package defined in pkg_contrib?

I see it was set empty (evaluates to None) for other platforms. Will it work if some extensions have been set?

It looks little bit confusing, and is not described anywhere.
I suggest to keep the things as is: if user needs some extension, he or she should provide extra packages via pkgs_extra or install them another way.

Thanks @vutny The extensions have dependency on some package in PG and its gray area causing issues like #172 - suggestions welcomed there. I agree to remove this from PR code.

noelmcloughlin · 2018-02-12T16:05:29Z

I've removed the if postgres.extensions check.

vutny · 2018-02-13T15:38:46Z

postgres/osmap.yaml

+  pkg_libpq_dev: libpqxx
+  pkg_dev: postgresql{{ release }}-devel
+  pkg_libs: postgresql{{ release }}-libs
+  pkg_contrib: postgresql{{ release }}-contrib


Here and below, the pkg_contrib parameter becomes unused. So no need to keep it.

vutny · 2018-02-13T15:40:41Z

postgres/osmap.yaml

-  pkg_libpq_dev: postgresql
+  pkg_libpq_dev: libpq5
+  pkg_libs: postgresql-libs
+  pkg_contrib: postgresql-contrib


You may put the postgresql-contrib back into the proper section in pillar.example file with comment saying this package could be a requirement for installing some extensions.

I don't want to. Pillar handling exposes production users to unacceptable risk.

Handling packages via osmap.yaml pattern is good.

Handling packages only via pillars (i.e. postgresql_contrib) is bad pattern.
-- Missing Postgresql-contrib package causes bug State postgres_extension-uuid-ossp fails #172.
-- Specifying wrong contrib_pkg pillar value corrupts package manager, PG installation, and worse, risks data loss. The PG schema may get upgraded.

And trying to handle State postgres_extension-uuid-ossp fails #172 via comments is wrong approach.

I think these risk need discussion.

I mostly agree with all your points.
My main point is that pkg_contrib have not been used within current state of your changes, so having this in defaults and OS mapping files is meaningless right now. It just does nothing.
And this does not really help to solve issue #172 taking into an account broad scope of distributions supported by the formula.
I will try to put my thoughts there. Let's concentrate on bringing in initial basic support for SUSE systems (maybe less functional), but without breaking things which already work.

I'll remove these entries from osmap.yaml but don't want pkg_contrib in pillar.example. Apart from the reasons above, I've done lots of verification on this formula past few weeks across many scenarios and I just don't agree with packages ONLY in pillars anymore, should be hidden from users. But, I do want the PR merged, so tell me if you want pillar.example restored.

okay committed.

vutny · 2018-02-14T12:14:34Z

postgres/osmap.yaml

  pkg: postgresql-server
  pkg_client: postgresql
-  pkg_libpq_dev: postgresql
+  pkg_libpq_dev: libpq5


I found that OpenSuse also has C++ bindings package named libpqxx similar to the upstream.

Thanks for checking. Should libpqxx value replace 'libpq5'?

Yeah, exactly as in the upstream case to have everything consistent.

Where did libpq5 come from? Cut and paste? maybe by me? ;-(

Committed

vutny · 2018-02-16T14:52:52Z

postgres/defaults.yaml

@@ -5,6 +5,7 @@ postgres:
  version: '9.5'
  pkg: postgresql
  pkgs_extra: []
+  pkg_contrib:


This one has been forgotten.

Got it!! Sorry.

vutny · 2018-02-26T10:21:34Z

postgres/osmap.yaml

+
+  prepare_cluster:
+    #Full path needed as initdb is NOT 'cross version compatible' binary
+    command: /usr/pgsql-{{ repo.version }}/bin/initdb --pgdata='{{ lib_dir }}'


This is generally OK. But I would recommend you to consider providing bin_dir and *_bins parameters similar to RedHat section for postgres.manage state to work on Suse out of the box as well.
But that could be added later.

Also #190 captures problem regarding package names in pillar data.

vutny · 2018-02-26T11:35:52Z

pillar.example

+  # pkg: 'postgresql'
+  # pkg_client: 'postgresql-client'
+  # service: postgresql
+  #pkgs_extra:


This pkg_extra parameter still concerns me.

I thought we reached agreement that pillar.example file could not be used as valid source for any possible platform which the formula supports. Hard-coding some package names is bad thing too.
But now this breaks the test kitchen run.
We have two options:

Uncomment pkgs_extra and add postgresql-contrib there mentioned this could be used only with use_upstream_repo: False.

...or replace uuid-ossp extension with plpgsql, that comes with PG server. The schema name also should be changed to some custom_schema for consistency.

I have reinstated pkgs_extra as per original pillar.example.

Yeah, I'm unconvinced about pillar.example. The README and pillar.example are both delivered artifacts, responsible for onboarding new formula users (perhaps in a 1 week agile sprint), and guiding adoption for happy and unhappy paths. test kitchen is a platform too. On the other hand, I agree the pillar.example must have limited scope, which aligns with your stated view.

vutny

Excellent work. Thank you for constructive collaboration.
👍

noelmcloughlin · 2018-03-05T23:49:52Z

Excellent work reviewing this, thanks.

aboe76 · 2018-03-06T11:28:14Z

@noelmcloughlin and @vutny thanks for all the hard work.

This was referenced Jan 14, 2018

Suse: Introduce support for upstream repo #169

Closed

Support for PostgresApp on MacOS #178

Closed

noelmcloughlin mentioned this pull request Jan 15, 2018

Support for Homebrew Postgres on MacOS #179

Closed

vutny requested changes Jan 24, 2018

View reviewed changes

vutny reviewed Jan 25, 2018

View reviewed changes

noelmcloughlin mentioned this pull request Jan 28, 2018

Upstream Packaging: /bin/bash: initdb: command not found #160

Closed

vutny reviewed Jan 29, 2018

View reviewed changes

vutny reviewed Jan 30, 2018

View reviewed changes

vutny reviewed Jan 31, 2018

View reviewed changes

This was referenced Feb 5, 2018

Support fromrepo in pkg.installed states #185

Closed

State postgres_extension-uuid-ossp fails #172

Closed

vutny requested changes Feb 12, 2018

View reviewed changes

vutny reviewed Feb 13, 2018

View reviewed changes

vutny reviewed Feb 14, 2018

View reviewed changes

vutny reviewed Feb 16, 2018

View reviewed changes

vutny reviewed Feb 26, 2018

View reviewed changes

Support for upstream postgresql.org zypp repo after PR review

22b11fd

noelmcloughlin mentioned this pull request Feb 27, 2018

Is bin_dir and *bins missing for Suse in osmap.yml #189

Closed

vutny approved these changes Feb 27, 2018

View reviewed changes

aboe76 merged commit 08ad908 into saltstack-formulas:master Mar 6, 2018

noelmcloughlin mentioned this pull request Dec 2, 2020

[BUG] nothing provides libicui18n.so.52.1()(64bit) for postgresql10 #302

Closed

Support for upstream postgresql.org zypp repo #176

Support for upstream postgresql.org zypp repo #176

Conversation

noelmcloughlin commented Jan 14, 2018 • edited Loading

noelmcloughlin commented Jan 15, 2018

noelmcloughlin commented Jan 19, 2018

vutny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noelmcloughlin Jan 24, 2018 • edited Loading

Choose a reason for hiding this comment

noelmcloughlin Jan 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noelmcloughlin Jan 25, 2018 • edited Loading

Choose a reason for hiding this comment

vutny Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

noelmcloughlin Jan 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noelmcloughlin Jan 30, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noelmcloughlin commented Feb 5, 2018

noelmcloughlin commented Feb 12, 2018

vutny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noelmcloughlin commented Feb 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noelmcloughlin Feb 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noelmcloughlin Feb 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noelmcloughlin Feb 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vutny left a comment

Choose a reason for hiding this comment

noelmcloughlin commented Mar 5, 2018

aboe76 commented Mar 6, 2018

noelmcloughlin commented Jan 14, 2018 •

edited

Loading

noelmcloughlin Jan 24, 2018 •

edited

Loading

noelmcloughlin Jan 24, 2018 •

edited

Loading

noelmcloughlin Jan 25, 2018 •

edited

Loading

vutny Jan 26, 2018 •

edited

Loading

noelmcloughlin Jan 28, 2018 •

edited

Loading

noelmcloughlin Jan 30, 2018 •

edited

Loading

noelmcloughlin Feb 14, 2018 •

edited

Loading

noelmcloughlin Feb 16, 2018 •

edited

Loading

noelmcloughlin Feb 16, 2018 •

edited

Loading