Improve BuildRequires use #28

phillxnet · 2023-07-23T19:15:36Z

We currently make almost no use of the BuildRequires section. It is proposed that we bring this key rpmbuild feature into play more to enable a better out-of-the-box experience re rpmbuild runs on say Minimal-VM images. Our new Poetry based install method requires that we build many of our required dependencies, and so we have specified these within the final install dependencies section. But this ignores the initial rpmbuild requirements that can be well served by BuildRequires.

These changes should be address first in the testing branch.

phillxnet · 2023-07-28T18:08:31Z

The results of some initial experimentation with this issue look promising:

lbuildvm:/opt/rockstor-rpmbuild # rpmbuild -ba -v rockstor.spec
warning: Downloading https://github.com/rockstor/rockstor-core/archive/refs/tags/5.0.1-0.tar.gz to /usr/src/packages/SOURCES/5.0.1-0.tar.gz
warning: Downloading https://github.com/rockstor/rockstor-jslibs/archive/refs/tags/5.0.1.tar.gz to /usr/src/packages/SOURCES/5.0.1.tar.gz
error: Failed build dependencies:
        dbus-1-devel is needed by rockstor-5.0.1-0.x86_64
        gcc-c++ is needed by rockstor-5.0.1-0.x86_64
        glib2-devel is needed by rockstor-5.0.1-0.x86_64
        postgresql13 is needed by rockstor-5.0.1-0.x86_64
        postgresql13-server is needed by rockstor-5.0.1-0.x86_64
        postgresql13-server-devel is needed by rockstor-5.0.1-0.x86_64
        python3-devel is needed by rockstor-5.0.1-0.x86_64

This is a far faster and more informative fail than during the Poetry venv creation for the %check run of all out tests.

I'll continue with abstracting all that is needed to build - on top of rpm-build's own dependencies, which do overlap with our requirements a little:

rpm-build dependencies (Leap 15.5 Minimal_VM image)

lbuildvm:/opt # zypper install --no-recommends rpm-build
...
The following 31 NEW packages are going to be installed:
  binutils bzip2 cpp cpp7 dwz gcc gcc7 gettext-tools glibc-devel glibc-locale libasan4 libatomic1 libcilkrts5 libctf-nobfd0 libctf0 libgomp1 libisl15 libitm1
  liblsan0 libmpc3 libmpfr6 libmpx2 libmpxwrappers2 libtsan0 libubsan0 libxcrypt-devel linux-glibc-devel make patch python-rpm-macros rpm-build

31 new packages to install.
Overall download size: 51.8 MiB. Already cached: 0 B. After the operation, additional 370.6 MiB will be used.
Continue? [y/n/v/...? shows all options] (y): y

phillxnet · 2023-07-28T18:11:56Z

zypper in --no-recommends dbus-1-devel gcc-c++ glib2-devel postgresql13 postgresql13-server postgresql13-server-devel python3-devel

Followed by:

lbuildvm:/opt/rockstor-rpmbuild # rpmbuild -ba -v rockstor.spec
...
+ /usr/bin/git init -q
/var/tmp/rpm-tmp.aVuAjQ: line 47: /usr/bin/git: No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.aVuAjQ (%prep)

So missing git: we use the git options available in %prep to prepare our code, adding accordingly to build requirements: they are likely also a recommended to rpm-build itself but I want to have the minimum required here to improve our chances of doing test rpm builds (and consequently running our tests) on small hosts.

zypper in --no-recommends git

phillxnet · 2023-07-28T18:27:03Z

We have of course to start, and possibly configure HBA access to postgresql before running rpmbuild.
In our README.md we suggest using a rockstor install: but this limits the ease (read complicates) our ability to source test rpm build hosts.

  File "/usr/src/packages/BUILD/rockstor-core-5.0.1-0/.venv/lib64/python3.6/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory

        Is the server running locally and accepting connections on that socket?

error: Bad exit status from /var/tmp/rpm-tmp.1nTwWV (%check)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.1nTwWV (%check)
``

phillxnet · 2023-07-28T18:49:36Z

Short of finding how we BuildRequires on a systemd service running we will have to advise:

systemctl start postgresql

but we then have, of course, to configure HBA on the local postgresql - as we do in initrock:

rpmbuild -ba -v rockstor.spec
...
django.db.utils.OperationalError: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  Peer authentication failed for user "rocky"

error: Bad exit status from /var/tmp/rpm-tmp.9b3k9N (%check)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.9b3k9N (%check)

phillxnet · 2023-07-28T19:05:38Z

See the modifications used by our initrock (rockstor-pre.service) to acomodate our testing requirements of postgresql:

rockstor/rockstor-core@500265b

phillxnet · 2023-07-28T19:20:15Z

To enable the use of this repo to help with testing rpmbuld function: we might consider adding a script that will allow configure Postgres sufficiently to run our tests - a requirement for a successful rpm package build. This may help with setting up transient workers to test-build rpm packages (and thus run our tests also) at key points in development.

phillxnet · 2023-07-31T12:34:44Z

HBA Host Based Authentication (Postgresql)

Akin to our behaviour in initrock (rockstor-pre.servcie invoked):
https://github.com/rockstor/rockstor-core/blob/5252197d35cde9ca7ad95bfafd4ac859da9d6103/src/rockstor/scripts/initrock.py#L66C21-L66C62

we can instantiate pg_hba.conf as follows (dedicated test systems only):
Note that this allows local host access as per rockstor's expectation to specifically named databases.
N.B. the following example is for the testing branch:

curl https://raw.githubusercontent.com/rockstor/rockstor-core/testing/conf/pg_hba.conf --output /var/lib/pgsql/data/pg_hba.conf

phillxnet · 2023-07-31T13:04:04Z

DB setup as per initrock (minimal for tests) - WILL DESTROY EXISTING DB.

To run our tests on a non prior rockstor instance we need to establish db as per initrock.
Briefely abstracted:

DB tune:

curl https://raw.githubusercontent.com/rockstor/rockstor-core/testing/conf/postgresql_tune.sql --output /tmp/postgresql_tune.sql 
su - postgres -c "psql -w -f /tmp/postgresql_tune.sql"

DB setup:

curl https://raw.githubusercontent.com/rockstor/rockstor-core/testing/conf/postgresql_setup.sql --output /tmp/postgresql_setup.sql
su - postgres -c "psql -w -f /tmp/postgresql_setup.sql"

In our proof of concept test-running Minimal-VM we have errors concerning pre-existing postgresql:

lbuildvm:/opt/rockstor-rpmbuild # su - postgres -c "psql -w -f /tmp/postgresql_setup.sql"
DROP DATABASE
DROP DATABASE
psql:/tmp/postgresql_setup.sql:3: NOTICE:  database "test_smartdb" does not exist, skipping
DROP DATABASE
psql:/tmp/postgresql_setup.sql:4: NOTICE:  database "test_storageadmin" does not exist, skipping
DROP DATABASE
psql:/tmp/postgresql_setup.sql:5: ERROR:  cannot drop the currently open database
DROP ROLE
CREATE ROLE
CREATE DATABASE
CREATE DATABASE
psql:/tmp/postgresql_setup.sql:10: ERROR:  database "postgres" already exists

However we have our slightly more than minimal db tune/setup for establishing a throw-away test runner via a %check rpmbuild:

+ poetry run django-admin test
Creating test database for alias 'default'...
Creating test database for alias 'smart_manager'...
System check identified no issues (0 silenced).
.F......F.............E......E.........................F.....E.FF.....F.................................................F..EE.F....................F.......F.................................................................................................
...

phillxnet · 2023-07-31T13:16:39Z

OS setup related failures that could constitute incomplete OS for test runs (remit of issue re BuildRequires):

FileNotFoundError: [Errno 2] No such file or directory: '/usr/sbin/dmsetup': '/usr/sbin/dmsetup'
FileNotFoundError: [Errno 2] No such file or directory: '/etc/postfix/master.cf'
'Failed to restart avahi-daemon.service: Unit avahi-daemon.service not found.'
Failed to enable unit: Unit file snmpd.service does not exist.
No such file or directory: '/etc/postfix/master.cf
No such file or directory: '/usr/bin/nmcli': '/usr/bin/nmcli'\n'
No such file or directory: '/usr/sbin/exportfs': '/usr/sbin/exportfs'\n'
User (lp) does not exist.', 'NoneType: None\n

All of the above very likely represent unit test improvements to be had regarding unwanted/unrequired OS sensitivity during test runs. I.e. insufficient mocking of OS components.

The above likely false dependencies/requirements, that might otherwise be satisfied by additional BuildRequires entries, have been spun-off to the now linked issue:

"Improve OS independence re unit tests" rockstor/rockstor-core#2633

EDIT: pull request for spin-off issue:
rockstor/rockstor-core#2637

Use our existing OS flavour/version sections for Leap/TW to add a proposed minimum BuildRequires for a rpmbuild (%check included) on a Minimal_VM host of our upstream OS. Given external postgresql install and config.

Improve BuildRequires use #28

phillxnet · 2023-08-02T09:51:28Z

Closing as:
Fixed by #39

phillxnet · 2023-08-03T12:08:26Z

Follow-up:
With the above Postgresql DB Tune & DB Setup preperations, we have:

5.0.1-0 release

+ poetry run django-admin test
Creating test database for alias 'default'...
Creating test database for alias 'smart_manager'...
System check identified no issues (0 silenced).
.F......F.............E......E.........................F.....E.FF.....F.................................................F..EE.F....................F.......F.................................................................................................

5.0.2-0

+ poetry run django-admin test
Creating test database for alias 'default'...
Creating test database for alias 'smart_manager'...
System check identified no issues (0 silenced).
......................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 262 tests in 19.366s

OK
...
Wrote: /usr/src/packages/SRPMS/rockstor-5.0.2-0.src.rpm
Wrote: /usr/src/packages/RPMS/x86_64/rockstor-5.0.2-0.x86_64.rpm

Where the latter host OS, in addition to Minimal_VM pre-installs, only has our BuildRequires installed.
Tested using a recent OS image: openSUSE-Leap-15.5-Minimal-VM.x86_64-kvm-and-xen.qcow2
Downloaded via:
wget https://download.opensuse.org/distribution/leap/15.5/appliances/openSUSE-Leap-15.5-Minimal-VM.x86_64-kvm-and-xen.qcow2

phillxnet · 2023-08-03T12:26:47Z

Test install of rpm built in last comment. Followin 15.5 adapted instructions re repos here:
https://rockstor.com/docs/howtos/rpm_install.html

OBS home repo for our shellinabox

zypper --non-interactive addrepo --refresh -p105 https://download.opensuse.org/repositories/home:/rockstor/15.5/ home_rockstor

Requires

Our additional package requires in 15.5 are currently as follows:

zypper --no-gpg-checks install /usr/src/packages/RPMS/x86_64/rockstor-5.0.2-0.x86_64.rpm  

The following 146 NEW packages are going to be installed:
  adcli at avahi catatonit check-create-certificate containerd cron cronie cryptsetup cups-config cyrus-sasl cyrus-sasl-gssapi cyrus-sasl-plain deltarpm device-mapper dmraid dnf
  dnf-data dnf-plugins-core dnf-yum docker e2fsprogs ed fontconfig hdparm hwdata keyutils kpartx krb5-client libaio1 libarchive13 libavahi-client3 libavahi-common3 libavahi-core7
  libbasicobjects0 libcares2 libcollection4 libcomps0 libcups2 libdaemon0 libdevmapper-event1_03 libdhash1 libdnf2 libdns_sd libduktape206 libevent-2_1-8 libfontconfig1 libfreeipmi17
  libgd3 libHX28 libini_config5 libipmimonitoring6 libjbig2 libjpeg8 libldb2 liblmdb-0_9_30 libltdl7 libmodulemd2 libndp0 libneon27 libnscd1 libnss_nis2 libpath_utils1
  libpolkit-agent-1-0 libpolkit-gobject-1-0 libpowerman0 libpwquality1 libref_array1 librepo0 libsensors4 libsnmp40 libsss_certmap0 libsss_idmap0 libsss_nss_idmap0 libtalloc2 libtdb1
  libtevent0 libtiff5 libupsclient1 liburing1 libusb-0_1-4 libwebp7 libX11-6 libX11-data libXau6 libxcb1 libXpm4 mailx nano net-snmp NetworkManager NetworkManager-branding-openSUSE
  nfs-client nfsidmap nfs-kernel-server nginx nss-mdns ntp nut nut-drivers-net perl-SNMP perl-Term-ReadKey polkit postfix python3-curses python3-dnf python3-dnf-plugins-core python3-gpg
  python3-hawkey python3-libcomps python3-libdnf python3-libmodulemd python3-python-dateutil python3-sssd-config realmd realmd-lang rockstor rpcbind runc samba samba-client
  samba-client-libs samba-libs samba-winbind samba-winbind-libs shellinabox smartmontools snmp-mibs sssd sssd-ad sssd-common sssd-dbus sssd-krb5-common sssd-ldap sssd-tools sudo
  systemtap-runtime system-user-daemon system-user-lp system-user-upsd thin-provisioning-tools typelib-1_0-Modulemd-2_0 usbutils wpa_supplicant ypbind yp-tools

146 new packages to install.
Overall download size: 85.8 MiB. Already cached: 0 B. After the operation, additional 331.9 MiB will be used.

phillxnet self-assigned this Jul 28, 2023

phillxnet mentioned this issue Jul 31, 2023

Improve OS independence re unit tests rockstor/rockstor-core#2633

Closed

10 tasks

phillxnet mentioned this issue Aug 1, 2023

Improve OS independence re unit tests #2633 rockstor/rockstor-core#2637

Merged

phillxnet mentioned this issue Aug 1, 2023

Improve BuildRequires use #28 #39

Merged

phillxnet added a commit that referenced this issue Aug 2, 2023

Merge pull request #39 from phillxnet/28_Improve_BuildRequires_use

8e2d45d

Improve BuildRequires use #28

phillxnet closed this as completed Aug 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve BuildRequires use #28

Improve BuildRequires use #28

phillxnet commented Jul 23, 2023

phillxnet commented Jul 28, 2023 •

edited

Loading

phillxnet commented Jul 28, 2023 •

edited

Loading

phillxnet commented Jul 28, 2023

phillxnet commented Jul 28, 2023

phillxnet commented Jul 28, 2023

phillxnet commented Jul 28, 2023

phillxnet commented Jul 31, 2023

phillxnet commented Jul 31, 2023 •

edited

Loading

phillxnet commented Jul 31, 2023 •

edited

Loading

phillxnet commented Aug 2, 2023

phillxnet commented Aug 3, 2023

phillxnet commented Aug 3, 2023

Improve BuildRequires use #28

Improve BuildRequires use #28

Comments

phillxnet commented Jul 23, 2023

phillxnet commented Jul 28, 2023 • edited Loading

rpm-build dependencies (Leap 15.5 Minimal_VM image)

phillxnet commented Jul 28, 2023 • edited Loading

phillxnet commented Jul 28, 2023

phillxnet commented Jul 28, 2023

phillxnet commented Jul 28, 2023

phillxnet commented Jul 28, 2023

phillxnet commented Jul 31, 2023

HBA Host Based Authentication (Postgresql)

phillxnet commented Jul 31, 2023 • edited Loading

DB setup as per initrock (minimal for tests) - WILL DESTROY EXISTING DB.

DB tune:

DB setup:

phillxnet commented Jul 31, 2023 • edited Loading

phillxnet commented Aug 2, 2023

phillxnet commented Aug 3, 2023

5.0.1-0 release

5.0.2-0

phillxnet commented Aug 3, 2023

OBS home repo for our shellinabox

Requires

phillxnet commented Jul 28, 2023 •

edited

Loading

phillxnet commented Jul 28, 2023 •

edited

Loading

phillxnet commented Jul 31, 2023 •

edited

Loading

phillxnet commented Jul 31, 2023 •

edited

Loading