Update test container OS versions.

Update Ubuntu 12.04 to 16.04. Version 16.04 is recently EOL but testing on an old version is beneficial. Update Ubuntu 18.04 to 20.04. Update Fedora 32 to 33. Version 34 would have been preferred but there were some build issues, i.e. the default shell did not work with configure, and after ksh was installed configure locked up. Add --no-install-recommends to apt-get commands to save a bit of time and space. Update test Dockerfile to run in multiple steps. This makes the container larger but also makes rebuilding after changes faster. The --squash option may be used to keep the container small. Remove obsolete casts in protocol/parallel module. These casts were included in the original migration because Ubuntu 12.04 32-bit gcc required them, but Ubuntu 16.04 32-bit gcc complains. There is no production issue here since at this point in the code the file descriptors are guaranteed to be >= 0.
pgbackrest · Jul 18, 2021 · 6397d73 · 6397d73
1 parent 81602f1
commit 6397d73
Show file tree

Hide file tree

Showing 15 changed files with 119 additions and 230 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -29,16 +29,16 @@ jobs:
       matrix:
         include:
           # All unit (without coverage) and integration tests for 32-bit
-          - param: test --vm=u12 --param=no-performance
+          - param: test --vm=u16 --param=no-performance
 
           # Debian/Ubuntu documentation
           - param: doc --vm=u18
 
           # All integration tests
-          - param: test --vm=u18 --param=build-package --param=module=mock --param=module=real
+          - param: test --vm=u20 --param=build-package --param=module=mock --param=module=real
 
           # All unit tests (with coverage) on the newest gcc available
-          - param: test --vm=f32 --param=c-only --param=tz=America/New_York
+          - param: test --vm=f33 --param=c-only --param=tz=America/New_York
 
           # RHEL/CentOS 8 documentation
           - param: doc --vm=co8

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -14,7 +14,7 @@ Coding standards are defined in [CODING.md](https://github.com/pgbackrest/pgback
 
 ## Building a Development Environment
 
-This example is based on Ubuntu 18.04, but it should work on many versions of Debian and Ubuntu.
+This example is based on Ubuntu 20.04, but it should work on many versions of Debian and Ubuntu.
 
 pgbackrest-dev => Install development tools
 ```
@@ -315,38 +315,38 @@ pgbackrest/test/test.pl --vm=none --dev --module=postgres
 
 #### With Docker
 
-Build a container to run tests. The vm must be pre-configured but a variety are available. A vagrant file is provided in the test directory as an example of running in a virtual environment. The vm names are all three character abbreviations, e.g. `u18` for Ubuntu 18.04.
+Build a container to run tests. The vm must be pre-configured but a variety are available. A vagrant file is provided in the test directory as an example of running in a virtual environment. The vm names are all three character abbreviations, e.g. `u20` for Ubuntu 20.04.
 
 pgbackrest-dev => Build a VM
 ```
-pgbackrest/test/test.pl --vm-build --vm=u18
+pgbackrest/test/test.pl --vm-build --vm=u20
 
 --- output ---
 
     P00   INFO: test begin on x86_64 - log level info
-    P00   INFO: Using cached pgbackrest/test:u18-base-20210521A image (a92925d1200fd12d5f3d59f3a3db555c6efa00be) ...
-    P00   INFO: Building pgbackrest/test:u18-test image ...
+    P00   INFO: Using cached pgbackrest/test:u20-base-20210717A image (738bc9b702a18d1fcb5ba33dae74c74e01e67527) ...
+    P00   INFO: Building pgbackrest/test:u20-test image ...
     P00   INFO: Build Complete
 ```
 > **NOTE:** to build all the vms, just omit the `--vm` option above.
 
 pgbackrest-dev => Run a Specific Test Run
 ```
-pgbackrest/test/test.pl --vm=u18 --dev --module=mock --test=archive --run=2
+pgbackrest/test/test.pl --vm=u20 --dev --module=mock --test=archive --run=2
 
 --- output ---
 
     P00   INFO: test begin on x86_64 - log level info
     P00   INFO: check code autogenerate
     P00   INFO: cleanup old data and containers
     P00   INFO: builds required: bin, bin host
-    P00   INFO:     bin dependencies have changed for u18, rebuilding...
-    P00   INFO:     build bin for u18 (/home/vagrant/test/bin/u18)
+    P00   INFO:     bin dependencies have changed for u20, rebuilding...
+    P00   INFO:     build bin for u20 (/home/docker/test/bin/u20)
     P00   INFO:     bin dependencies have changed for none, rebuilding...
-    P00   INFO:     build bin for none (/home/vagrant/test/bin/none)
+    P00   INFO:     build bin for none (/home/docker/test/bin/none)
     P00   INFO: 1 test selected
                 
-    P00   INFO: P1-T1/1 - vm=u18, module=mock, test=archive, run=2
+    P00   INFO: P1-T1/1 - vm=u20, module=mock, test=archive, run=2
     P00   INFO: no code modules had all tests run required for coverage
     P00   INFO: TESTS COMPLETED SUCCESSFULLY
 ```
@@ -498,7 +498,7 @@ TEST_ERROR(cmdCheck(), ConfigError, "primary database not found\nHINT: check ind
 
 Unit tests are run for all files that are listed in `define.yaml` and a coverage report generated for each file listed under the tag `coverage:`. Note that some files are listed in multiple `coverage:` sections for a module; in this case, each test for the file being modified should be specified for the module in which the file exists (e.g. `--module=storage --test=posix --test=gcs`, etc.) or, alternatively, simply run the module without the `--test` option. It is recommended that a `--vm` be specified since running the same test for multiple vms is unnecessary for coverage. The following example would run the test set from the **define.yaml** section detailed above.
 ```
-pgbackrest/test/test.pl --vm-out --dev --module=command --test=check --vm=u18
+pgbackrest/test/test.pl --vm-out --dev --module=command --test=check --vm=u20
 ```
 > **NOTE:** Not all systems perform at the same speed, so if a test is timing out, try rerunning with another vm.
 

diff --git a/doc/release.pl b/doc/release.pl
@@ -225,7 +225,7 @@ =head1 SYNOPSIS
 
             # Generate coverage summary
             &log(INFO, "Generate Coverage Summary");
-            executeTest("${strTestExe} --vm=f32 --no-valgrind --clean --vm-max=2 --coverage-summary", {bShowOutputAsync => true});
+            executeTest("${strTestExe} --vm=f33 --no-valgrind --clean --vm-max=2 --coverage-summary", {bShowOutputAsync => true});
         }
 
         # Remove permanent cache file

diff --git a/doc/xml/contributing.xml b/doc/xml/contributing.xml
@@ -24,7 +24,7 @@
     </variable-list>
 
     <!-- Setup hosts used to build the documentation =========================================================================== -->
-    <host-define if="{[os-type-is-debian]}" image="{[host-contrib-image]}" from="ubuntu:18.04">
+    <host-define if="{[os-type-is-debian]}" image="{[host-contrib-image]}" from="ubuntu:20.04">
         {[copy-ca-cert]}
 
         # Fix root tty
@@ -65,9 +65,9 @@
     <section id="environment">
         <title>Building a Development Environment</title>
 
-        <p>This example is based on <proper>Ubuntu 18.04</proper>, but it should work on many versions of <proper>Debian</proper> and <proper>Ubuntu</proper>.</p>
+        <p>This example is based on <proper>Ubuntu 20.04</proper>, but it should work on many versions of <proper>Debian</proper> and <proper>Ubuntu</proper>.</p>
 
-        <host-add id="{[host-contrib-id]}" name="{[host-contrib]}" user="{[host-contrib-user]}" image="{[host-contrib-image]}" os="u18" option="-v /var/run/docker.sock:/var/run/docker.sock -v {[cwd]}/test:{[cwd]}/test" mount="/home/{[host-contrib-user]}/pgbackrest:/home/{[host-contrib-user]}/pgbackrest"/>
+        <host-add id="{[host-contrib-id]}" name="{[host-contrib]}" user="{[host-contrib-user]}" image="{[host-contrib-image]}" os="u20" option="-v /var/run/docker.sock:/var/run/docker.sock -v {[cwd]}/test:{[cwd]}/test" mount="/home/{[host-contrib-user]}/pgbackrest:/home/{[host-contrib-user]}/pgbackrest"/>
 
         <execute-list host="{[host-contrib]}">
             <title>Install development tools</title>
@@ -368,13 +368,13 @@ pgbackrest/doc/doc.pl --help
             <section id="with-docker" depend="/environment">
                 <title>With Docker</title>
 
-                <p>Build a container to run tests. The vm must be pre-configured but a variety are available. A vagrant file is provided in the test directory as an example of running in a virtual environment. The vm names are all three character abbreviations, e.g. <id>u18</id> for <proper>Ubuntu 18.04</proper>.</p>
+                <p>Build a container to run tests. The vm must be pre-configured but a variety are available. A vagrant file is provided in the test directory as an example of running in a virtual environment. The vm names are all three character abbreviations, e.g. <id>u20</id> for <proper>Ubuntu 20.04</proper>.</p>
 
                 <execute-list host="{[host-contrib]}">
                     <title>Build a VM</title>
 
                     <execute output="y">
-                        <exe-cmd>pgbackrest/test/test.pl --vm-build --vm=u18</exe-cmd>
+                        <exe-cmd>pgbackrest/test/test.pl --vm-build --vm=u20</exe-cmd>
                         <exe-cmd-extra>{[test-cmd-extra]}</exe-cmd-extra>
                     </execute>
                 </execute-list>
@@ -385,7 +385,7 @@ pgbackrest/doc/doc.pl --help
                     <title>Run a Specific Test Run</title>
 
                     <execute output="y">
-                        <exe-cmd>pgbackrest/test/test.pl {[dash]}-vm=u18 {[dash]}-dev {[dash]}-module=mock {[dash]}-test=archive {[dash]}-run=2</exe-cmd>
+                        <exe-cmd>pgbackrest/test/test.pl {[dash]}-vm=u20 {[dash]}-dev {[dash]}-module=mock {[dash]}-test=archive {[dash]}-run=2</exe-cmd>
                         <exe-cmd-extra>{[test-cmd-extra]}</exe-cmd-extra>
                     </execute>
                 </execute-list>
@@ -563,7 +563,7 @@ TEST_ERROR(cmdCheck(), ConfigError, "primary database not found\nHINT: check ind
             <p>Unit tests are run for all files that are listed in <file>define.yaml</file> and a coverage report generated for each file listed under the tag <code>coverage:</code>. Note that some files are listed in multiple <code>coverage:</code> sections for a module; in this case, each test for the file being modified should be specified for the module in which the file exists (e.g. <code>--module=storage --test=posix --test=gcs</code>, etc.) or, alternatively, simply run the module without the <code>--test</code> option. It is recommended that a <code>--vm</code> be specified since running the same test for multiple vms is unnecessary for coverage. The following example would run the test set from the <b>define.yaml</b> section detailed above.</p>
 
             <code-block>
-pgbackrest/test/test.pl --vm-out --dev --module=command --test=check --vm=u18
+pgbackrest/test/test.pl --vm-out --dev --module=command --test=check --vm=u20
             </code-block>
 
 <admonition type="note">Not all systems perform at the same speed, so if a test is timing out, try rerunning with another vm.</admonition>

diff --git a/doc/xml/user-guide.xml b/doc/xml/user-guide.xml
@@ -302,7 +302,8 @@
         RUN export DEBIAN_FRONTEND=noninteractive &amp;&amp; \
             rm /etc/apt/apt.conf.d/70debconf &amp;&amp; \
             apt-get update &amp;&amp; \
-            apt-get install -y sudo ssh wget vim gnupg lsb-release iputils-ping ca-certificates tzdata locales 2>&amp;1
+            apt-get install -y --no-install-recommends sudo ssh wget vim gnupg lsb-release iputils-ping ca-certificates \
+                    tzdata locales 2>&amp;1
 
         {[sudo-disable-core-dump]}
 
@@ -315,10 +316,10 @@
                 tee -a /etc/apt/sources.list.d/pgdg.list &amp;&amp; \
             wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - &amp;&amp; \
             apt-get update &amp;&amp; \
-            apt-get install -y postgresql-common 2>&amp;1 &amp;&amp; \
+            apt-get install -y --no-install-recommends postgresql-common 2>&amp;1 &amp;&amp; \
             sed -i 's/^\#create\_main\_cluster.*$/create\_main\_cluster \= false/' \
                 /etc/postgresql-common/createcluster.conf &amp;&amp; \
-            apt-get install -y postgresql-{[pg-version]} postgresql-{[pg-version-upgrade]} 2>&amp;1
+            apt-get install -y --no-install-recommends postgresql-{[pg-version]} postgresql-{[pg-version-upgrade]} 2>&amp;1
 
         # Create an ssh key for root so all hosts can ssh to each other as root
         RUN \ {[ssh-key-install]}

diff --git a/src/protocol/parallel.c b/src/protocol/parallel.c
@@ -128,7 +128,7 @@ protocolParallelProcess(ProtocolParallel *this)
         if (this->clientJobList[clientIdx] != NULL)
         {
             int fd = protocolClientIoReadFd(*(ProtocolClient **)lstGet(this->clientList, clientIdx));
-            FD_SET((unsigned int)fd, &selectSet);
+            FD_SET(fd, &selectSet);
 
             // Find the max file descriptor needed for select()
             MAX_ASSIGN(fdMax, fd);
@@ -158,7 +158,7 @@ protocolParallelProcess(ProtocolParallel *this)
 
                 if (job != NULL &&
                     FD_ISSET(
-                        (unsigned int)protocolClientIoReadFd(*(ProtocolClient **)lstGet(this->clientList, clientIdx)),
+                        protocolClientIoReadFd(*(ProtocolClient **)lstGet(this->clientList, clientIdx)),
                         &selectSet))
                 {
                     MEM_CONTEXT_TEMP_BEGIN()

diff --git a/test/Dockerfile b/test/Dockerfile
@@ -1,38 +1,34 @@
-# Force remove all containers:
-# docker rm -f $(docker ps -a -q)
-# Force remove all images:
-# docker rmi -f $(docker images -q)
-#
 # export PGB=pgbackrest;export PGB_IMAGE=$PGB/test:u20-main;PGB_REPO=~/Documents/Code/$PGB
-# docker build -f $PGB_REPO/test/Dockerfile -t $PGB_IMAGE .
+# docker build --squash -f $PGB_REPO/test/Dockerfile -t $PGB_IMAGE .
 # docker run --privileged -itd --name $PGB-test -h $PGB-test -v $PGB_REPO:/home/docker/$PGB $PGB_IMAGE
-
 FROM ubuntu:focal
 
-# Suppress "dpkg-reconfigure: unable to re-open stdin: No file or directory" warning
-RUN export DEBCONF_NONINTERACTIVE_SEEN=true DEBIAN_FRONTEND=noninteractive && \
-
 # Install packages
-    apt-get update && \
-    apt-get install -y \
-        sudo vim htop jq rsync sysstat \
+RUN apt-get update
+RUN DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \
+        sudo vim htop jq rsync sysstat curl \
         libdbd-pg-perl libxml-checker-perl libyaml-perl \
         devscripts build-essential lintian git cloc txt2man debhelper libssl-dev zlib1g-dev libperl-dev libxml2-dev liblz4-dev \
-        liblz4-tool libpq-dev valgrind lcov autoconf-archive zstd libzstd-dev bzip2 libbz2-dev pkg-config && \
+        liblz4-tool libpq-dev valgrind lcov autoconf-archive zstd libzstd-dev bzip2 libbz2-dev pkg-config libyaml-dev
 
 # Install Docker
-    groupadd -g5000 docker && \
-    curl -fsSL https://get.docker.com | sh && \
+RUN groupadd -g5000 docker
+RUN curl -fsSL https://get.docker.com | sh
 
 # Create docker user
-    adduser --ingroup=docker -uid=5000 --disabled-password --gecos \"\" docker && \
+RUN adduser --ingroup=docker -uid=5000 --disabled-password --gecos \"\" docker
 
 # Configure sudo
-    echo '%docker ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers && \
+RUN echo '%docker ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
 
 # Mount tmpfs at /home/docker/test for faster testing
-    sudo -u docker mkdir -m 770 /home/docker/test && \
-    echo 'tmpfs /home/docker/test tmpfs size=4096M 0 1' >> /etc/fstab
+RUN sudo -u docker mkdir -m 770 /home/docker/test
+RUN echo 'tmpfs /home/docker/test tmpfs size=4096M 0 1' >> /etc/fstab
+
+# Cleanup
+RUN apt-get autoremove -y
+RUN apt-get clean
+RUN rm -rf /var/lib/apt/lists/*
 
 USER docker
 WORKDIR /home/docker

diff --git a/test/Vagrantfile b/test/Vagrantfile
@@ -76,7 +76,7 @@ Vagrant.configure(2) do |config|
             echo 'Install Build Tools' && date
             apt-get install -y devscripts build-essential lintian git cloc txt2man debhelper libssl-dev zlib1g-dev libperl-dev \
                  libxml2-dev liblz4-dev liblz4-tool libpq-dev valgrind lcov autoconf-archive zstd libzstd-dev bzip2 libbz2-dev \
-                 pkg-config
+                 pkg-config libyaml-dev
 
             #-----------------------------------------------------------------------------------------------------------------------
             echo 'Install Docker' && date

diff --git a/test/container.yaml b/test/container.yaml
@@ -12,23 +12,14 @@
 # - docker login -u pgbackrest
 # - VM=XXX;DATE=YYYYMMDDX;BASE=pgbackrest/test:${VM?}-base;docker tag ${BASE?} ${BASE?}-${DATE?} && docker push ${BASE?}-${DATE?}
 # **********************************************************************************************************************************
-20210521A:
-  x86_64:
-    u18: a92925d1200fd12d5f3d59f3a3db555c6efa00be
-
-20210503A:
+20210718A:
   aarch64:
-    u20: 00a309971d5566ff3eb0db92c66b4f4ef0566d36
-    co7: 41b9ffa64e5914753708eb3cdde3892010683fd7
-
-20210113A:
-  x86_64:
-    co7: 2c1e13990f92635cc7d959dcbe497b765861c2b2
-
-20200507A:
-  x86_64:
-    f32: 61792779061d2a675509c65bfa64b61b8d4cea17
+    u20: d6c377617ac2c112b80a3c1f090345d54769ae1b
+    co7: 8726f3be088e4ba34eb45e32c1994a32ca0e5c1d
 
-20200505A:
+20210717A:
   x86_64:
-    u12: 8a88ab44aace049d7da5ca1094375ff8b9aeb7ab
+    co7: 03a73386264355a5d1750ccd13371db97d92ab28
+    f33: 80326dab5c7d0137062b8d181ca07f9b14142de7
+    u16: 599d0d35a6e29ad3258c76ba4f9ea622d47fa8c0
+    u20: d6c377617ac2c112b80a3c1f090345d54769ae1b