Improve release candidate container build cachability and reproducabi…

…lity

The release candidate container setup script makes reusing cached image
layers difficult because any change to the setup script (e.g. sbt/npm
version) invalidates the previous cached layers. This means anytime a
change is made to this script the entire container image must be rebuilt
which takes a lot of time and wasted space.

To avoid this, the container setup script is removed and all
functionality is moved to the Dockerfile. This results in more layers,
but makes it less likely to need to rebuild layers if there are changes.
Commands that are more likely to change due to versions or contents
(e.g. sbt/yarn install, release-candidate script) are put closer to the
end so if they are changed, less needs to be rebuilt.

More importantly, the installation of yarn and sbt do not provide a
specific version, so the version in the container depends on the latest
version available in the npm/sbt repositories when the container is
built. To improve reproducability, they are changed to install a
pinned version (the latest versions as of this change).

Additional minor updates to the container:
- Switch from fedora-minimal to fedora. The size difference is minimal
  compared to the final size of the container, and it replaces microdnf
  with dnf, which has more capabilities to make the Dockerfile cleaner,
  (e.g. quieter build, repo/repofrompath options).
- Add the "--group-add keep-groups" option to the podman run command
  needed for newer versions of podman.
- Some files only need to be verified when building Daffodil and not VS
  Code, only mention the correct files to avoid confusion.
- Update sbt-pgp plugin to latest version
- Update wix_wine.sh to remove debug message that SBT makes look like
  errors.

DAFFODIL-2713

containers/release-candidate/Dockerfile

@@ -15,20 +15,61 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM registry.fedoraproject.org/fedora-minimal:36
+FROM registry.fedoraproject.org/fedora:36
 
 WORKDIR /root
 
-COPY setup-container.sh /usr/bin/
-RUN /usr/bin/setup-container.sh
+# Delete cruft fedora leaves in the homedir
+RUN rm -rf /root/*
 
-# Needed to get WiX to run in wine on Linux. See wix_wine.sh for more details
-# on why we need to do this and how it works
+# Install dependencies, explicitly not enabling the updates repo so we are
+# pinned at a particular fedora release.
+RUN \
+  dnf -y --quiet --repo=fedora install \
+    clang \
+    git \
+    java-1.8.0-devel \
+    llvm \
+    mxml-devel \
+    npm \
+    pinentry \
+    rpm-build \
+    rpm-sign \
+    subversion \
+    unzip \
+    vim-minimal \
+    wine \
+    winetricks && \
+  dnf clean all
+
+# Enable SBT pgp plugin
+COPY src/plugins.sbt /root/.sbt/1.0/plugins/
+
+# Install wix, including changes to allow WiX to run in wine on Linux. See
+# src/wix_wine.sh for more details on why we need to do this and how it works
+RUN \
+  curl -sS -L https://github.com/wixtoolset/wix3/releases/download/wix3112rtm/wix311-binaries.zip -o wix311-binaries.zip && \
+  echo "6fd961c85e1e6adafb99ef50c9659e3eb83c84ecaf49f523e556788845b206e1857aba2c39409405d4cda1df9b30a552ce5aab808be5f8368a37a447d78d1a05 wix311-binaries.zip" | sha512sum --quiet -c - && \
+  mkdir /opt/wix311 && \
+  unzip -q wix311-binaries.zip -d /opt/wix311/ && \
+  rm wix311-binaries.zip
 RUN mv /opt/wix311/{candle.exe,real-candle.exe}
 RUN mv /opt/wix311/{light.exe,real-light.exe}
-COPY wix_wine.sh /opt/wix311/candle.exe
-COPY wix_wine.sh /opt/wix311/light.exe
+COPY src/wix_wine.sh /opt/wix311/candle.exe
+COPY src/wix_wine.sh /opt/wix311/light.exe
+
+# Install a pinned version of sbt, needed because the spt-rpm repository
+# frequently adds new versions.
+RUN \
+  dnf -y --quiet --repofrompath=sbt-rpm,https://repo.scala-sbt.org/scalasbt/rpm --repo=sbt-rpm --nogpgcheck install \
+    sbt-1.7.1 && \
+  dnf clean all
 
-COPY daffodil-release-candidate /usr/bin/
+# Install a pinned version of yarn, needed because the npm repository
+# frequently adds new versions.
+RUN \
+  npm --no-update-notifier install --global yarn@1.22.19
 
+# Install and set the entrypoint
+COPY src/daffodil-release-candidate /usr/bin/
 ENTRYPOINT ["/usr/bin/daffodil-release-candidate"]

containers/release-candidate/README.md

@@ -27,7 +27,7 @@ To build the Daffodil release candidate container image:
 
 To use the container image to build a release run the following:
 
-    podman run -it --privileged --rm \
+    podman run -it --privileged --group-add keep-groups --rm \
       -v ~/.gitconfig:/root/.gitconfig \
       -v ~/.gnupg/:/root/.gnupg/ \
       -v ~/.ssh/:/root/.ssh/ \

containers/release-candidate/daffodil-release-candidate → containers/release-candidate/src/daffodil-release-candidate

@@ -339,10 +339,13 @@ echo
 echo "Things to verify: "
 echo
 echo "- Files in $DAFFODIL_DIST_DIR"
-echo "- Files in $DAFFODIL_DOCS_DIR"
-echo "- Files in $DAFFODIL_TUTORIALS_DIR"
 echo "- Git tag created in $DAFFODIL_CODE_DIR for $PROJECT_REPO v$VERSION-$PRE_RELEASE"
-echo "- Staged published files at https://repository.apache.org/"
+if [ "$PROJECT_REPO" = "daffodil" ]
+then
+  echo "- Files in $DAFFODIL_DOCS_DIR"
+  echo "- Files in $DAFFODIL_TUTORIALS_DIR"
+  echo "- Staged published files at https://repository.apache.org/"
+fi
 echo
 
 if [ "$DRY_RUN" = true ]; then

containers/release-candidate/src/plugins.sbt

@@ -0,0 +1,16 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+addSbtPlugin("com.github.sbt" % "sbt-pgp" % "2.1.2")

containers/release-candidate/wix_wine.sh → containers/release-candidate/src/wix_wine.sh

@@ -27,6 +27,10 @@
 # failed.
 export WINEDEBUG=-all
 
+# Create initial wine config, redirecting stderr to stdout. The command outputs
+# debug message to stderr, which SBT makes look like an actual error.
+winecfg 2>&1
+
 # The sbt native-packager plugin executes the $WIX/{candle,light}.exe
 # executables to build the Daffodil MSI. The problem is that those are Windows
 # executables and so can't be directly executed in the Linux container. To get
@@ -59,9 +63,9 @@ do
 	((++i))
 done
 
-# Tell bash to output the command we are about to execute, helpful for
-# debugging when something goes wrong with wine
-set -x
+# Uncomment to tell bash to output the wine command we are about to execute,
+# helpful for debugging when something goes wrong with wine
+# set -x
 
 # Execute wine with the real WiX command and modified arguments
 wine $WIX/$REAL_CMD "${NEWARGS[@]}"