Skip to content

Packaging

Jump to bottom Edit New page
XiaowenHu edited this page Aug 19, 2021 · 10 revisions

Packaging Souffle

This page serves as a reference point for the maintainers. If you just want to install Souffle, check https://souffle-lang.github.io/build.

How it works

On each release, a GitHub workflow is triggered and:

  1. Generates a package for each platform and architecture combination.
  2. Uploads the generated package to PackageCloud.

Explanation of GitHub Workflow

The GiHub workflow that is triggered on each release is located in .github/workflows/create-packages.yml.

A job has been created for each OS and architecture combination that is currently supported. For the purposes of this demonstration, we will use Ubuntu 21.04 64bit as an example.

Below is the job configuration for Ubuntu 21.04 64bit; however, the structure of the configuration for all the other operating systems is identical.

jobs:
  Package-Ubuntu-2104-64bit:
    strategy:
      fail-fast: false
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - name: Package souffle
        uses: ./.github/actions/create-package/ubuntu/21.04/64bit/
        with:
          package_cloud_api_key: ${{ secrets.PACKAGECLOUD_TOKEN }}

The following is a brief explanation of the configuration:

  • All GitHub jobs run on ubuntu-latest job-runners. However, this is not a requirement since the build and packaging process occurs entirely within a Docker container. Therefore, this configuration option can be changed to any operating system that supports and runs Docker. At the time of writing, GitHub Actions can be run on the following platforms, which all support Docker:
    • Ubuntu 20.04 (our selection)
    • Ubuntu 18.04
    • Windows Server 2016 and 2019
    • MacOS Big Sur and Catalina
  • The first step of our job is to clone the repository via the actions/checkout@v2 action, with a fetch-depth of 0. A fetch-depth of 0 means that we are fetching all history for all branches and tags. This configuration option is necessary to obtain the release version information of our GitHub project, which is used in the later steps by CMake to automatically detect and configure the release version of the generated package.
  • The second step of our job is to run a private GitHub action located in ./.github/actions/create-package/ubuntu/21.04/64bit/. Each combination of operating system and architecture should have a corresponding private GitHub action located in their own directory. Note that we pass a required package_cloud_api_key argument to our private GitHub action. This argument corresponds to the PACKAGECLOUD_TOKEN secret that needs to be configured via the GitHub for the Souffle repository.
    • This can be done in Souffle's GitHub repo by going to Settings → Secrets → New repository secret and then creating a new PACKAGECLOUD_TOKEN variable that corresponds to the PackageCloud API token that will be used to upload the packages.

Explanation of GitHub Private Action

The YAML configuration file for the private GitHub action that is located in ./.github/actions/create-package/ubuntu/21.04/64bit/ for Ubuntu 21.04 64bit is shown below:

name: Package for Ubuntu 21.04 64bit
description: Package for Ubuntu 21.04 64bit
inputs:
  package_cloud_api_key:
    description: 'Package Cloud API Key'
    required: true
    default: ''
runs:
  using: 'docker'
  image: 'Dockerfile'
  args:
    - ${{ inputs.package_cloud_api_key }}

The following is a brief explanation of the private GitHub action:

  • The action specifies that a variable called package_cloud_api_key must be provided as an input. The default value is just an empty string.
  • The action simply runs a Dockerfile located in the same directory as this configuration file and passes the package_cloud_api_key variable as an argument to the Dockerfile. Note that each combination of operating system and architecture should have its own corresponding Dockerfile.

Explanation of Dockerfile

The Dockerfile that the above private GitHub action runs is shown below:

FROM ubuntu:21.04

ARG DEBIAN_FRONTEND=noninteractive

# Create a souffle directory
WORKDIR /souffle

# Install souffle build dependencies
RUN apt-get update && \
	apt-get -y install \
	bash-completion \
	sudo \
	autoconf \
	automake \
	bison \
	build-essential \
	clang \
	doxygen \
	flex \
	g++ \
	git \
	libffi-dev \
	libncurses5-dev \
	libtool \
	libsqlite3-dev \
	make \
	mcpp \
	python \
	sqlite \
	zlib1g-dev \
	cmake

# For CMakeLists.txt to figure out the specific version of Ubuntu
RUN apt-get -y install lsb-release

# Install dependencies for packagecloud CLI
RUN apt-get -y install ruby-full
RUN sudo gem install package_cloud

# Copy everything into souffle directory
COPY . .

ENV DOMAIN_SIZE "64bit"
ENV PKG_EXTENSION ".deb"
ENV PKG_CLOUD_OS_NAME "ubuntu/hirsute"

ENTRYPOINT [".github/actions/create-package/entrypoint.sh"]

We first specify the operating system and architecture we are building the package for. In our case, this is specified using the following for Ubuntu 21.04 64bit.

FROM ubuntu:21.04

The following line simply allows us to run apt-get commands non-interactively.

ARG DEBIAN_FRONTEND=noninteractive

We then create a working directory called souffle in the root directory.

WORKDIR /souffle

We then install all the build dependencies of Souffle. Note that if there are any new dependencies that are introduced in the future, this part must be updated for every Dockerfile.

# Install souffle build dependencies
RUN apt-get update && \
	apt-get -y install \
	bash-completion \
	sudo \
	autoconf \
	automake \
	bison \
	build-essential \
	clang \
	doxygen \
	flex \
	g++ \
	git \
	libffi-dev \
	libncurses5-dev \
	libtool \
	libsqlite3-dev \
	make \
	mcpp \
	python \
	sqlite \
	zlib1g-dev \
	cmake

# For CMakeLists.txt to figure out the specific version of Ubuntu
RUN apt-get -y install lsb-release

I opted for the above method of installing dependencies instead of using a bash script like the one below (which is located in ./sh/setup/install_ubuntu_deps.sh) as I wanted to leverage Docker's build cache system during development of the Dockerfile.

#!/bin/sh

apt-get update -q
apt-get install -y -q autoconf automake bash-completion bison build-essential clang debhelper default-jdk-headless devscripts doxygen fakeroot flex g++ gdb git graphviz libffi-dev libncurses5-dev libsqlite3-dev libtool make mcpp pkg-config python3-dev sqlite swig zlib1g-dev cmake

We then had to install the dependencies for the package_cloud CLI tool as well as the CLI tool itself.

RUN apt-get -y install ruby-full
RUN sudo gem install package_cloud

We then copy the contents of our project into the /souffle directory of the Docker container:

COPY . .

Finally, we specify the environment variables that will be used within the following [entrypoint.sh](http://entrypoint.sh) script.

  • DOMAIN_SIZE should correspond to the architecture of the operating system. Accepted values are "64bit" or "32bit".
  • PKG_EXTENSION should correspond to the extension of the package that will be generated via CMake. Examples of values include ".deb" or ".rpm".
  • PKG_CLOUD_OS_NAME should correspond to the name of the operating system we wish to upload the package for. This <OS name>/<Version name> combination must be supported by PackageCloud.
ENV DOMAIN_SIZE "64bit"
ENV PKG_EXTENSION ".deb"
ENV PKG_CLOUD_OS_NAME "ubuntu/hirsute"

ENTRYPOINT [".github/actions/create-package/entrypoint.sh"]

Explanation of entrypoint.sh

The entrypoint.sh script that is run by the Dockerfile is shown below and is common to all Dockerfiles. The environment variables such as DOMAIN_SIZE, PKG_EXTENSION and PKG_CLOUD_OS_NAME specified in the Dockerfile above are all used in script as shown below.

#!/bin/sh

PACKAGE_CLOUD_API_KEY="$1"

# Run the build command
case "$DOMAIN_SIZE" in
  	"64bit")
		cmake -S . -B ./build -DSOUFFLE_DOMAIN_64BIT=ON
    ;;

  	"32bit")
		cmake -S . -B ./build
    ;;
esac

# Create the package
cmake --build ./build --parallel "$(nproc)" --target package

cd build

# Upload the package to packagecloud.io
PACKAGECLOUD_TOKEN="$PACKAGE_CLOUD_API_KEY" package_cloud push souffle-lang/souffle-test/$PKG_CLOUD_OS_NAME "$(ls *$PKG_EXTENSION | head -n1)"

We first save the package_cloud_api_key input that was used as an argument to our Dockerfile as a bash variable as shown below:

PACKAGE_CLOUD_API_KEY="$1"

We then run the cmake build command that varies based on the architecture of our operating system.

case "$DOMAIN_SIZE" in
  	"64bit")
		cmake -S . -B ./build -DSOUFFLE_DOMAIN_64BIT=ON
    ;;

  	"32bit")
		cmake -S . -B ./build
    ;;
esac

We then create our package by running the following command.

cmake --build ./build --parallel "$(nproc)" --target package

Note that the above command is functionally equivalent to running the cpack command within the ./build directory. The reason we use the above variant is for performance reasons.

When generating packages using the cpack command, it automatically rebuilds the entire project. This behaviour only affects users that use Unix Makefiles as their CPACK_CMAKE_GENERATOR and a deeper explanation about why this is the case can be found here. Unfortunately, since the cpack command does not accept any command line arguments that specify a parallel build level, building the Souffle project in single-threaded mode takes a very long time. Our alternative (specified above) allows the build phase to run on as many cores as possible using the nproc command.

Finally, we navigate to our ./build directory and upload the newly created package to PackageCloud.

cd build

PACKAGECLOUD_TOKEN="$PACKAGE_CLOUD_API_KEY" package_cloud push souffle-lang/souffle-test/$PKG_CLOUD_OS_NAME "$(ls *$PKG_EXTENSION | head -n1)"

PackageCloud has a concept called "repos", which is simply a repository or directory of related packages. At this stage in time, all packages are uploaded to the souffle-test repo (as shown in the above command). This repo can be explored at this link. However, at a future date when we are ready to distribute packages via PackageCloud, the repo should be changed to souffle, which can be found here.

Explanation of changes made to CMakeLists.txt

The changes we made to CMakeLists.txt to enable packaging are shown below:

# --------------------------------------------------
# Utility function to help us distinguish between Linux distros when packaging
# --------------------------------------------------

function(get_linux_lsb_release_information)
    find_program(LSB_RELEASE_EXEC lsb_release)
    if(NOT LSB_RELEASE_EXEC)
        message(FATAL_ERROR "Could not detect lsb_release executable, can not gather required information")
    endif()

    execute_process(COMMAND "${LSB_RELEASE_EXEC}" --short --id OUTPUT_VARIABLE LSB_RELEASE_ID_SHORT OUTPUT_STRIP_TRAILING_WHITESPACE)
    execute_process(COMMAND "${LSB_RELEASE_EXEC}" --short --release OUTPUT_VARIABLE LSB_RELEASE_VERSION_SHORT OUTPUT_STRIP_TRAILING_WHITESPACE)
    execute_process(COMMAND "${LSB_RELEASE_EXEC}" --short --codename OUTPUT_VARIABLE LSB_RELEASE_CODENAME_SHORT OUTPUT_STRIP_TRAILING_WHITESPACE)

    set(LSB_RELEASE_ID_SHORT "${LSB_RELEASE_ID_SHORT}" PARENT_SCOPE)
    set(LSB_RELEASE_VERSION_SHORT "${LSB_RELEASE_VERSION_SHORT}" PARENT_SCOPE)
    set(LSB_RELEASE_CODENAME_SHORT "${LSB_RELEASE_CODENAME_SHORT}" PARENT_SCOPE)
endfunction()

# --------------------------------------------------
# CPack configuration
# --------------------------------------------------

SET(CPACK_PACKAGE_CONTACT "Patrick H.")
SET(CPACK_PACKAGE_DESCRIPTION "Souffle - A Datalog Compiler")
SET(CPACK_PACKAGE_DESCRIPTION_SUMMARY "A Datalog Compiler")

# Use all available threads (primarily for compression of files)
SET(CPACK_THREADS 0)

# Make sure changelog, bash-completion and other important files in debian directory also packaged
SET(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${CMAKE_SOURCE_DIR}/debian/changelog.in" "${CMAKE_SOURCE_DIR}/debian/souffle.bash-completion" "${CMAKE_SOURCE_DIR}/debian/copyright")

# --------------------------------------------------
# CPack configuration for Linux
# --------------------------------------------------
if (CMAKE_SYSTEM_NAME MATCHES "Linux")
	get_linux_lsb_release_information()
  if (LSB_RELEASE_ID_SHORT MATCHES "Ubuntu")
		# Generate just DEB
		SET(CPACK_GENERATOR "DEB")
		# --------------------------------------------------
		# Variables relevent to DEB packages
		# --------------------------------------------------

		# Specifying runtime dependencies
		set(CPACK_DEBIAN_PACKAGE_DEPENDS "g++ (>= 7), libffi-dev, libncurses5-dev, libsqlite3-dev, mcpp, zlib1g-dev")

		# Auto-generate any runtime dependencies that are required
		SET(CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON)

		# Architectures are actually auto-detected so no need to set this variable
		# SET(CPACK_DEBIAN_PACKAGE_ARCHITECTURE "i386")
	endif()

  if (LSB_RELEASE_ID_SHORT MATCHES "Fedora")
		# Generate both DEB and RPM packages
		SET(CPACK_GENERATOR "RPM")

		# --------------------------------------------------
		# Variables relevent to RPM packages
		# --------------------------------------------------

		# Specifying runtime dependencies
		set(CPACK_RPM_PACKAGE_REQUIRES "g++ >= 7, libffi, libffi-devel, ncurses-devel, libsqlite3x, mcpp, zlib-devel")

		# Don't auto-detect dependencies and provides
		SET(CPACK_RPM_PACKAGE_AUTOREQPROV "no")
	endif()
endif()

INCLUDE(CPack)

The following is a utility function that depends on the lsb-release package to be installed. Its purpose is to distinguish between the various flavours of Linux and save this information into variables that can be used later on. The main variable to be used is LSB_RELEASE_ID_SHORT.

function(get_linux_lsb_release_information)
    find_program(LSB_RELEASE_EXEC lsb_release)
    if(NOT LSB_RELEASE_EXEC)
        message(FATAL_ERROR "Could not detect lsb_release executable, can not gather required information")
    endif()

    execute_process(COMMAND "${LSB_RELEASE_EXEC}" --short --id OUTPUT_VARIABLE LSB_RELEASE_ID_SHORT OUTPUT_STRIP_TRAILING_WHITESPACE)
    execute_process(COMMAND "${LSB_RELEASE_EXEC}" --short --release OUTPUT_VARIABLE LSB_RELEASE_VERSION_SHORT OUTPUT_STRIP_TRAILING_WHITESPACE)
    execute_process(COMMAND "${LSB_RELEASE_EXEC}" --short --codename OUTPUT_VARIABLE LSB_RELEASE_CODENAME_SHORT OUTPUT_STRIP_TRAILING_WHITESPACE)

    set(LSB_RELEASE_ID_SHORT "${LSB_RELEASE_ID_SHORT}" PARENT_SCOPE)
    set(LSB_RELEASE_VERSION_SHORT "${LSB_RELEASE_VERSION_SHORT}" PARENT_SCOPE)
    set(LSB_RELEASE_CODENAME_SHORT "${LSB_RELEASE_CODENAME_SHORT}" PARENT_SCOPE)
endfunction()

We then set some basic configuration variables for our package as shown:

SET(CPACK_PACKAGE_CONTACT "Patrick H.")
SET(CPACK_PACKAGE_DESCRIPTION "Souffle - A Datalog Compiler")
SET(CPACK_PACKAGE_DESCRIPTION_SUMMARY "A Datalog Compiler")

We then set the following variable in order to instruct cpack to use all available threads when performing parallelised operations such as compression of files.

SET(CPACK_THREADS 0)

The following line ensures that we copy the [changelog.in](http://changelog.in) file, the souffle.bash-completion file and the copyright file into the final package. Note that copying the souffle.bash-completion file into the final package is not a requirement for bash completion to work (which will be explained in a later section).

SET(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${CMAKE_SOURCE_DIR}/debian/changelog.in" "${CMAKE_SOURCE_DIR}/debian/souffle.bash-completion" "${CMAKE_SOURCE_DIR}/debian/copyright")

We then set configuration variables for packaging depending on the platform cmake is currently running on. For example, if the system is Linux-based, we first run our utility function via get_linux_lsb_release_information(). If the Linux operating system is Ubuntu, we set the generator to "DEB" since we want to generate Debian packages. We also specify Souffle's run-time dependencies via the CPACK_DEBIAN_PACKAGE_DEPENDS variable.

A similar process is repeated if the Linux operating system is Fedora. However, notice that the dependencies for Fedora are named differently than Ubuntu. For example, the equivalent dependency for libncurses5-dev in Ubuntu is called ncurses-devel in Fedora. The process of discovering equivalent dependencies requires research and experimentation.

To package for other flavours of Linux, create a new if branch and repeat the demonstrated steps above.

if (CMAKE_SYSTEM_NAME MATCHES "Linux")
	get_linux_lsb_release_information()
  if (LSB_RELEASE_ID_SHORT MATCHES "Ubuntu")
		SET(CPACK_GENERATOR "DEB")
		set(CPACK_DEBIAN_PACKAGE_DEPENDS "g++ (>= 7), libffi-dev, libncurses5-dev, libsqlite3-dev, mcpp, zlib1g-dev")
		SET(CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON)
	endif()

  if (LSB_RELEASE_ID_SHORT MATCHES "Fedora")
		SET(CPACK_GENERATOR "RPM")
		set(CPACK_RPM_PACKAGE_REQUIRES "g++ >= 7, libffi, libffi-devel, ncurses-devel, libsqlite3x, mcpp, zlib-devel")
		SET(CPACK_RPM_PACKAGE_AUTOREQPROV "no")
	endif()
endif()

Finally, we add the following line so that cmake can generate packages.

INCLUDE(CPack)

Bash completion

Once the user installs the package, the relevant files to allow bash-completion to work are installed on the user's directory as specified by the following cmake instructions.

find_package (bash-completion)
if (BASH_COMPLETION_FOUND)
    message(STATUS "Using bash completion dir ${BASH_COMPLETION_COMPLETIONSDIR}")
else()
    set (BASH_COMPLETION_COMPLETIONSDIR "/etc/bash_completion.d")
    message (STATUS "Using fallback bash completion dir ${BASH_COMPLETION_COMPLETIONSDIR}")
endif()

install(
    FILES "${CMAKE_SOURCE_DIR}/debian/souffle.bash-completion"
    DESTINATION ${BASH_COMPLETION_COMPLETIONSDIR}
    RENAME "souffle"
)

Note the above install command copies our souffle.bash-completion file to the relevant directory specified by the bash-completion package via the BASH_COMPLETION_COMPLETIONSDIR configuration variable. For Ubuntu, the BASH_COMPLETION_COMPLETIONSDIR variable corresponds to /usr/share/bash-completion/completions directory, which is where bash-completion files are located. We also must rename our souffle.bash-completion file to souffle as this is requirement for the the bash-completion package as outlined in their documentation here:

The completion filename for command foo in this directory should be either foo, or foo.bash.

How to create packages for a new OS and architecture combination

Suppose that we would like to create packages for Oracle Linux, we would need to make the following changes to our CMakeLists.txt file in the root directory.

Changes to make to CMakeLists.txt file

Create a new if branch specifically for OracleLinux as shown below.

if (CMAKE_SYSTEM_NAME MATCHES "Linux")

	...
	# Creating a new if branch for Oracle Linux
  if (LSB_RELEASE_ID_SHORT MATCHES "OracleServer")
		SET(CPACK_GENERATOR "RPM")
		set(CPACK_RPM_PACKAGE_REQUIRES "g++ >= 7 ...")
		SET(CPACK_RPM_PACKAGE_AUTOREQPROV "...")
	endif()
endif()

Note that we must first determine what the LSB_RELEASE_ID_SHORT variable corresponds to for Oracle Linux.

This can be confirmed by running the following shell command within an Oracle Linux environment (such as an Oracle Linux Docker container).

For the purposes of this example, I ran the following command to spin up a docker container running Oracle Linux:

docker container run -it oraclelinux:8 /bin/bash

From within the above docker container, I installed redhat-lsb using the below command.

yum -y install redhat-lsb

This allows me to run the following lsb_release command to figure out what value corresponds to the LSB_RELEASE_ID_SHORT variable for Oracle Linux:

lsb_release --short --id OUTPUT_VARIABLE LSB_RELEASE_ID_SHORT OUTPUT_STRIP_TRAILING_WHITESPACE

# Belw is the output of the above command in an Oracle Linux Docker container
OracleServer

As such, we can use this value as a condition within our if statement as shown below:

if (LSB_RELEASE_ID_SHORT MATCHES "OracleServer")
	...
endif()

This if condition will ensure that when cmake is called from within any version and architecture (32bit, 64bit, etc.) of Oracle Linux, the configuration variables nested within the above if statement are used.

Therefore, within the above if statement, we set the configuration variables that are specific to Oracle Linux.

Below is an example of what this might look like for Oracle Linux:

if (LSB_RELEASE_ID_SHORT MATCHES "OracleServer")
	# Specify the type of package to generate for Oracle Linux
	SET(CPACK_GENERATOR "RPM")
	# Specify the dependencies and versions we wish to require for Souffle in Oracle Linux
	set(CPACK_RPM_PACKAGE_REQUIRES "g++ >= 7 ...")
	# Specify whether you wish Cpack to automatically detect dependencies
	SET(CPACK_RPM_PACKAGE_AUTOREQ "...")
	# Specify whether you wish Cpack to automatically list shared libraries that are provided by Souffle
	SET(CPACK_RPM_PACKAGE_AUTOPROV "...")
	# You can also set both of the above variables in one go with the following
	SET(CPACK_RPM_PACKAGE_AUTOREQPROV "...")
endif()

More information about CPACK_RPM_PACKAGE_AUTOREQ, CPACK_RPM_PACKAGE_AUTOPROV and CPACK_RPM_PACKAGE_AUTOREQPROV can be found in the cmake documentation here, here and here respectively.

Note that we do not need to specify the architecture within our if statement since by default, cmake will autodetect the architecture and set the value of CPACK_RPM_PACKAGE_ARCHITECTURE to the appropriate value as explained in the documentation.

Changes to make to create-packages.yml GitHub workflow

After we have edited our CMakeLists.txt file so that CPack knows how to generate packages for Oracle Linux, we must then create a Github job within .github/workflows/create-packages.yml for each version and architecture we wish to create and upload packages for.

For example, if we would like to create and upload packages for the 64bit version of Oracle Linux version 8, we would create the following job in our create-packages.yml file.

...
Package-Oracle-Linux-8-64bit:
  strategy:
    fail-fast: false

  runs-on: ubuntu-latest
  steps:
    - name: Checkout
      uses: actions/checkout@v2
      with:
        fetch-depth: 0
    - name: Package souffle
      uses: ./.github/actions/create-package/oracle-linux/8/64bit/
      with:
        package_cloud_api_key: ${{ secrets.PACKAGECLOUD_TOKEN }}

Notice that this GitHub job is calling a private GitHub Action that is located in the following directory:

./.github/actions/create-package/oracle-linux/8/64bit/

Therefore, after we have specified the above GitHub job, we must also make sure the above directory structure exists in our project.

Within the .github/actions/create-package/oracle-linux/8/64bit/ directory, we must create an action.yml and a Dockerfile as shown below:

.github/actions/create-package/oracle-linux/
└── 8
    └── 64bit
        ├── action.yml
        └── Dockerfile

Creation of an action.yml file

Below is what a private GitHub action for Oracle Linux version 8 64bit would look like.

name: Package for Oracle Linux 8 64bit
description: Package for Oracle Linux 8 64bit
inputs:
  package_cloud_api_key:
    description: 'Package Cloud API Key'
    required: true
    default: ''
runs:
  using: 'docker'
  image: 'Dockerfile'
  args:
    - ${{ inputs.package_cloud_api_key }}

As we can see, the above private GitHub action file is identical across every operating system and architecture combination except we change the name and description fields:

name: Package for Oracle Linux 8 64bit
description: Package for Oracle Linux 8 64bit
...
# Everything else is identical across all operating systems

Creation of a Dockerfile

We must then create a Dockerfile that creates a 64bit Oracle Linux version 8 environment for cmake to create packages within:

FROM oraclelinux:8

WORKDIR /souffle

# Specify all the build-time dependencies for Oracle Linux to build Souffle
RUN yum -y install \
    bash-completion \
    ...

# These build-time dependencies are specific to the OS you are building within. We need redhat-lsb in order for Cmake to determinie the specific version of Linux it is running on via the lsb_release command. rpm-build is also required for CMake to generate RPM packages. rpm-build is not necessary for Debian OSs.
RUN yum -y install redhat-lsb rpm-build

# Install ruby as it is a dependency for the package_cloud CLI tool
RUN yum -y install ruby ruby-devel

# Install package cloud CLI tool
RUN sudo gem install package_cloud

# Copy everything into souffle directory of container
COPY . .

ENV DOMAIN_SIZE "64bit"
ENV PKG_EXTENSION ".rpm"
ENV PKG_CLOUD_OS_NAME "oraclelinux/8"

ENTRYPOINT [".github/actions/create-package/entrypoint.sh"]

Installing ruby as a dependency

Depending on the operating system, sometimes installing ruby as a dependency is insufficient for the package_cloud CLI tool to operate.

For instance on Fedora, installing ruby alone yields the following error when I try to install package_cloud:

...
/usr/bin/ruby -I /usr/share/rubygems -r ./siteconf20210805-202311-syz9wk.rb extconf.rb
mkmf.rb can't find header files for ruby at /usr/share/include/ruby.h

You might have to install separate package for the ruby development
environment, ruby-dev or ruby-devel for example.
...

As prompted by the above error message, I must also install ruby-devel first before installing package_cloud. This is why I have both dependencies included in the above Dockerfile example before I attempt to install package_cloud.

RUN yum -y install ruby ruby-devel

RUN sudo gem install package_cloud

However, it is important to note that this is not necessary for Debian-based operating systems. Installing the ruby-full package in the following manner is all that is required for package_cloud to install successfully:

RUN apt-get -y install ruby-full

Setting PKG_CLOUD_OS_NAME in Dockerfile

Note that you must correctly set the PKG_CLOUD_OS_NAME Docker environment variable to the operating system name and version that would normally be used by the package_cloud CLI tool in the following command:

package_cloud push souffle-lang/souffle-test/<PKG_CLOUD_OS_NAME> ...

For example, if I was pushing a package to the souffle-test repository in PackageCloud for Ubuntu 21.04, I would use the following package_cloud command.

package_cloud push souffle-lang/souffle-test/ubuntu/hirsute ...

Therefore, the PKG_CLOUD_OS_NAME variable in my Dockerfile for Ubuntu 21.04 would be ubuntu/hirsute.

To discover the operating system name and version combinations supported by PackageCloud, use their CLI tool to interactively upload packages.

For example, to push a random package interactively via the package_cloud CLI tool, run the following command:

PACKAGECLOUD_TOKEN=<token> package_cloud push souffle-lang/souffle-test random-package.deb

This would yield the following operating systems supported by PackageCloud:

Using https://packagecloud.io with token:******9dcc
/home/phao5814/.gem/ruby/gems/json_pure-1.8.1/lib/json/common.rb:155: warning: Using the last argument as keyword parameters is deprecated
deb packages require an operating system and version to be selected.

If you don't see your OS or version here, send us an email at support@packagecloud.io:

	0. Ubuntu
	1. Debian
	2. LinuxMint
	3. Raspbian
	4. elementary OS

 0-4:

For example, if I'm interested in seeing which versions of Ubuntu are supported by PackageCloud, selecting Ubuntu yields the following options:

You selected Ubuntu. Select a version:

	0. 4.10 Warty Warthog (warty)
	1. 5.04 Hoary Hedgehog (hoary)
	2. 5.10 Breezy Badger (breezy)
	3. 6.06 LTS Dapper Drake (dapper)
	4. 6.10 Edgy Eft (edgy)
	5. 7.04 Feisty Fawn (feisty)
	6. 7.10 Gutsy Gibbon (gutsy)
	7. 8.04 LTS Hardy Heron (hardy)
	8. 8.10 Intrepid Ibex (intrepid)
	9. 9.04 Jaunty Jackalope (jaunty)
	10. 9.10 Karmic Koala (karmic)
	11. 10.04 LTS Lucid Lynx (lucid)
	12. 10.10 Maverick Meerkat (maverick)
	13. 11.04 Natty Narwhal (natty)
	14. 11.10 Oneiric Ocelot (oneiric)
	15. 12.04 LTS Precise Pangolin (precise)
	16. 12.10 Quantal Quetzal (quantal)
	17. 13.04 Raring Ringtail (raring)
	18. 13.10 Saucy Salamander (saucy)
	19. 14.04 LTS Trusty Tahr (trusty)
	20. 14.10 Utopic Unicorn (utopic)
	21. 15.04 Vivid Vervet (vivid)
	22. 15.10 Wily Werewolf (wily)
	23. 16.04 LTS Xenial Xerus (xenial)
	24. 16.10 Yakkety Yak (yakkety)
	25. 17.04 Zesty Zapus (zesty)
	26. 17.10 Artful Aardvark (artful)
	27. 18.04 LTS Bionic Beaver (bionic)
	28. 18.10 Cosmic Cuttlefish (cosmic)
	29. 19.04 Disco Dingo (disco)
	30. 19.10 Eoan Ermine (eoan)
	31. 20.04 Focal Fossa (focal)
	32. 20.10 Groovy Gorilla (groovy)
	33. 21.04 Hirsute Hippo (hirsute)

A common ruby version issue I encountered when creating Dockerfiles

A common issue I encountered during the creation of Dockerfiles related to the installation of ruby as a dependency of the package_cloud CLI tool.

...
RUN yum -y install ruby ruby-devel
RUN sudo gem install package_cloud
...

At the time of writing, according to PackageCloud's documentation, installing the latest version of ruby is all that is necessary to install and use the package_cloud CLI tool as explained below:

If you don't already have Ruby, you need to install it. You can find install instructions for most platforms here.

However, unfortunately, my experience has been that if you install ruby version 3.x at the time of writing, the package_cloud CLI tool will throw errors when you attempt to use it to upload packages to PackageCloud.

For example, with ruby version 3.0.2 installed, if I attempt to upload a package using the package_cloud CLI tool, I get the following error:

...
/usr/local/share/gems/gems/json_pure-1.8.1/lib/json/common.rb:155:in `initialize': wrong number of arguments (given 2, expected 1) (ArgumentError)
...

In the official ruby documentation, this issue is well documented as a breaking change from ruby version 2.x to 3.x:

In Ruby 3.0, positional arguments and keyword arguments will be separated. Ruby 2.7 will warn for behaviors that will change in Ruby 3.0. If you see the following warnings, you need to update your code...

Unfortunately, it appears the package_cloud CLI tool has not adjusted their code for this change.

As a result, when installing ruby, make sure that you are installing 2.x to ensure that the package_cloud CLI tool behaves as intended. Various workarounds to ensure this happens may be required.

For example, one such method is to install rvm first (which is ruby version manager) and then install ruby version 2.7.4 using rvm. This is not ideal; however, is a viable workaround until PackageCloud fix their CLI tool.

Conclusion

These are the changes that are required to create and upload packages for a new operating system and architecture combination.

How to create packages for a new architecture?

Currently, the two supported architectures that we can create packages for are 32bit and 64bit.

This is dictated by the switch statements within the .github/actions/create-package/entrypoint.sh file that is shared across all Dockerfiles as shown below:

# Run the build command
case "$DOMAIN_SIZE" in
  	"64bit")
		cmake -S . -B ./build -DSOUFFLE_DOMAIN_64BIT=ON
    ;;

  	"32bit")
		cmake -S . -B ./build
    ;;
esac

The user can specify which architecture they are building for by specifying the DOMAIN_SIZE environment variable in the Dockerfile as shown below:

...
ENV DOMAIN_SIZE "64bit"
...

Therefore, in order to support the creation of packages for other architectures, we can simply expand the switch statement in our entrypoint.sh file. For example, if we wanted to build packages for arm64, we can simply create the following case branch. We would also need to specify the corresponding cmake command for arm64.

case "$DOMAIN_SIZE" in
		...
		# If we want to support arm64, we create a new case like so
  	"arm64")
				# cmake build command for arm64 goes here
    ;;
esac

Now we can trigger this new arm64 branch from a Dockerfile for an arm64 operating system by setting the DOMAIN_SIZE variable as arm64 as shown below:

...
ENV DOMAIN_SIZE "arm64"
...

How to change the PackageCloud repository we upload packages to?

Below is the last line of our .github/actions/create-package/entrypoint.sh file that is used by all Dockerfiles to create and upload packages to PackageCloud.

...
# Upload the package to packagecloud.io
PACKAGECLOUD_TOKEN="$PACKAGE_CLOUD_API_KEY" package_cloud push souffle-lang/souffle-test/$PKG_CLOUD_OS_NAME "$(ls *$PKG_EXTENSION | head -n1)"

Currently, we are uploading packages to the souffle-test repository in the souffle-lang account.

In order to change the repository we wish to upload packages to, first check that the repository you wish to upload to exists.

At the time of writing the four repositories that currently exist on PackageCloud are:

  • souffle
  • prod
  • souffle-test
  • demo

You can view the available PackageCloud repos here.

For example, if you wish to upload all packages to the souffle repository instead, change the last line of the above entrypoint.sh file to the following:

...
# Upload the package to packagecloud.io
PACKAGECLOUD_TOKEN="$PACKAGE_CLOUD_API_KEY" package_cloud push souffle-lang/souffle/$PKG_CLOUD_OS_NAME "$(ls *$PKG_EXTENSION | head -n1)"

If you wish to upload packages to a repo that does not exist, you must create the repo first. This can be done manually via the package_cloud CLI tool.

The command to do this is as follows and more documentation about this command can be found here:

package_cloud repository create <name of repo>

Instructions for users to install packages uploaded to PackageCloud

The instructions for users to install the packages, which have been uploaded to PackageCloud generally follow the following structure.

  1. Install the souffle-test PackageCloud repository via the following command, which is the same command across all operating systems:
curl -s https://packagecloud.io/install/repositories/souffle-lang/souffle-test/script.deb.sh | sudo bash
  1. Download and install the package via their package manager e.g apt, apt-get, yum, dnf etc.

Example instructions for Ubuntu 21.04 users

Assuming that we have uploaded the souffle package for Ubuntu 21.04 to the souffle-test PackageCloud repository, a user running Ubuntu 21.04 would be able to follow the below steps to install souffle via their package manager.

  1. Install the souffle-test PackageCloud repository via the following command.
curl -s https://packagecloud.io/install/repositories/souffle-lang/souffle-test/script.deb.sh | sudo bash
  1. Download and install the package via apt or apt-get.

If the user wants to install a specific version that we have uploaded to PackageCloud, run the following command:

sudo apt-get install souffle=2.0.2

If the user wants to install the latest version that we have uploaded to PackageCloud, run the following command:

sudo apt-get install souffle

The above instructions are also conveniently outlined on the webpage for each individual package that has been uploaded to PackageCloud. The link to the webpage listing all the packages that have been uploaded to the souffle-test repository can be accessed here.

Important note when uploading to a different PackageCloud repository

Currently, we are uploading all packages to the souffle-test PackageCloud repository. However, if this changes to a different repository in the future e.g. we decide to upload packages to the souffle repository instead, the user instructions for installing a package must also be modified.

Namely, the link to install the PackageCloud repository would change to the following since we are now uploading to the souffle repository instead of the souffle-test repository:

curl -s https://packagecloud.io/install/repositories/souffle-lang/souffle/script.deb.sh | sudo bash
Add a custom footer

Build Soufflé for developers

Design of Souffle

Clone this wiki locally