Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
89a9aa5
Add a script for installing all the software mentioned in the README
smklein Apr 1, 2022
147c97c
Update Readme
smklein Apr 1, 2022
0e5d984
Merge update and install
smklein Apr 3, 2022
6c01f56
Make DB downloads idempotent, contained in 'out/' directory
smklein Apr 3, 2022
876911f
Automate PATH verification, simplify README
smklein Apr 3, 2022
a80dd23
Patch paths in github action
smklein Apr 3, 2022
2626233
Merge branch 'better-db-install' into check-paths-too
smklein Apr 3, 2022
215fea3
stray newline
smklein Apr 3, 2022
866e016
Patch more PATH
smklein Apr 3, 2022
b6c70aa
Merge branch 'better-db-install' into check-paths-too
smklein Apr 3, 2022
40bdee8
Restore old formatting, xtrace
smklein Apr 4, 2022
c0be073
stray tab
smklein Apr 4, 2022
5644c46
Merge branch 'better-db-install' into check-paths-too
smklein Apr 4, 2022
a4ea71c
Add documentation, check clickhouse/cockroach in PATH
smklein Apr 4, 2022
54650f8
Merge branch 'main' into better-install
smklein Apr 4, 2022
3de53f1
More docs, hopefully more clear rc usage
smklein Apr 4, 2022
63e8e9a
Merge branch 'better-install' into better-db-install
smklein Apr 4, 2022
0c0cc8e
Merge branch 'better-db-install' into check-paths-too
smklein Apr 4, 2022
fd77f51
Idempotency message on error
smklein Apr 4, 2022
a3bf2b0
Merge branch 'better-install' into better-db-install
smklein Apr 4, 2022
bdfd478
Merge branch 'better-db-install' into check-paths-too
smklein Apr 4, 2022
809f5f8
Avoid using associative arrays. Macs have old bash
smklein Apr 4, 2022
60cbd71
Merge branch 'main' into check-paths-too
smklein Apr 8, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 1 addition & 51 deletions README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -35,62 +35,12 @@ You can **run the https://github.com/rust-lang/rust-clippy[Clippy linter]** usin

=== Installing Prerequisite Software

The following software may be installed automatically with the following scripts:
The following software may be installed automatically with the following script:

----
$ ./tools/install_prerequisites.sh
----

Alternatively, the manual installation steps as follows:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I admit I'm a little sad to see this section go. I feel like this is useful documentation for the important third-party pieces that we're using. At the same time, it probably doesn't make sense to have both this and an automated version that can get out of sync. I wonder if there's a useful compromise -- what if we move more of this documentation into install_prerequisites.sh itself? Things I could imagine noting in that script:

  • That Diesel uses the native libpq to talk to to PostgreSQL (CockroachDB in our case)
  • That the test suite expects the cockroach and clickhouse commands on your PATH
  • That the brand/omicron1/tools package is used for deploying control plane components into zones on a test machine [as opposed to a local build or test suite run]

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good suggestion, I've added much more documentation to the script itself.


Both normal execution and the test suite expect certain binaries (described below) on your PATH.

. libpq, the PostgreSQL client library
+
--
We use Diesel's PostgreSQL support to connect to CockroachDB (which is wire-compatible with PostgreSQL). Diesel uses the native libpq to do this. You can get the client library with:

* Helios: `pkg install library/postgresql-13`
* Linux: `sudo apt-get install libpq-dev`
* Mac: `brew install postgresql`

After doing this, you should have the `pg_config` command on your PATH. For example, on Helios, you'd want `/opt/ooce/bin` on your PATH.
--
. pkg-config, a tool for querying installed libraries
+
--

* Helios: `pkg install pkg-config`
* Linux: `sudo apt-get install pkg-config`
* Mac: `brew install pkg-config`

After doing this, you should have the `pkg-config` command on your PATH. For
example, on Helios, you'd want `/usr/bin` on your PATH.
--
. CockroachDB v21.1.10.
+
The build and test suite expects to be able to start a single-node CockroachDB cluster using the `cockroach` executable on your PATH.
On illumos, MacOS, and Linux, you should be able to use the `tools/ci_download_cockroachdb` script to fetch the official CockroachDB binary. It will be put into `./out/cockroachdb/bin/cockroach`.
Alternatively, you can follow the https://www.cockroachlabs.com/docs/stable/install-cockroachdb.html[official CockroachDB installation instructions for your platform].

. ClickHouse >= v21.7.1.
+
The test suite expects a running instance of the ClickHouse database server.
The script `./tools/ci_download_clickhouse` can be used to download a pre-built binary for illumos, Linux, or macOS platforms. Once complete, you must manually add the binary (located at `./out/clickhouse/clickhouse`) to your PATH.
You may also install ClickHouse manually; instructions can be found https://clickhouse.tech/docs/en/getting-started/install[here].
See <<_configuring_clickhouse>> for details on ClickHouse's setup and configuration files.
+
. Additional software requirements:
+
On an illumos-based machine (Helios, OmniOS), if you want to run the real (non-simulated) Sled Agent to run actual VMs with Propolis, make sure your packages are up to date, and you have the `brand/omicron1/tools` package:
+
[source,text]
----
pkg install brand/omicron1/tools
pkg install pkg:/package/pkg
pkg update
----

=== Running (Simulated) Omicron

NOTE: If you'd like to run on Helios, refer to <<Deploying Omicron>> below.
Expand Down
77 changes: 77 additions & 0 deletions tools/install_prerequisites.sh
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,22 @@ function confirm
esac
}

# Packages to be installed on all OSes:
#
# - libpq, the PostgreSQL client lib.
# We use Diesel's PostgreSQL support to connect to CockroachDB (which is
# wire-compatible with PostgreSQL). Diesel uses the native libpq to do this.
# `pg_config` is a utility which may be used to query for the installed
# PostgreSQL libraries, and is expected by the Omicron build to exist in
# the developer's PATH variable.
# - pkg-config, a tool for querying installed libraries.
#
# Packages to be installed on Helios only:
#
# - pkg, the IPS client (though likely it will just be updated)
# - build-essential: Common development tools
# - brand/omicron1/tools: Oxide's omicron1-brand Zone

HOST_OS=$(uname -s)
if [[ "${HOST_OS}" == "Linux" ]]; then
packages=(
Expand Down Expand Up @@ -70,5 +86,66 @@ else
exit -1
fi

# CockroachDB and Clickhouse are used by Omicron for storage of
# control plane metadata and metrics.
#
# They are used in a couple of spots within Omicron:
#
# - Test Suite: The test suite, regardless of host OS, builds temporary
# databases for testing, and expects `cockroach` and `clickhouse` to
# exist as a part of the PATH.
# - Packaging: When constructing packages on Helios, these utilities
# are packaged into zones which may be launched by the sled agent.

./tools/ci_download_cockroachdb
./tools/ci_download_clickhouse

# Validate the PATH:
expected_in_path=(
'pg_config'
'pkg-config'
'cockroach'
'clickhouse'
)

function show_hint
{
case "$1" in
"pg_config")
if [[ "${HOST_OS}" == "SunOS" ]]; then
echo "On illumos, $1 is typically found in '/opt/ooce/bin'"
fi
;;
"pkg-config")
if [[ "${HOST_OS}" == "SunOS" ]]; then
echo "On illumos, $1 is typically found in '/usr/bin'"
fi
;;
"cockroach")
echo "$1 should have been installed to '$PWD/out/cockroachdb/bin'"
;;
"clickhouse")
echo "$1 should have been installed to '$PWD/out/clickhouse'"
;;
*)
;;
esac
}

# Check all paths before returning an error.
ANY_PATH_ERROR="false"
for command in "${expected_in_path[@]}"; do
rc=0
which "$command" &> /dev/null || rc=$?
if [[ "$rc" -ne 0 ]]; then
echo "ERROR: $command seems installed, but was not found in PATH. Please add it."
show_hint "$command"
ANY_PATH_ERROR="true"
fi
done

if [[ "$ANY_PATH_ERROR" == "true" ]]; then
exit -1
fi

echo "All prerequisites installed successfully, and PATH looks valid"