Introduce thing-flinger

package/Cargo.toml

@@ -6,7 +6,9 @@ license = "MPL-2.0"
 
 [dependencies]
 anyhow = "1.0"
+crossbeam = "0.8"
 omicron-common = { path = "../common" }
+
 # We depend on the propolis-server here -- a binary, not a library -- to
 # make it visible to the packaging tool, which can compile it and shove
 # it in a tarball.
@@ -28,3 +30,7 @@ walkdir = "2.3"
 [[bin]]
 name = "omicron-package"
 doc = false
+
+[[bin]]
+name = "thing-flinger"
+doc = false

package/README.adoc

@@ -0,0 +1,142 @@
+Omicron is a complex piece of software consisting of many build and install-time dependencies. It's
+intended to run primarily on illumos based systems, and as such is built to use runtime facilities
+of illumos, such as https://illumos.org/man/5/smf[SMF]. Furthermore, Omicron is fundamentally a
+distributed system, with its components intended to run on multiple servers communicating over the
+network. In order to secure the system, certain cryptographic primitives, such as asymmetric key
+pairs and shared secrets are required. Due to the nature of these cryptographic primitives, there is
+a requirement for the distribution or creation of files unique to a specific server, such that no
+other server has access to those files. Examples of this are private keys, and threshold key
+shares, although other non-cryptographic unique files may also become necessary over time.
+
+In order to satisfy the above requirements of building and deploying a complex distributed system
+consisting of unique, private files, two CLI tools have been created:
+
+  . link:src/bin/omicron-package.rs[omicron-package] - build, package, install on local machine
+  . link:src/bin/thing-flinger.rs[thing-flinger] - build, package, deploy to remote machines
+
+
+If a user is working on their local illumos based machine, and only wants to run
+omicron in single node mode, they should follow the install instruction in
+the link:../README.adoc[Omicron README] and use `omicron-package`. If the user
+wishes for a more complete workflow, where they can code on their local laptop,
+use a remote build machine, and install to multiple machines for a more realistic
+deployment, they should use `thing-flinger`.
+
+The remainder of this document will describe a typical workflow for using
+thing-flinger, pointing out room for improvement.
+
+== Environment and Configuration
+
+
+     +------------------+                +------------------+
+     |                  |                |                  |
+     |                  |                |                  |
+     |      Client      |---------------->     Builder      |
+     |                  |                |                  |
+     |                  |                |                  |
+     +------------------+                +------------------+
+                                                   |
+                                                   |
+                                                   |
+                                                   |
+                       +---------------------------+--------------------------+
+                       |                           |                          |
+                       |                           |                          |
+                       |                           |                          |
+              +--------v---------+       +---------v--------+       +---------v--------+
+              |                  |       |                  |       |                  |
+              |                  |       |                  |       |                  |
+              | Deployed Server  |       | Deployed Server  |       | Deployed Server  |
+              |                  |       |                  |       |                  |
+              |                  |       |                  |       |                  |
+              +------------------+       +------------------+       +------------------+
+
+
+`thing-flinger` defines three types of nodes:
+
+ * Client - Where a user typically edits their code and runs thing-flinger. This can run any OS.
+ * Builder - A Helios box where Omicron is built and packaged
+ * Deployed Server - Helios machines where Omicron will be installed and run
+
+It's not at all necessary for these to be separate nodes. For example, a client and builder can be
+the same machine, as long as it's a Helios box. Same goes for Builder and a deployment server. The
+benefit of this separation though, is that it allows editing on something like a laptop, without
+having to worry about setting up a development environment on an illumos based host.
+
+Machine topology is configured in a `TOML` file that is passed on the command line. All illumos
+machines are listed under `servers`, and just the names are used for configuring a builder and
+deployment servers. An link:src/bin/deployment-example.toml[example] is provided.
+
+Thing flinger works over SSH, and so the user must have the public key of their client configured
+for their account on all servers. SSH agent forwarding is used to prevent the need for the keys of
+the builder to also be on the other servers, thus minimizing needed server configuration.
+
+== Typical Workflow
+
+=== Prerequisites
+
+Ensure you have an account on all illumos boxes, with the client public key in
+`~/.ssh/authorized_keys`.
+
+.The build machine must have Rust and cargo installed, as well as
+all the dependencies for Omicron installed. Following the *prerequisites* in the
+https://github.com/oxidecomputer/omicron/#build-and-run[Build and run] section of the main Omicron
+README is probably a good idea.
+
+=== Command Based Workflow
+
+==== Build thing-flinger on client
+`thing-flinger` is part of the `omicron-package` crate.
+
+`cargo build -p omicron-package`
+
+==== sync
+Copy your source code to the builder. Note that this copies over your `.git` subdirectory on purpose so
+that a branch can be configured for building with the `git_treeish` field in the toml `builder`
+table.
+
+`./target/debug/thing-flinger -c <CONFIG.toml> sync`
+
+==== build-minimal
+Build necessary parts of omicron on the builder, required for future use by thing-flinger.
+
+`./target/debug/thing-flinger -c <CONFIG> build-minimal`
+
+==== package 
+Build and package omicron using `omicron-package` on the builder.
+
+`./target/debug/thing-flinger -c <CONFIG> package`
+
+==== overlay
+Create files that are unique to each deployment server.
+
+`./target/debug/thing-flinger -c <CONFIG> overlay`
+
+==== install 
+Install omicron to all machines, in parallel. This consists of copying the packaged omicron tarballs
+along with overlay files, and omicron-package and its manifest to a `staging` directory on each
+deployment server, and then running omicron-package, installing overlay files, and restarting
+services.
+
+`./target/debug/thing-flinger -c <CONFIG> install`
+
+=== Current Limitations
+
+`thing-flinger` is an early prototype. It has served so far to demonstrate that unique files,
+specifically secret shares, can be created and distributed over ssh, and that omicron can be
+installed remotely using `omicron-package`. It is not currently complete enough to fully test a
+distributed omicron setup, as the underlying dependencies are not configured yet. Specifically,
+`CockroachDB` and perhaps `Clickhouse`, need to be configured to run in multiple server mode. It's
+anticipated that the `overlay` feature of `thing-flinger` can be used to generate and distribute
+configs for this.
+
+=== Design rationale
+
+`thing-flinger` is a command line program written in rust. It was written this way to build upon
+`omicron-package`, which is also in rust, as that is our default language of choice at Oxide.
+`thing-flinger` is based around SSH, as that is the minimal viable requirement for a test tool such
+as this. Additionally, it provides for the most straightforward implementation, and takes the least
+effort to use securely. This particular implementation wraps the openssh ssh client via
+`std::process::Command`, rather than using the `ssh2` crate, because ssh2, as a wrapper around
+`libssh`, does not support agent-forwarding.
+

package/src/bin/deployment-example.toml

@@ -0,0 +1,46 @@
+# This manifest describes the servers that omicron will be installed to, along
+# with any ancillary information specific to a given server.
+#
+# It is ingested by the `thing-flinger` tool.
+
+# This must be an absolute path
+local_source = "/Users/ajs/oxide/omicron"
+
+[builder]
+# `server` must refer to one of the `servers` in the servers table
+server = "atrium"
+
+# This must be an absolute path
+omicron_path = "/home/andrew/oxide/omicron"
+
+# Git branch, sha, etc...
+git_treeish = "thing-flinger2"
+
+[deployment]
+servers = ["sock", "buskin"]
+rack_secret_threshold = 2
+
+# Location where files to install will be placed before running
+# `omicron-package install`
+#
+# This must be an absolute path
+# We specifically allow for $HOME in validating the absolute path
+staging_dir = "$HOME/omicron_staging"
+
+[servers.tarz]
+username = "ajs"
+addr = "tarz.local"
+
+[servers.atrium]
+username = "andrew"
+addr = "atrium.eng.oxide.computer"
+
+[servers.sock]
+username = "andrew"
+addr = "sock.eng.oxide.computer"
+
+[servers.buskin]
+username = "andrew"
+addr = "buskin.eng.oxide.computer"
+
+

package/src/bin/omicron-package.rs

@@ -9,6 +9,7 @@
 use omicron_common::packaging::sha256_digest;
 
 use anyhow::{anyhow, bail, Context, Result};
+use omicron_package::{parse, SubCommand};
 use rayon::prelude::*;
 use reqwest;
 use serde_derive::Deserialize;
@@ -19,7 +20,6 @@ use std::path::{Path, PathBuf};
 use std::process::Command;
 use structopt::StructOpt;
 use tar::Builder;
-use thiserror::Error;
 use tokio::io::AsyncWriteExt;
 
 const S3_BUCKET: &str = "https://oxide-omicron-build.s3.amazonaws.com";
@@ -29,59 +29,6 @@ const PKG: &str = "pkg";
 // Name for the directory component where downloaded blobs are stored.
 const BLOB: &str = "blob";
 
-#[derive(Debug, StructOpt)]
-enum SubCommand {
-    /// Builds the packages specified in a manifest, and places them into a target
-    /// directory.
-    Package {
-        /// The output directory, where artifacts should be placed.
-        ///
-        /// Defaults to "out".
-        #[structopt(long = "out", default_value = "out")]
-        artifact_dir: PathBuf,
-
-        /// The binary profile to package.
-        ///
-        /// True: release, False: debug (default).
-        #[structopt(
-            short,
-            long,
-            help = "True if bundling release-mode binaries"
-        )]
-        release: bool,
-    },
-    /// Checks the packages specified in a manifest, without building.
-    Check,
-    /// Installs the packages to a target machine.
-    Install {
-        /// The directory from which artifacts will be pulled.
-        ///
-        /// Should match the format from the Package subcommand.
-        #[structopt(long = "in", default_value = "out")]
-        artifact_dir: PathBuf,
-
-        /// The directory to which artifacts will be installed.
-        ///
-        /// Defaults to "/opt/oxide".
-        #[structopt(long = "out", default_value = "/opt/oxide")]
-        install_dir: PathBuf,
-    },
-    /// Removes the packages from the target machine.
-    Uninstall {
-        /// The directory from which artifacts were be pulled.
-        ///
-        /// Should match the format from the Package subcommand.
-        #[structopt(long = "in", default_value = "out")]
-        artifact_dir: PathBuf,
-
-        /// The directory to which artifacts were installed.
-        ///
-        /// Defaults to "/opt/oxide".
-        #[structopt(long = "out", default_value = "/opt/oxide")]
-        install_dir: PathBuf,
-    },
-}
-
 #[derive(Debug, StructOpt)]
 #[structopt(name = "packaging tool")]
 struct Args {
@@ -121,15 +68,6 @@ fn run_cargo_on_package(
     Ok(())
 }
 
-/// Errors which may be returned when parsing the server configuration.
-#[derive(Error, Debug)]
-enum ParseError {
-    #[error("Cannot parse toml: {0}")]
-    Toml(#[from] toml::de::Error),
-    #[error("IO error: {0}")]
-    Io(#[from] std::io::Error),
-}
-
 #[derive(Deserialize, Debug)]
 #[serde(rename_all = "lowercase")]
 enum Build {
@@ -200,12 +138,6 @@ struct Config {
     packages: BTreeMap<String, PackageInfo>,
 }
 
-fn parse<P: AsRef<Path>>(path: P) -> Result<Config, ParseError> {
-    let contents = std::fs::read_to_string(path.as_ref())?;
-    let cfg = toml::from_str::<Config>(&contents)?;
-    Ok(cfg)
-}
-
 async fn do_check(config: &Config) -> Result<()> {
     for (package_name, package) in &config.packages {
         println!("Checking {}", package_name);
@@ -356,6 +288,11 @@ fn do_install(
         anyhow!("Cannot create installation directory: {}", err)
     })?;
 
+    println!(
+        "Copying digest.toml from {} to {}",
+        artifact_dir.to_string_lossy(),
+        install_dir.to_string_lossy()
+    );
     // Move the digest of expected packages.
     std::fs::copy(
         artifact_dir.join("digest.toml"),
@@ -454,7 +391,7 @@ fn do_uninstall(
 #[tokio::main]
 async fn main() -> Result<()> {
     let args = Args::from_args_safe().map_err(|err| anyhow!(err))?;
-    let config = parse(&args.manifest)?;
+    let config = parse::<_, Config>(&args.manifest)?;
 
     // Use a CWD that is the root of the Omicron repository.
     if let Ok(manifest) = env::var("CARGO_MANIFEST_DIR") {