From 2e19a3227701feffd86a60f62c0945def950e1a3 Mon Sep 17 00:00:00 2001 From: Fletcher Nichol Date: Sun, 10 Jul 2016 16:44:17 -0500 Subject: [PATCH] Rewrite package install strategy. This changeset contains several logical changes, all of which are coupled together as they were discovered in the process of refactoring the installation logic. The logical changes are broken out into sections below. Pull-Through Local Caching Package Installation ----------------------------------------------- The implementation for package installation was re-written in order to aggressively use the artifact cache (by default `/hab/cache/artifacts` on disk). The previous implementation was built up over time with installation from local artifacts added at a later date. Consequently, many installation inconsistencies were present: * When performing an installation from a local package artifact, all dependencies were blindly installed from an upstream Depot--whether or not they were installed or even cached on disk. * When installing from a package identifier string, the fully dependency list was queried from an upstream Depot even though the fetched package artifact contained this metadata. Furthermore, the artifact's own metadata should be trusted over any other source, assuming the package is confirmed "verified". * The local package artifact installation read package dependencies from its metadata file correctly which uncovered a bug (fixed in #1013) which would have been found earlier had all installation strategies uses the same code path. * When performing an installation from a local package artifact, the artifact file was never copied into the artifact cache for future re-use. Neither were any locally available dependency package artifacts. This change introduces a new internal struct called an `InstallTask` that maintains knowledge of various directories as well as a common Depot client which is only constructed once. There are 2 primary installation strategies still present: install from a package identifier string (ex: `core/redis`) or install from a local package artifact (ex: `*.hart`). Effort was made to ensure that both approaches use the same code path as much as possible to minimize future behavioral bugs. The updated installation high level strategies are summarized here: From Package Identifier: 1. Is the given package identifier fully qualified? If not, query the upstream Depot for the latest package identifier that satisfies the given "fuzzy" package identifier. In this way a given `core/redis` may return a fully qualified `core/redis/3.0.7/20160614001713` identifier. 2. Is the specific release already installed for the determined fully qualified package identifier? If so, return early as there is no further work to do. 3. Is there a package artifact for the specific release in the artifact cache? If there is, we use this one. Otherwise, fetch the specific release from the upstream Depot and write the package artifact into the artifact cache. 4. Verify the package artifact to ensure that its fully qualified package identifier precisely matches the intended target package identifier. Next, verify that the package artifact is correctly signed and is self-consistent. 5. Determine the fully list of runtime dependencies by extracting the `TDEPS` metadata file from the local package artifact. For each dependency, perform steps 2 through 4 above so that all dependencies are either confirmed pre-installed or each cached package artifact has been fully verified. 6. Extract/unpack each package artifact dependency that was not found to be pre-installed, followed by the target package artifact last. The order is important to ensure that target package is not installed until all of its dependencies are correctly installed. From Local Package Artifact: 1. Extract the fully qualified package identifier from the `IDENT` metadata file in the package artifact. 2. Is the specific release already installed for the determined fully qualified package identifier? If so, return early as there is no further work to do. 3. Copy the artifact into the artifact cache and rewrite the file name based on the computed artifact name. This means that we can install from a Habitat package with any arbitrary name like `file.1` and it would be rewritten to `core-redis-3.0.7-20160614001713-x86_64-linux.hart` in the artifact cache. 4. Verify the package artifact to ensure that its fully qualified package identifier precisely matches the intended target package identifier. Next, verify that the package artifact is correctly signed and is self-consistent. 5. Determine the fully list of runtime dependencies by extracting the `TDEPS` metadata file from the local package artifact. For each dependency, perform steps 2 through 4 in the "From Package Identifier" section so that all dependencies are either confirmed pre-installed or each cached package artifact has been fully verified. There is one difference however: for each dependency, first check to see if a package artifact exists in the same directory that contained the original target package artifact. There is a good chance that if the directory contains many other package artifacts that we can use these rather than re-fetching the exact same ones from an upstream Depot. Unlike the "From Package Identifier" scenario, we have an explicit directory we can check, so there is no harm in trying. 6. Extract/unpack each package artifact dependency that was not found to be pre-installed, followed by the target package artifact last. The order is important to ensure that target package is not installed until all of its dependencies are correctly installed. New: Supervisor Start From Package Artifact ------------------------------------------- In the course of the installation refactoring above, an easy new improvement (or feature) presented itself: the ability to start supervising a package that has not yet been installed, given a local package artifact. In other words, this change now supports the following: hab start ./results/core-redis-3.0.7-20160614001713-x86_64-linux.hart or: hab sup start /tmp/redis.hart assuming that `/tmp/redis.hart` is a legitimate Habitat package artifact and was simply renamed. As with the installation implementation rewrite above, if the package is already installed, then there is no work to do, otherwise the "From Local Package Artifact" strategy above is used. The intended primary use case for this pattern in for Plan authors working on their Plans in a Studio instance. This will allow them to build a package artifact and them immediately start it without having to install it beforehand. Secondarily, this pattern may be useful in environments with slow or limited internet access and eventually fully "air gapped" environments. Fixed: at-once Update Strategy ------------------------------ Also in the course of the above refactoring 2 bugs were found in the "at-once" update strategy in the Supervisor: 1. Once a new version of a package is found in an upstream Depot, it is installed but the new version is not loaded when `start_package()` is called (source: https://github.com/habitat-sh/habitat/blob/9e25233fafe0433864ab47d0d8ede6344d30be8a/components/sup/src/command/start.rs#L120) with the consequence that the original release is re-loaded and new version is never used until the Supervisor completely restarts. 2. The former custom install logic in the "at-once" strategy only fetched and extracted the new target package meaning that if the new release contained updated dependency versions, these would not be installed and could result in the supervised process entering a crash loop. The fix for this was simply to invoke the same installation logic as used everywhere else in the system and is explained in much greater detail above. Improved: Removed Optional Depot URL in Supervisor -------------------------------------------------- While refactoring the Supervisor component, it became clear that earlier versions of the codebase accepted an *optional* upstream Depot URL. Currently, fall-back default URL values are used and so we can guarantee that a value is always present for the Supervisor. The config value of `url` was updated to simply be a `String` (formerly an `Option`) and all extra checking was removed where a value may not have been present. Less branching logic is almost always a good thing! Bonus: Remove Unused Imports ---------------------------- In the course of regression testing, several "unused import" warnings were found and removed. A few were due to code refactoring, but others were most likely due to upgrading the Rust compiler toolchain in the last few weeks. Just keeping the shop floor swept. Signed-off-by: Fletcher Nichol --- .../common/src/command/package/install.rs | 380 +++++++++--------- components/common/src/error.rs | 10 + components/hab/src/command/pkg.rs | 14 +- components/hab/src/exec.rs | 14 +- components/hab/src/gossip.rs | 2 +- components/sup/src/command/start.rs | 93 +++-- components/sup/src/config.rs | 18 +- components/sup/src/main.rs | 22 +- components/sup/src/topology/mod.rs | 4 +- components/sup/tests/util/command.rs | 1 - 10 files changed, 297 insertions(+), 261 deletions(-) diff --git a/components/common/src/command/package/install.rs b/components/common/src/command/package/install.rs index 32b812a335..f664f0febd 100644 --- a/components/common/src/command/package/install.rs +++ b/components/common/src/command/package/install.rs @@ -35,19 +35,19 @@ //! * Unpack it //! +use std::fs; use std::path::{Path, PathBuf}; use std::str::FromStr; use ansi_term::Colour::{Blue, Green, Yellow}; use depot_client::Client; +use hcore; use hcore::crypto::{artifact, SigKeyPair}; use hcore::crypto::keys::parse_name_with_rev; -use hcore::fs::cache_artifact_path; use hcore::package::{Identifiable, PackageArchive, PackageIdent, PackageInstall}; -use protocol::depotsrv; use command::ProgressBar; -use error::Result; +use error::{Error, Result}; pub fn start(url: &str, ident_or_archive: &str, @@ -56,221 +56,223 @@ pub fn start(url: &str, fs_root_path: &P1, cache_artifact_path: &P2, cache_key_path: &P3) - -> Result<()> + -> Result where P1: AsRef, P2: AsRef, P3: AsRef { + let task = try!(InstallTask::new(url, + product, + version, + fs_root_path.as_ref(), + cache_artifact_path.as_ref(), + cache_key_path.as_ref())); + if Path::new(ident_or_archive).is_file() { - try!(from_archive(url, - &ident_or_archive, - product, - version, - fs_root_path, - cache_artifact_path, - cache_key_path)); + task.from_artifact(&Path::new(ident_or_archive)) } else { - let ident = try!(PackageIdent::from_str(ident_or_archive)); - try!(from_url(url, - &ident, - product, - version, - fs_root_path, - cache_artifact_path, - cache_key_path)); + task.from_ident(try!(PackageIdent::from_str(ident_or_archive))) } - Ok(()) } -/// Given a package name and a base url, downloads the package -/// to the cache artifact path. Returns the filename in the cache as a String -/// -/// # Failures -/// -/// * Fails if it cannot download the package from the upstream -pub fn from_url(url: &str, - ident: &PackageIdent, - product: &str, - version: &str, - fs_root_path: &P1, - cache_artifact_path: &P2, - cache_key_path: &P3) - -> Result - where P1: AsRef, - P2: AsRef, - P3: AsRef -{ - println!("{}", - Yellow.bold().paint(format!("» Installing {}", ident))); - let depot_client = try!(Client::new(url, product, version, Some(fs_root_path.as_ref()))); - let pkg_data = try!(depot_client.show_package(ident.clone())); - for dep in pkg_data.get_tdeps().into_iter() { - let d: PackageIdent = (*dep).clone().into(); - try!(install_from_depot(url, - &d, - &d, - product, - version, - fs_root_path.as_ref(), - cache_artifact_path.as_ref(), - cache_key_path.as_ref())); - } - try!(install_from_depot(url, - &pkg_data.get_ident().clone().into(), - ident, - product, - version, - fs_root_path.as_ref(), - cache_artifact_path.as_ref(), - cache_key_path.as_ref())); - println!("{}", - Blue.paint(format!("★ Install of {} complete with {} packages installed.", - ident, - 1 + &pkg_data.get_tdeps().len()))); - Ok(pkg_data) +struct InstallTask<'a> { + depot_client: Client, + fs_root_path: &'a Path, + cache_artifact_path: &'a Path, + cache_key_path: &'a Path, } -pub fn from_archive(url: &str, - path: &P1, - product: &str, - version: &str, - fs_root_path: &P2, - cache_artifact_path: &P3, - cache_key_path: &P4) - -> Result<()> - where P1: AsRef, - P2: AsRef, - P3: AsRef, - P4: AsRef -{ - println!("{}", - Yellow.bold().paint(format!("» Installing {}", path.as_ref().display()))); - let mut archive = PackageArchive::new(PathBuf::from(path.as_ref())); - let ident = try!(archive.ident()); - let tdeps = try!(archive.tdeps()); - for dep in &tdeps { - try!(install_from_depot(url, - &dep, - dep.as_ref(), - product, - version, - fs_root_path.as_ref(), - cache_artifact_path.as_ref(), - cache_key_path.as_ref())); +impl<'a> InstallTask<'a> { + pub fn new(url: &str, + product: &str, + version: &str, + fs_root_path: &'a Path, + cache_artifact_path: &'a Path, + cache_key_path: &'a Path) + -> Result { + Ok(InstallTask { + depot_client: try!(Client::new(url, product, version, Some(fs_root_path))), + fs_root_path: fs_root_path, + cache_artifact_path: cache_artifact_path, + cache_key_path: cache_key_path, + }) } - try!(install_from_archive(url, - archive, - &ident, - product, - version, - fs_root_path.as_ref(), - cache_key_path.as_ref())); - println!("{}", - Blue.paint(format!("★ Install of {} complete with {} packages installed.", - &ident, - 1 + &tdeps.len()))); - Ok(()) -} -fn install_from_depot(url: &str, - ident: &PackageIdent, - given_ident: &PackageIdent, - product: &str, - version: &str, - fs_root_path: &Path, - cache_artifact_path: &Path, - cache_key_path: &Path) - -> Result<()> { - match PackageInstall::load(ident, Some(&fs_root_path)) { - Ok(_) => { - if given_ident.fully_qualified() { + pub fn from_ident(&self, ident: PackageIdent) -> Result { + println!("{}", + Yellow.bold().paint(format!("» Installing {}", &ident))); + let mut ident = ident; + if !ident.fully_qualified() { + ident = try!(self.fetch_latest_pkg_ident_for(ident)); + } + if try!(self.is_package_installed(&ident)) { + println!("{} {}", Green.paint("→ Using"), &ident); + println!("{}", + Blue.paint(format!("★ Install of {} complete with {} new packages \ + installed.", + &ident, + 0))); + return Ok(ident); + } + + self.install_package(ident, None) + } + + pub fn from_artifact(&self, artifact_path: &Path) -> Result { + let ident = try!(PackageArchive::new(artifact_path).ident()); + if try!(self.is_package_installed(&ident)) { + println!("{} {}", Green.paint("→ Using"), &ident); + println!("{}", + Blue.paint(format!("★ Install of {} complete with {} new packages \ + installed.", + &ident, + 0))); + return Ok(ident); + } + try!(self.cache_artifact(&ident, artifact_path)); + let src_path = artifact_path.parent().unwrap(); + + self.install_package(ident, Some(src_path)) + } + + fn install_package(&self, + ident: PackageIdent, + src_path: Option<&Path>) + -> Result { + let mut artifact = try!(self.get_cached_artifact(ident.clone(), src_path)); + let mut artifacts: Vec = Vec::new(); + + for ident in try!(artifact.tdeps()) { + if try!(self.is_package_installed(&ident)) { println!("{} {}", Green.paint("→ Using"), ident); } else { - println!("{} {} which satisfies {}", - Green.paint("→ Using"), - ident.as_ref(), - given_ident); + artifacts.push(try!(self.get_cached_artifact(ident, src_path))); } } - Err(_) => { - println!("{} {}", - Green.bold().paint("↓ Downloading"), - ident.as_ref()); - let mut progress = ProgressBar::default(); - let depot_client = try!(Client::new(url, product, version, Some(fs_root_path))); - let mut archive = try!(depot_client.fetch_package((*ident).clone(), - cache_artifact_path, - Some(&mut progress))); - let ident = try!(archive.ident()); - try!(verify(url, - &archive, - &ident, - product, - version, - fs_root_path, - cache_key_path)); - try!(archive.unpack(Some(fs_root_path))); - println!("{} {}", Green.bold().paint("✓ Installed"), ident.as_ref()); + artifacts.push(artifact); + + let num_installed = artifacts.len(); + for mut artifact in artifacts { + try!(self.extract_artifact(&mut artifact)); } + println!("{}", + Blue.paint(format!("★ Install of {} complete with {} new packages installed.", + &ident, + num_installed))); + Ok(ident) } - Ok(()) -} -fn install_from_archive(url: &str, - archive: PackageArchive, - ident: &PackageIdent, - product: &str, - version: &str, - fs_root_path: &Path, - cache_key_path: &Path) - -> Result<()> { - match PackageInstall::load(ident.as_ref(), Some(&fs_root_path)) { - Ok(_) => { - println!("{} {}", Green.paint("→ Using"), ident); + fn get_cached_artifact(&self, + ident: PackageIdent, + src_path: Option<&Path>) + -> Result { + if try!(self.is_artifact_cached(&ident)) { + debug!("Found {} in artifact cache, skipping remote download", + &ident); + } else { + try!(self.fetch_artifact(&ident, src_path)); } - Err(_) => { - println!("{} {} from cache", - Green.bold().paint("← Extracting"), - ident); - try!(verify(url, - &archive, - &ident, - product, - version, - fs_root_path, - cache_key_path)); - try!(archive.unpack(Some(fs_root_path))); - println!("{} {}", Green.bold().paint("✓ Installed"), ident); + + let mut artifact = PackageArchive::new(try!(self.cached_artifact_path(&ident))); + try!(self.verify_artifact(&ident, &mut artifact)); + Ok(artifact) + } + + fn extract_artifact(&self, artifact: &mut PackageArchive) -> Result<()> { + try!(artifact.unpack(Some(self.fs_root_path))); + println!("{} {}", + Green.bold().paint("✓ Installed"), + try!(artifact.ident())); + Ok(()) + } + + fn is_package_installed(&self, ident: &PackageIdent) -> Result { + match PackageInstall::load(ident, Some(self.fs_root_path)) { + Ok(_) => Ok(true), + Err(hcore::Error::PackageNotFound(_)) => Ok(false), + Err(e) => Err(Error::HabitatCore(e)), } } - Ok(()) -} -/// get the signer for the artifact and see if we have the key locally. -/// If we don't, attempt to download it from the depot. -fn verify(url: &str, - archive: &PackageArchive, - ident: &PackageIdent, - product: &str, - version: &str, - fs_root_path: &Path, - cache_key_path: &Path) - -> Result<()> { - let nwr = try!(artifact::artifact_signer(&archive.path)); - if let Err(_) = SigKeyPair::get_public_key_path(&nwr, cache_key_path) { + fn is_artifact_cached(&self, ident: &PackageIdent) -> Result { + Ok(try!(self.cached_artifact_path(ident)).is_file()) + } + + fn cached_artifact_path(&self, ident: &PackageIdent) -> Result { + let name = match ident.archive_name() { + Some(n) => n, + None => { + return Err(Error::HabitatCore(hcore::Error::InvalidPackageIdent(ident.to_string()))) + } + }; + Ok(self.cache_artifact_path.join(name)) + } + + fn fetch_latest_pkg_ident_for(&self, fuzzy_ident: PackageIdent) -> Result { + Ok(try!(self.depot_client.show_package(fuzzy_ident)).into()) + } + + fn fetch_artifact(&self, ident: &PackageIdent, src_path: Option<&Path>) -> Result<()> { + if let Some(src_path) = src_path { + let name = match ident.archive_name() { + Some(n) => n, + None => return Err( + Error::HabitatCore(hcore::Error::InvalidPackageIdent(ident.to_string()))), + }; + let local_artifact = src_path.join(name); + if local_artifact.is_file() { + try!(self.cache_artifact(ident, &local_artifact)); + return Ok(()); + } + } + + println!("{} {}", Green.bold().paint("↓ Downloading"), ident); + let mut progress = ProgressBar::default(); + try!(self.depot_client + .fetch_package(ident.clone(), self.cache_artifact_path, Some(&mut progress))); + Ok(()) + } + + fn fetch_origin_key(&self, name_with_rev: &str) -> Result<()> { println!("{} {} public origin key", Green.bold().paint("↓ Downloading"), - &nwr); - let (name, rev) = try!(parse_name_with_rev(&nwr)); + &name_with_rev); + let (name, rev) = try!(parse_name_with_rev(&name_with_rev)); let mut progress = ProgressBar::default(); - let depot_client = try!(Client::new(url, product, version, Some(fs_root_path))); - try!(depot_client.fetch_origin_key(&name, &rev, cache_key_path, Some(&mut progress))); + try!(self.depot_client + .fetch_origin_key(&name, &rev, self.cache_key_path, Some(&mut progress))); println!("{} {} public origin key", Green.bold().paint("☑ Cached"), - &nwr); + &name_with_rev); + Ok(()) + } + + fn cache_artifact(&self, ident: &PackageIdent, artifact_path: &Path) -> Result<()> { + let name = match ident.archive_name() { + Some(n) => n, + None => { + return Err(Error::HabitatCore(hcore::Error::InvalidPackageIdent(ident.to_string()))) + } + }; + try!(fs::copy(artifact_path, self.cache_artifact_path.join(name))); + Ok(()) } - try!(archive.verify(&cache_key_path)); - info!("Verified {} signed by {}", &ident, &nwr); - Ok(()) + fn verify_artifact(&self, ident: &PackageIdent, artifact: &mut PackageArchive) -> Result<()> { + let artifact_ident = try!(artifact.ident()); + if ident != &artifact_ident { + return Err(Error::ArtifactIdentMismatch((artifact.file_name(), + artifact_ident.to_string(), + ident.to_string()))); + } + + let nwr = try!(artifact::artifact_signer(&artifact.path)); + if let Err(_) = SigKeyPair::get_public_key_path(&nwr, self.cache_key_path) { + try!(self.fetch_origin_key(&nwr)); + } + + try!(artifact.verify(&self.cache_key_path)); + info!("Verified {} signed by {}", ident, &nwr); + Ok(()) + } } diff --git a/components/common/src/error.rs b/components/common/src/error.rs index 4d087eb8ce..3d543fe6a0 100644 --- a/components/common/src/error.rs +++ b/components/common/src/error.rs @@ -27,6 +27,7 @@ pub type Result = result::Result; #[derive(Debug)] pub enum Error { + ArtifactIdentMismatch((String, String, String)), CantUploadGossipToml, CryptoKeyError(String), GossipFileRelativePath(String), @@ -46,6 +47,12 @@ pub enum Error { impl fmt::Display for Error { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { let msg = match *self { + Error::ArtifactIdentMismatch((ref a, ref ai, ref i)) => { + format!("Artifact ident {} for `{}' does not match expected ident {}", + ai, + a, + i) + } Error::CantUploadGossipToml => { format!("Can't upload gossip.toml, it's a reserved file name") } @@ -72,6 +79,9 @@ impl fmt::Display for Error { impl error::Error for Error { fn description(&self) -> &str { match *self { + Error::ArtifactIdentMismatch((_, _, _)) => { + "Artifact ident does not match expected ident" + } Error::CantUploadGossipToml => "Can't upload gossip.toml, it's a reserved filename", Error::CryptoKeyError(_) => "Missing or invalid key", Error::GossipFileRelativePath(_) => { diff --git a/components/hab/src/command/pkg.rs b/components/hab/src/command/pkg.rs index ed2df65107..3d6782eeeb 100644 --- a/components/hab/src/command/pkg.rs +++ b/components/hab/src/command/pkg.rs @@ -219,13 +219,13 @@ pub mod export { println!("Searching for {} in remote {}", &format_ident.to_string(), &default_depot_url()); - try!(install::from_url(&default_depot_url(), - format_ident, - PRODUCT, - VERSION, - Path::new(FS_ROOT_PATH), - &cache_artifact_path(None), - &default_cache_key_path(None))); + try!(install::start(&default_depot_url(), + &format_ident.to_string(), + PRODUCT, + VERSION, + Path::new(FS_ROOT_PATH), + &cache_artifact_path(None), + &default_cache_key_path(None))); } } let pkg_arg = OsString::from(&ident.to_string()); diff --git a/components/hab/src/exec.rs b/components/hab/src/exec.rs index 66d81c6aa9..bb710621b2 100644 --- a/components/hab/src/exec.rs +++ b/components/hab/src/exec.rs @@ -98,13 +98,13 @@ pub fn command_from_pkg(command: &str, println!("{}", Cyan.bold() .paint(format!("∵ Package for {} not found, installing", &ident))); - try!(common::command::package::install::from_url(&default_depot_url(), - ident, - PRODUCT, - VERSION, - fs_root_path, - &cache_artifact_path(None), - cache_key_path)); + try!(common::command::package::install::start(&default_depot_url(), + &ident.to_string(), + PRODUCT, + VERSION, + fs_root_path, + &cache_artifact_path(None), + cache_key_path)); command_from_pkg(&command, &ident, &cache_key_path, retry + 1) } Err(e) => return Err(Error::from(e)), diff --git a/components/hab/src/gossip.rs b/components/hab/src/gossip.rs index d07fd29c3c..d1698a1b7a 100644 --- a/components/hab/src/gossip.rs +++ b/components/hab/src/gossip.rs @@ -84,7 +84,7 @@ pub mod hab_gossip { use common::gossip_file::GossipFile; use common::wire_message::WireMessage; use hcore::crypto::SymKey; - use rustc_serialize::{json, Encodable}; + use rustc_serialize::json; use utp::UtpSocket; use uuid::Uuid; diff --git a/components/sup/src/command/start.rs b/components/sup/src/command/start.rs index e2b570226d..26492908b3 100644 --- a/components/sup/src/command/start.rs +++ b/components/sup/src/command/start.rs @@ -60,7 +60,6 @@ use std::env; use std::path::Path; use ansi_term::Colour::Yellow; -use common::command::ProgressBar; use common::command::package::install; use depot_client::Client; use hcore::crypto::default_cache_key_path; @@ -68,7 +67,7 @@ use hcore::fs::{cache_artifact_path, FS_ROOT_PATH}; use hcore::package::PackageIdent; use {PRODUCT, VERSION}; -use error::{Error, Result}; +use error::Result; use config::{Config, UpdateStrategy}; use package::Package; use topology::{self, Topology}; @@ -85,36 +84,38 @@ static LOGKEY: &'static str = "CS"; /// * Fails if an unknown topology was specified on the command line pub fn package(config: &Config) -> Result<()> { match Package::load(config.package(), None) { - Ok(package) => { + Ok(mut package) => { let update_strategy = config.update_strategy(); match update_strategy { UpdateStrategy::None => {} _ => { - if let &Some(ref url) = config.url() { - outputln!("Checking remote for newer versions..."); - // It is important to pass `config.package()` to `show_package()` instead of the - // package identifier of the loaded package. This will ensure that if the operator - // starts a package while specifying a version number, they will only automaticaly - // receive release updates for the started package. - // - // If the operator does not specify a version number they will automatically receive - // updates for any releases, regardless of version number, for the started package. - let depot_client = try!(Client::new(url, PRODUCT, VERSION, None)); - let latest_pkg_data = - try!(depot_client.show_package((*config.package()).clone())); - let latest_ident: PackageIdent = latest_pkg_data.get_ident().clone().into(); - if &latest_ident > package.ident() { - outputln!("Downloading latest version from remote: {}", latest_ident); - let mut progress = ProgressBar::default(); - let archive = try!(depot_client.fetch_package(latest_ident, - &cache_artifact_path(None), - Some(&mut progress))); - try!(archive.verify(&default_cache_key_path(None))); - try!(archive.unpack(None)); - } else { - outputln!("Already running latest."); - }; - } + let url = config.url(); + outputln!("Checking Depot for newer versions..."); + // It is important to pass `config.package()` to `show_package()` instead + // of the package identifier of the loaded package. This will ensure that + // if the operator starts a package while specifying a version number, they + // will only automaticaly receive release updates for the started package. + // + // If the operator does not specify a version number they will + // automatically receive updates for any releases, regardless of version + // number, for the started package. + let depot_client = try!(Client::new(url, PRODUCT, VERSION, None)); + let latest_pkg_data = + try!(depot_client.show_package((*config.package()).clone())); + let latest_ident: PackageIdent = latest_pkg_data.get_ident().clone().into(); + if &latest_ident > package.ident() { + outputln!("Downloading latest version from Depot: {}", latest_ident); + let new_pkg_data = try!(install::start(url, + &latest_ident.to_string(), + PRODUCT, + VERSION, + Path::new(FS_ROOT_PATH), + &cache_artifact_path(None), + &default_cache_key_path(None))); + package = try!(Package::load(&new_pkg_data, None)); + } else { + outputln!("Already running latest."); + }; } } start_package(package, config) @@ -122,24 +123,32 @@ pub fn package(config: &Config) -> Result<()> { Err(_) => { outputln!("{} is not installed", Yellow.bold().paint(config.package().to_string())); - match *config.url() { - Some(ref url) => { + let url = config.url(); + let new_pkg_data = match config.local_artifact() { + Some(artifact) => { + try!(install::start(url, + &artifact, + PRODUCT, + VERSION, + Path::new(FS_ROOT_PATH), + &cache_artifact_path(None), + &default_cache_key_path(None))) + } + None => { outputln!("Searching for {} in remote {}", Yellow.bold().paint(config.package().to_string()), url); - let new_pkg_data = try!(install::from_url(url, - config.package(), - PRODUCT, - VERSION, - Path::new(FS_ROOT_PATH), - &cache_artifact_path(None), - &default_cache_key_path(None))); - let package = try!(Package::load(&new_pkg_data.get_ident().clone().into(), - None)); - start_package(package, config) + try!(install::start(url, + &config.package().to_string(), + PRODUCT, + VERSION, + Path::new(FS_ROOT_PATH), + &cache_artifact_path(None), + &default_cache_key_path(None))) } - None => Err(sup_error!(Error::PackageNotFound(config.package().clone()))), - } + }; + let package = try!(Package::load(&new_pkg_data, None)); + start_package(package, config) } } } diff --git a/components/sup/src/config.rs b/components/sup/src/config.rs index 44408b7735..49f6b89230 100644 --- a/components/sup/src/config.rs +++ b/components/sup/src/config.rs @@ -85,7 +85,8 @@ impl Default for Command { pub struct Config { command: Command, package: PackageIdent, - url: Option, + local_artifact: Option, + url: String, topology: Topology, group: String, path: String, @@ -261,12 +262,12 @@ impl Config { /// Set the url pub fn set_url(&mut self, url: String) -> &mut Config { - self.url = Some(url); + self.url = url; self } /// Return the url - pub fn url(&self) -> &Option { + pub fn url(&self) -> &str { &self.url } @@ -382,6 +383,15 @@ impl Config { &self.package } + pub fn set_local_artifact(&mut self, artifact: String) -> &mut Config { + self.local_artifact = Some(artifact); + self + } + + pub fn local_artifact(&self) -> Option<&str> { + self.local_artifact.as_ref().map(String::as_ref) + } + pub fn set_organization(&mut self, org: String) -> &mut Config { self.organization = Some(org); self @@ -439,7 +449,7 @@ mod tests { fn url() { let mut c = Config::new(); c.set_url(String::from("http://foolio.com")); - assert_eq!(c.url().as_ref().unwrap(), "http://foolio.com"); + assert_eq!(c.url(), "http://foolio.com"); } #[test] diff --git a/components/sup/src/main.rs b/components/sup/src/main.rs index e2c930ecb7..971c6038fa 100644 --- a/components/sup/src/main.rs +++ b/components/sup/src/main.rs @@ -24,6 +24,7 @@ extern crate libc; #[macro_use] extern crate clap; +use std::path::Path; use std::process; use std::result; use std::str::FromStr; @@ -34,7 +35,7 @@ use hcore::env as henv; use hcore::fs; use hcore::crypto::{default_cache_key_path, SymKey}; use hcore::crypto::init as crypto_init; -use hcore::package::PackageIdent; +use hcore::package::{PackageArchive, PackageIdent}; use hcore::url::{DEFAULT_DEPOT_URL, DEPOT_URL_ENVVAR}; use sup::config::{Command, Config, UpdateStrategy}; @@ -74,9 +75,15 @@ fn config_from_args(subcommand: &str, sub_args: &ArgMatches) -> Result { if let Some(ref archive) = sub_args.value_of("archive") { config.set_archive(archive.to_string()); } - if let Some(ref package) = sub_args.value_of("package") { - let ident = try!(PackageIdent::from_str(package)); - config.set_package(ident); + if let Some(ref ident_or_artifact) = sub_args.value_of("pkg_ident_or_artifact") { + if Path::new(ident_or_artifact).is_file() { + let ident = try!(PackageArchive::new(Path::new(ident_or_artifact)).ident()); + config.set_package(ident); + config.set_local_artifact(ident_or_artifact.to_string()); + } else { + let ident = try!(PackageIdent::from_str(ident_or_artifact)); + config.set_package(ident); + } } if let Some(key) = sub_args.value_of("key") { config.set_key(key.to_string()); @@ -255,12 +262,13 @@ fn main() { }; let sub_start = SubCommand::with_name("start") - .about("Start a Habitat-supervised service from a package") + .about("Start a Habitat-supervised service from a package or artifact") .aliases(&["st", "sta", "star"]) - .arg(Arg::with_name("package") + .arg(Arg::with_name("pkg_ident_or_artifact") .index(1) .required(true) - .help("Name of package to start")) + .help("A Habitat package identifier (ex: acme/redis) or a filepath to a Habitat \ + Artifact (ex: /home/acme-redis-3.0.7-21120102031201-x86_64-linux.hart)")) .arg(arg_url()) .arg(arg_group()) .arg(arg_org()) diff --git a/components/sup/src/topology/mod.rs b/components/sup/src/topology/mod.rs index 6ab91fb8ec..7f9d9a9dbd 100644 --- a/components/sup/src/topology/mod.rs +++ b/components/sup/src/topology/mod.rs @@ -148,9 +148,7 @@ impl<'a> Worker<'a> { UpdateStrategy::None => {} _ => { let pkg_lock_2 = pkg_lock.clone(); - if let &Some(ref url) = config.url() { - pkg_updater = Some(package::PackageUpdater::start(url, pkg_lock_2)); - } + pkg_updater = Some(package::PackageUpdater::start(config.url(), pkg_lock_2)); } } diff --git a/components/sup/tests/util/command.rs b/components/sup/tests/util/command.rs index af3ae9085b..abc0397a9f 100644 --- a/components/sup/tests/util/command.rs +++ b/components/sup/tests/util/command.rs @@ -12,7 +12,6 @@ // See the License for the specific language governing permissions and // limitations under the License. -use std::io::prelude::*; use std::io; use std::process::{Command, Child, Stdio, ExitStatus}; use std::fmt;