Skip to content

Commit

Permalink
Merge pull request #13 from nikogura/v3.0.0
Browse files Browse the repository at this point in the history
V3.0.0
  • Loading branch information
nikogura committed Dec 31, 2019
2 parents f14d8d0 + 4580a29 commit 9a684a4
Show file tree
Hide file tree
Showing 22 changed files with 439 additions and 265 deletions.
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ version: 2
jobs:
build:
docker:
- image: circleci/golang:1.11.2
- image: circleci/golang:1.13.4

environment: GO111MODULE=on

Expand Down
36 changes: 27 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ A framework for running self-updating, signed binary tools from a central, trust

DBT consists of a binary ```dbt``` and a config file. The ```dbt``` binary checks a trusted repository for tools, which are themselves signed binaries.

Tools are automatically downloaded, and verified for checksum and signature before running.
Tools are automatically downloaded, and verified for checksum and signature, then if they pass, they're run. e.g.

The DBT binary itself auto-updates from the trusted repository, and if it's checksum and signature checks out, it executes a 'Single White Female' on itself, replacing itself on the fly with the new version and then running the downloaded tool.

Expand Down Expand Up @@ -54,7 +54,19 @@ DBT does your due diligence for you, and lets you get on with your day.

Another real-world example: Imagine this, you've got a system of dynamic VM's and Containers, all leveraging common tooling. You might even have a serious DAG or web of 'things' dynamically generating other 'things' in a busy and automated fashion. What is there's a problem, or an upgrade? With normal utility tools and scripts you have to re-bake your machine images and containers to pick up the changes. You might say that that's a good thing. But what if it's not?

With DBT, you have the best of both worlds. You can force your tools to use an explicit version (```dbt -v 1.2.3 <tool>```). You can also dispense with the '-v' and run the latest. Voila! You're automatically picking up the latest version of the tooling from your trusted repository.
With DBT, you have the best of both worlds. You can force your tools to use an explicit version (```dbt -v 1.2.3 -- <tool>```). You can also dispense with the '-v' and run the latest. Voila! You're automatically picking up the latest version of the tooling from your trusted repository.

# Usage

Generally speaking, you will run your tools with a command of the form:

dbt [flags] -- <command> <command args and flags>

Take special note of the `--` That double dash separates the flags for `dbt` itself from those of the command. It can get confusing if you don't spot the double dash and grok it's meaning.

Without it, any flags you try to run on `<command>` will be consumed by `dbt` itself, and the result will probably not be what you intend.

Of course, if your command has no flags itself, only positional arguments, you can run it straight without the double dash.

# Security

Expand All @@ -74,9 +86,11 @@ DBT, as you see it here is set up for *my* test repo. You'll need to make some

1. Fork the repo.

2. Change the `metadata.json` file to reflect your own repository setup and preferences. Specifically you need to change the `repository` and `package` lines.
2. Change the `metadata.json` file to reflect your own repository setup and preferences. Specifically you need to change the `repository` and `package` lines.

3. Run `gomason publish`. If you have it all set up correctly, it should build and install the binary as well as the installer script for your version of DBT.
3. You'll also need to change the package name in go.mod, cmd/dbt/main.go, cmd/boilerplate/main.go, and cmd/catalog/main.go. Basically you'll need to wire it up so that your fork is referencing itself, not my public repo. Basic golang stuff. Don't forget to check your changes into your fork. (Sorry. When I work out a good way to make that easier, I will implement it.)

4. Install `gomason` via `go get github.com/nikogura/gomason`. Then run `gomason publish`. If you have it all set up correctly, it should build and install the binary as well as the installer script for your version of DBT.

The details of what all is supported in `metadata.json` can be found in [https://github.com/nikogura/gomason](https://github.com/nikogura/gomason).

Expand All @@ -92,7 +106,13 @@ Then you should see a file `http://localhost:8081/artifactory/dbt/install_dbt.sh
And voila! Your DBT is now installed.

You will, however need to populate the `truststore` file, which by default, with the above config would be located at `http://localhost:8081/artifactory/dbt/truststore`. This file contains the public keys of the entities you trust to create DBT binaries. You can edit this file by hand, it's just a bunch of PEM data squashed together, or you can use one of the tools listed in the next section.
You will, however need to populate the `truststore` file, which by default, with the above config would be located at `http://localhost:8081/artifactory/dbt/truststore`. This file contains the public keys of the entities you trust to create DBT binaries. You can edit this file by hand, it's just a bunch of PEM data squashed together.

_AUTHOR'S NOTE: When I personally maintain an internal fork, I set up a clone of the fork with 2 upstreams: 'origin' is my internal fork, and 'upstream' which is the public github.com/nikogura/dbt. Then I make all my internal changes as required, and when upstream changes, do a `git pull upstream ...`. Usually the only changes/conflicts are in the `metadata.json`._

_Correct the conflicts in `metadata.json`, commit, and `git push origin master` and my CI system takes it from there. It sounds complicated, and it's certainly not trivial, but it's been very reliable to date._

_Rest assured, when I come across a better method, I will not keep it to myself._

# Included Tools

Expand All @@ -106,13 +126,11 @@ There are, however, some common tasks that any user of DBT might want at their f

* *Boilerplate* A tool for generating tool boilerplate. You could do it by hand, but why?

* *Trustmgr* A tool for managing who's public keys are trusted by DBT. (Still under construction. For now you'll need to fill the 'truststore' file with the PEM encoded key or keys you've decided to trust.)

If for some reason you don't want to use the included tools, just remove them from your `metadata.json` and they won't publish.

# Repository Support

The initial versions of DBT are targeted at the [Artifactory Open Source](https://www.jfrog.com/open-source) repo. Any sort of WebDAV server that supports authenticated REST should work fine though.
The initial versions of DBT are targeted at the [Artifactory Open Source](https://www.jfrog.com/open-source) repo. Any sort of WebDAV server that supports authenticated PUT's and GET's should work fine though.

# Configuration

Expand Down Expand Up @@ -244,7 +262,7 @@ The file can consist of multiple public keys such as:
=KIOK
-----END PGP PUBLIC KEY BLOCK-----
There's nothing magical about this file. It's just the keys you've decided to trust. You're free to maintain it by hand if you like, or you can use the dbt tool ```trustmgr```.
There's nothing magical about this file. It's just the keys you've decided to trust, concatenated together. Comments after an `-----END PGP PUBLIC KEY BLOCK-----` or before an `-----BEGIN PGP PUBLIC KEY BLOCK---` are ignored, and can be quite useful for humans trying to maintain this file.

## tools

Expand Down
12 changes: 1 addition & 11 deletions cmd/boilerplate/cmd/gen.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright © 2018 NAME HERE <EMAIL ADDRESS>
// Copyright © 2018 Nik Ogura <nik.ogura@gmail.com>
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -67,14 +67,4 @@ Then it will generate a basic, working tool for you that will compile and publis

func init() {
RootCmd.AddCommand(genCmd)

// Here you will define your flags and configuration settings.

// Cobra supports Persistent Flags which will work for this command
// and all subcommands, e.g.:
// genCmd.PersistentFlags().String("foo", "", "A help for foo")

// Cobra supports local flags which will only run when this command
// is called directly, e.g.:
// genCmd.Flags().BoolP("toggle", "t", false, "Help message for toggle")
}
41 changes: 0 additions & 41 deletions cmd/boilerplate/cmd/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,8 @@ Sure, a DBT tool is just a signed go binary. You can create 'em any old way.
Sometimes, however, you have better things to do with your time, and just want to get something working hence DBT boilerplate.
`,
// Uncomment the following line if your bare application
// has an action associated with it:
// Run: func(cmd *cobra.Command, args []string) { },
}

// Execute adds all child commands to the root command and sets flags appropriately.
// This is called by main.main(). It only needs to happen once to the rootCmd.
func Execute() {
if err := RootCmd.Execute(); err != nil {
fmt.Println(err)
Expand All @@ -49,40 +44,4 @@ func Execute() {
}

func init() {
cobra.OnInitialize(initConfig)

// Here you will define your flags and configuration settings.
// Cobra supports persistent flags, which, if defined here,
// will be global for your application.
//RootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", "config file (default is $HOME/.creator.yaml)")

// Cobra also supports local flags, which will only run
// when this action is called directly.
//RootCmd.Flags().BoolP("toggle", "t", false, "Help message for toggle")
}

// initConfig reads in config file and ENV variables if set.
func initConfig() {
//if cfgFile != "" {
// // Use config file from the flag.
// viper.SetConfigFile(cfgFile)
//} else {
// // Find home directory.
// home, err := homedir.Dir()
// if err != nil {
// fmt.Println(err)
// os.Exit(1)
// }
//
// // Search config in home directory with name ".creator" (without extension).
// viper.AddConfigPath(home)
// viper.SetConfigName(".boilerplate")
//}
//
//viper.AutomaticEnv() // read in environment variables that match
//
//// If a config file is found, read it in.
//if err := viper.ReadInConfig(); err == nil {
// fmt.Println("Using config file:", viper.ConfigFileUsed())
//}
}
12 changes: 1 addition & 11 deletions cmd/catalog/cmd/list.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright © 2017 NAME HERE <EMAIL ADDRESS>
// Copyright © 2017 Nik Ogura <nik.ogura@gmail.com>
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -39,14 +39,4 @@ List available tools.

func init() {
RootCmd.AddCommand(listCmd)

// Here you will define your flags and configuration settings.

// Cobra supports Persistent Flags which will work for this command
// and all subcommands, e.g.:
// listCmd.PersistentFlags().String("foo", "", "A help for foo")

// Cobra supports local flags which will only run when this command
// is called directly, e.g.:
// listCmd.Flags().BoolP("toggle", "t", false, "Help message for toggle")
}
33 changes: 1 addition & 32 deletions cmd/catalog/cmd/root.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright © 2017 NAME HERE <EMAIL ADDRESS>
// Copyright © 2017 Nik Ogura <nik.ogura@gmail.com>
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
Expand All @@ -16,9 +16,7 @@ package cmd

import (
"fmt"
homedir "github.com/mitchellh/go-homedir"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"os"
)

Expand All @@ -33,13 +31,8 @@ Tool for showing available DBT tools.
DBT tools are made available in a trusted repository. This tool show's what's available there.
`,
// Uncomment the following line if your bare application
// has an action associated with it:
// Run: func(cmd *cobra.Command, args []string) { },
}

// Execute adds all child commands to the root command and sets flags appropriately.
// This is called by main.main(). It only needs to happen once to the rootCmd.
func Execute() {
if err := RootCmd.Execute(); err != nil {
fmt.Println(err)
Expand All @@ -48,29 +41,5 @@ func Execute() {
}

func init() {
cobra.OnInitialize(initConfig)

RootCmd.PersistentFlags().BoolVarP(&versions, "versions", "v", false, "Show all version information for tools.")

}

// initConfig reads in config file and ENV variables if set.
func initConfig() {
// Find home directory.
home, err := homedir.Dir()
if err != nil {
fmt.Println(err)
os.Exit(1)
}

// Search config in home directory with name ".catalog" (without extension).
viper.AddConfigPath(home)
viper.SetConfigName(".catalog")

viper.AutomaticEnv() // read in environment variables that match

// If a config file is found, read it in.
if err := viper.ReadInConfig(); err == nil {
fmt.Println("Using config file:", viper.ConfigFileUsed())
}
}
127 changes: 2 additions & 125 deletions cmd/dbt/main.go
Original file line number Diff line number Diff line change
@@ -1,130 +1,7 @@
package main

import (
"fmt"
"github.com/nikogura/dbt/pkg/dbt"
"log"
"os"
"syscall"
)

// DBT the file path of the installed dbt binary
const DBT = "/usr/local/bin/dbt"

// VERSION the version of dbt. Must match version in metadata.json
const VERSION = "2.2.5"

// there are only two options for dbt itself, 'version' and 'offline'
var version string
var offline bool
import "github.com/nikogura/dbt/cmd"

func main() {
args := os.Args[1:]

// exit early if there are no args, or if the first arg is 'help'
exitEarlyIf(args)

dbtObj, err := dbt.NewDbt()
if err != nil {
log.Fatalf("Error creating DBT object: %s", err)
}

possibles := []string{"-o", "-ov"}

if dbt.StringInSlice(args[0], possibles) { // set quiet
offline = true
}

homedir, err := dbt.GetHomeDir()
if err != nil {
log.Fatalf("Failed to discover user homedir: %s\n", err)
}

// if we're not explicitly offline, try to upgrade in place
if !offline {
// first fetch the current truststore
err = dbtObj.FetchTrustStore(homedir, false)
if err != nil {
log.Fatalf("Failed to fetch current truststore: %s.\n\nIf you want to try in 'offline' mode, try your command again with '-o'.", err)
}

ok, err := dbtObj.IsCurrent("")
if err != nil {
log.Printf("Failed to confirm whether we're up to date: %s", err)
}

if !ok {
log.Printf("Downloading and verifying new version of dbt.")
err = dbtObj.UpgradeInPlace("")
if err != nil {
err = fmt.Errorf("upgrade in place failed: %s", err)
log.Fatalf("Error: %s", err)
}

// Single white female ourself
_ = syscall.Exec(DBT, os.Args, os.Environ())
}
}

if len(args) > 0 {
versionPossibles := []string{"-v", "-ov"}
offlinePossibles := []string{"-o"}

if dbt.StringInSlice(args[0], versionPossibles) {
if len(args) > 2 {
version = args[1]

err = dbtObj.RunTool(version, args[2:], homedir, offline)
if err != nil {
log.Fatalf("Error running tool: %s", err)
}

} else {
log.Fatalf("-v flag requires a version.")
}

} else if dbt.StringInSlice(args[0], offlinePossibles) {
err = dbtObj.RunTool(version, args[1:], homedir, offline)
if err != nil {
log.Fatalf("Error running tool: %s", err)
}
} else {
err = dbtObj.RunTool(version, args, homedir, offline)
if err != nil {
log.Fatalf("Error running tool: %s", err)
}
}

} else {
helpMessage()
log.Fatal(1)
}
}

func exitEarlyIf(args []string) {
if len(args) == 0 { // no args, print help and exit
helpMessage()
os.Exit(0)
}

if args[0] == "help" { // if we asked for help, give the help
helpMessage()
os.Exit(0)
}

}

func helpMessage() {

log.Printf(`DBT Dynamic Binary Toolkit version: %s
Usage:
dbt [-o -v <version>] <tool> [tool args]
-v version Specify version of tool to run. (Defaults to latest)
-o offline Offline mode. Does not attempt to upgrade or find tools, just uses what's already on disk, and errors if it's not available.
Run 'dbt catalog list' to see a list of what tools are available in your repository.
`, VERSION)
cmd.Execute()
}

0 comments on commit 9a684a4

Please sign in to comment.