Skip to content

Commit

Permalink
DT-3190 [GitHub] Add support for "GitHub" credentials type in credent…
Browse files Browse the repository at this point in the history
…ials-sync (#87)

* DT-3190 [GitHub] Add support for "GitHub" credentials type in credentials-sync

Main changes:
* Added new source type "github_app"
* Updated the documentation in the README

Other changes:
* Ran go fmt and golint on the codebase
* Fixed warnings reported by the IDE
* (go.mod) Updated gojenkins library to latest released version
* (credentials.go) Better error messages when parsing of config fails
* (credentials_test.go) Added more tests to cover all valid source types

Co-authored-by: Denis Blanchette <dblanchette@coveo.com>
  • Loading branch information
ycanty and dblanchette authored Jul 21, 2020
1 parent 8dbd023 commit 4ef0e65
Show file tree
Hide file tree
Showing 9 changed files with 369 additions and 45 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,6 @@ credentials-sync
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
coverage.txt

# JetBrains products
.idea
70 changes: 57 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,16 @@
[![codecov](https://codecov.io/gh/coveooss/credentials-sync/branch/master/graph/badge.svg)](https://codecov.io/gh/coveooss/credentials-sync)
[![Go Report Card](https://goreportcard.com/badge/github.com/coveooss/credentials-sync)](https://goreportcard.com/report/github.com/coveooss/credentials-sync)

Sync credentials from various sources to various targets. It currently only supports Jenkins, but LastPass is planned because that is what we use. However, we are open to supporting more targets.
Sync credentials from various sources to various targets. Source credentials are defined and stored in standardized
formats, and are converted to the target's format upon sync.

The supported sources and targets are listed below. We are open to supporting more targets.

What's the point?
1. Easier credentials rotations. Rotating credentials manually is simply not an option when credentials rotations are done too often
2. Uses a push-model instead of a pull-model which means that you can put your credentials in a secure environment to which targets don't have access, targets may have varying degrees of security (prod vs dev)
3. Decouples your credentials and the systems which use these credentials. Standardized credentials format for all targets
1. Easier credentials rotations. Rotating credentials manually is simply not an option when credentials rotations are done too often.
2. Uses a push-model instead of a pull-model which means that you can put your credentials in a secure environment to
which targets don't have access, targets may have varying degrees of security (prod vs dev).
3. Decouples your credentials and the systems which use these credentials. Standardized credentials format for all targets.

## Installation

Expand All @@ -25,27 +29,30 @@ What's the point?
credentials-sync sync -c config.yml
```

Run without any argument for the full list of available commands.

## Logging

The log level can be set with either:
- The `--log-level` option
- The `SYNC_LOG_LEVEL` env variable

Valid levels are `debug`, `info`, `warning` and `error`
Valid levels are `debug`, `info`, `warning` and `error`.

![example](https://raw.githubusercontent.com/coveooss/credentials-sync/master/example.png)

## Configuration file
A configuration file must be given to the application. Its path can either be a local path or a S3 path
The path can either be passed as a parameter (`-c/--config`) or as an environment variable (`SYNC_CONFIG`)
The path can either be passed as a parameter (`-c/--config`) or as an environment variable (`SYNC_CONFIG`).

A configuration file contains [sources](#supported-sources) which contain [credentials](#supported-types-of-credentials). It also defines targets to which these credentials will be synced
A configuration file contains [sources](#supported-sources) which contain [credentials](#supported-types-of-source-credentials).
It also defines targets to which these credentials will be synced.

Here is the accepted format:
```yaml
sources:
local:
- path: /home/jdoe/path/to/file.yaml
- file: /home/jdoe/path/to/file.yaml
aws_s3:
- bucket: name
key: path/to/file.yaml
Expand All @@ -66,9 +73,10 @@ targets:
## Supported sources
Here are the supported sources:
- Local (Single file)
- AWS S3 (Single object)
- AWS SecretsManager (Single secret or a secret prefix)
- **local**: Local (Single file)
- **aws_s3**: AWS S3 (Single object)
- **aws_secretsmanager**: AWS SecretsManager (Single secret or a secret prefix)
The source's value must either be a list or a map in the following formats (JSON or YAML):
```yaml
Expand All @@ -91,9 +99,14 @@ my_other_cred:
...
```

## Supported types of credentials
Credentials are defined as JSON, here are the supported types of credentials with definition examples:
## Supported types of source credentials
Credentials are defined as JSON or YAML, here are the supported types of source credentials with definition examples:
- Secret text
- Username/Password
- AWS IAM
- SSH Key
- [Github App](https://developer.github.com/apps/about-apps/#about-github-apps)

```yaml
secret_text:
description: A secret text cred is only composed of a secret
Expand Down Expand Up @@ -135,6 +148,37 @@ ssh_key:
-----END RSA PRIVATE KEY-----
```
- github App credentials
```yaml
github_app:
description:
type: github_app
app_id: The github app ID. It can be found on github in the app's settings, on the General page in the About section.
private_key: |
The private key with which to authenticate to github. It must be in PKCS#8 format.
Github gives it in PKCS#1 format. Convert it to PKCS#8 with:
`openssl pkcs8 -topk8 -inform PEM -outform PEM -in current-key.pem -out new-key.pem -nocrypt`
owner: The organisation or user that this app is to be used for. Only required if this app is installed to multiple
organisations.
```
## Supported targets
Here are the supported targets:
- **jenkins**: Jenkins global credentials
### Jenkins target
The jenkins target supports the following configuration parameters:
```yaml
jenkins:
- name: Name of this target
url: URL to the Jenkins server
credentials_id: The ID of the global credential to modify in Jenkins
```
## Other features
### Unsynced credentials
Expand Down
23 changes: 14 additions & 9 deletions credentials/credentials.go
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
package credentials

import (
"errors"
"fmt"

"github.com/hashicorp/go-multierror"
Expand Down Expand Up @@ -52,10 +51,10 @@ func (credBase *Base) BaseToString() string {
// BaseValidate verifies that the credentials fields common to all types of credentials contain valid values
func (credBase *Base) BaseValidate() error {
if credBase.ID == "" {
return fmt.Errorf("Credentials (%s) has no defined ID", credBase.BaseToString())
return fmt.Errorf("credentials (%s) has no defined ID", credBase.BaseToString())
}
if credBase.CredType == "" {
return fmt.Errorf("Credentials (%s) has no type. This is probably a bug in the software", credBase.ID)
return fmt.Errorf("credentials (%s) has no type. This is probably a bug in the software", credBase.ID)
}
return nil
}
Expand Down Expand Up @@ -120,7 +119,7 @@ func (credBase *Base) ShouldSync(targetName string, targetTags map[string]string
// ParseCredentials transforms a list of maps into a list of Credentials
// The credentials type is determined by the `type` attribute
func ParseCredentials(credentialsMaps []map[string]interface{}) ([]Credentials, error) {
credentialsList := []Credentials{}
credentialsList := make([]Credentials, 0)
for _, credentialsMap := range credentialsMaps {
newCredentials, err := ParseSingleCredentials(credentialsMap)
if err != nil {
Expand All @@ -136,12 +135,13 @@ func ParseCredentials(credentialsMaps []map[string]interface{}) ([]Credentials,
func ParseSingleCredentials(credentialsMap map[string]interface{}) (Credentials, error) {
var credentialsType string
var credentials Credentials
var id = credentialsMap["id"]
if value, ok := credentialsMap["type"]; ok {
if credentialsType, ok = value.(string); !ok {
return nil, errors.New("Credentials type is not a string")
return nil, fmt.Errorf("entry %s: credentials type '%v' is not a string", id, credentialsType)
}
} else {
return nil, errors.New("Unable to find the credentials type")
return nil, fmt.Errorf("entry %s: unable to find the credentials type %s", id, credentialsType)
}

switch credentialsType {
Expand All @@ -153,10 +153,15 @@ func ParseSingleCredentials(credentialsMap map[string]interface{}) (Credentials,
credentials = NewSecretText()
case "ssh":
credentials = NewSSHCredentials()
case "github_app":
credentials = NewGithubAppCredentials()
default:
return nil, errors.New("Unknown credentials type")
return nil, fmt.Errorf("entry %s: unknown credentials type: %s", id, credentialsType)
}
err := mapstructure.Decode(credentialsMap, credentials)
if err != nil {
return nil, fmt.Errorf("entry %s: invalid credentials data: %v", id, err)
}
mapstructure.Decode(credentialsMap, credentials)
var validationErrors error
if err := credentials.BaseValidate(); err != nil {
validationErrors = multierror.Append(validationErrors, err)
Expand All @@ -165,7 +170,7 @@ func ParseSingleCredentials(credentialsMap map[string]interface{}) (Credentials,
validationErrors = multierror.Append(validationErrors, err)
}
if validationErrors != nil {
return nil, fmt.Errorf("The following credentials failed to validate: %v -> %v", credentials.ToString(false), validationErrors)
return nil, fmt.Errorf("the following credentials failed to validate: %v -> %v", credentials.ToString(false), validationErrors)
}
return credentials, nil
}
Expand Down
47 changes: 47 additions & 0 deletions credentials/credentials_github_app.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
package credentials

import (
"fmt"
)

// GithubAppCredentials represents credentials composed of an App ID, private key, and owner
type GithubAppCredentials struct {
Base `mapstructure:",squash"`
AppID int `mapstructure:"app_id"`
PrivateKey string `mapstructure:"private_key"`
Owner string `mapstructure:"owner"`
}

// NewGithubAppCredentials instantiates a GithubAppCredentials struct
func NewGithubAppCredentials() *GithubAppCredentials {
cred := &GithubAppCredentials{}
cred.CredType = "Github App"
return cred
}

// ToString prints out the content of a GithubAppCredentials struct.
func (cred *GithubAppCredentials) ToString(showSensitive bool) string {
privateKeyText := "********"
if showSensitive {
privateKeyText = cred.PrivateKey
}

appIDOwner := fmt.Sprintf("%d", cred.AppID)
if len(cred.Owner) > 0 {
appIDOwner = fmt.Sprintf("%s(%s)", appIDOwner, cred.Owner)
}
return fmt.Sprintf("%s - %s:%s", cred.BaseToString(), appIDOwner, privateKeyText)
}

// Validate verifies that the credentials is valid.
// A GithubAppCredentials must have an app id and a private key. Owner is optional.
func (cred *GithubAppCredentials) Validate() error {
switch {
case cred.AppID == 0:
return fmt.Errorf("the credentials with ID %s does not define an app ID", cred.ID)
case len(cred.PrivateKey) == 0:
return fmt.Errorf("the credentials with ID %s does not define a private key", cred.ID)
default:
return nil
}
}
70 changes: 70 additions & 0 deletions credentials/credentials_github_app_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
package credentials

import (
"testing"

"github.com/stretchr/testify/assert"
)

func TestGithubAppCredentials(t *testing.T) {
tests := map[string]struct {
givenID string
givenAppID int
givenOwner string
givenPrivate string
givenShowSensitive bool
expectString string
}{
"without owner": {givenID: "test", givenAppID: 1, givenOwner: "", givenPrivate: "private", givenShowSensitive: false, expectString: "test -> Type: Github App - 1:********"},
"with owner": {givenID: "test", givenAppID: 2, givenOwner: "owner", givenPrivate: "private", givenShowSensitive: false, expectString: "test -> Type: Github App - 2(owner):********"},
"without owner showSensitive": {givenID: "test", givenAppID: 1, givenOwner: "", givenPrivate: "private", givenShowSensitive: true, expectString: "test -> Type: Github App - 1:private"},
"with owner showSensitive": {givenID: "test", givenAppID: 2, givenOwner: "owner", givenPrivate: "private", givenShowSensitive: true, expectString: "test -> Type: Github App - 2(owner):private"},
}
for name, test := range tests {
t.Run(name, func(t *testing.T) {
cred := NewGithubAppCredentials()
cred.ID = test.givenID
cred.AppID = test.givenAppID
cred.Owner = test.givenOwner
cred.PrivateKey = test.givenPrivate
assert.Equal(t, test.expectString, cred.ToString(test.givenShowSensitive))
})
}
}

func TestGithubAppCredentialsValidation(t *testing.T) {
tests := map[string]struct {
givenCred GithubAppCredentials
expectError bool
}{
"valid": {givenCred: GithubAppCredentials{
AppID: 12345,
PrivateKey: "private",
Owner: "Me",
}, expectError: false},
"valid no owner": {givenCred: GithubAppCredentials{
AppID: 12345,
PrivateKey: "private",
}, expectError: false},
"missing app id": {givenCred: GithubAppCredentials{
PrivateKey: "private",
}, expectError: true},
"missing private key": {givenCred: GithubAppCredentials{
AppID: 12345,
}, expectError: true},
}
for name, test := range tests {
t.Run(name, func(t *testing.T) {
cred := NewGithubAppCredentials()
cred.AppID = test.givenCred.AppID
cred.PrivateKey = test.givenCred.PrivateKey
cred.Owner = test.givenCred.Owner
err := cred.Validate()
if test.expectError {
assert.Error(t, err)
} else {
assert.NoError(t, err)
}
})
}
}
Loading

0 comments on commit 4ef0e65

Please sign in to comment.