New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vault install secrets yaml #2363

Merged
merged 25 commits into from Dec 4, 2018

Conversation

Projects
None yet
5 participants
@agentgonzo
Copy link
Member

agentgonzo commented Nov 29, 2018

Submitter checklist

  • Change is code complete and matches issue description.
  • Change is covered by existing or new tests.

Description

Saving more secrets in vault.

from a jx install --gitops --vault, the following secrets get generated:
install-secrets/adminSecrets.yaml:

{
  "ChartMuseum": {
    "ChartMuseumEnv": {
      "ChartMuseumSecret": {
        "Password": "abcdefgh",
        "User": "admin"
      }
    }
  },
  "Grafana": {
    "GrafanaSecret": {
      "Password": "abcdefgh",
      "User": "admin"
    }
  },
  "IngressBasicAuth": "admin:{SHA}14GxPt7uA4zq+ekoWVkY5tmO/AU=",
  "Jenkins": {
    "JenkinsSecret": {
      "Password": "abcdefgh"
    }
  },
  "Nexus": {
    "DefaultAdminPassword": "abcdefgh"
  },
  "PipelineSecrets": {
    "MavenSettingsXML": "<settings>\n      <!-- sets the local maven repository outside of the ~/.m2 folder for easier mounting of secrets and repo -->\n      <localRepository>${user.home}/.mvnrepository</localRepository>\n      <!-- lets disable the download progress indicator that fills up logs -->\n      <interactiveMode>false</interactiveMode>\n      <mirrors>\n          <mirror>\n          <id>nexus</id>\n          <mirrorOf>external:*</mirrorOf>\n          <url>http://nexus/repository/maven-group/</url>\n          </mirror>\n      </mirrors>\n      <servers>\n          <server>\n          <id>local-nexus</id>\n          <username>admin</username>\n          <password>abcdefgh</password>\n          </server>\n      </servers>\n      <profiles>\n          <profile>\n              <id>nexus</id>\n              <properties>\n                  <altDeploymentRepository>local-nexus::default::http://nexus/repository/maven-snapshots/</altDeploymentRepository>\n                  <altReleaseDeploymentRepository>local-nexus::default::http://nexus/repository/maven-releases/</altReleaseDeploymentRepository>\n                  <altSnapshotDeploymentRepository>local-nexus::default::http://nexus/repository/maven-snapshots/</altSnapshotDeploymentRepository>\n              </properties>\n          </profile>\n          <profile>\n              <id>release</id>\n              <properties>\n                  <gpg.executable>gpg</gpg.executable>\n                  <gpg.passphrase>mysecretpassphrase</gpg.passphrase>\n              </properties>\n          </profile>\n      </profiles>\n      <activeProfiles>\n          <!--make the profile active all the time -->\n          <activeProfile>nexus</activeProfile>\n      </activeProfiles>\n  </settings>\n"
  }
}

install-secrets/gitSecrets.yaml:

{
  "PipelineSecrets": {
    "GitCreds": "https://jenkins-x-bot-test:<REDACTED>@github.com\nhttp://jenkins-x-bot-test:<REDACTED>@github.com"
  }
}

At the moment, this is basically taking these two yaml files (that were stored on disk) and storing them in vault.
Whilst the end-goal would be to split all these yaml files into separate actual secrets (ie, password, bot token etc) that can be done as a separate PR. Storing the YAML in vault rather than the file-system is a stepping-stone on our way to the final solution. Otherwise this PR would be enormous and boil the ocean

Special notes for the reviewer(s)

Summary of changes:

  • Installs the vault cli for the user if not installed already
  • Extracted a VaultClient interface rather than using api.Client
  • Secrets created with --gitops in the jenkins-x-dev-enviromnent will be stored in vault (under /secrets/install-secrets/) as separate secrets (which are still yaml)
  • When doing jx step helm apply, if the secrets.yaml file does not exist, it will be populated from the above stored secrets. After helm is applied, it will securely delete the files
  • secrets are now (more) securely deleted from the filesystem instead of just unlinking them.

Which issue this PR fixes

fixes #

@agentgonzo agentgonzo requested a review from ccojocar Nov 29, 2018

@jenkins-x-bot jenkins-x-bot requested review from i0n and rajdavies Nov 29, 2018

Show resolved Hide resolved pkg/vault/vault_client.go Outdated
Show resolved Hide resolved pkg/vault/vault_client.go Outdated
Show resolved Hide resolved pkg/vault/vault_client.go Outdated
"gopkg.in/yaml.v2"
)

type VaultClient interface {

This comment has been minimized.

@houndci-bot

houndci-bot Nov 29, 2018

Collaborator

exported type VaultClient should have comment or be unexported
type name will be used as vault.VaultClient by other packages, and that stutters; consider calling this Client

Show resolved Hide resolved pkg/vault/constants.go Outdated
Show resolved Hide resolved pkg/util/downloads.go
Show resolved Hide resolved pkg/util/downloads.go
Show resolved Hide resolved pkg/util/downloads.go
Show resolved Hide resolved pkg/util/downloads.go
Show resolved Hide resolved pkg/jx/cmd/install.go Outdated
Show resolved Hide resolved pkg/jx/cmd/install.go Outdated
if err != nil {
return errors.Wrapf(err, "failed to write %s", gitIgnore)
}

readme := filepath.Join(gitOpsDir, "README.md")
err = ioutil.WriteFile(readme, []byte(devGitOpsReadMe), util.DefaultWritePermissions)
err = secretStore.Write(readme, []byte(devGitOpsReadMe))

This comment has been minimized.

@ccojocar

ccojocar Nov 29, 2018

Member

also this does not seem a secret

Show resolved Hide resolved pkg/jx/cmd/install.go Outdated
Show resolved Hide resolved pkg/jx/cmd/install.go Outdated
Show resolved Hide resolved pkg/jx/cmd/install.go Outdated
Show resolved Hide resolved pkg/jx/cmd/storage/secret_store.go Outdated
Show resolved Hide resolved pkg/jx/cmd/storage/secret_store.go Outdated
Show resolved Hide resolved pkg/util/structs.go Outdated
Show resolved Hide resolved pkg/util/structs.go
Show resolved Hide resolved pkg/vault/vault_client.go Outdated

@ccojocar ccojocar requested review from jstrachan and removed request for i0n and rajdavies Nov 29, 2018

@agentgonzo

This comment has been minimized.

Copy link
Member

agentgonzo commented Nov 30, 2018

/retest

1 similar comment
@agentgonzo

This comment has been minimized.

Copy link
Member

agentgonzo commented Nov 30, 2018

/retest

@ccojocar

This comment has been minimized.

Copy link
Member

ccojocar commented Nov 30, 2018

/hold

1 similar comment
@ccojocar

This comment has been minimized.

Copy link
Member

ccojocar commented Nov 30, 2018

/hold

@ccojocar ccojocar force-pushed the agentgonzo:vault-install-secrets-yaml branch from d5bbc06 to f6cf75c Dec 3, 2018

@ccojocar

This comment has been minimized.

Copy link
Member

ccojocar commented Dec 3, 2018

/hold cancel

1 similar comment
@ccojocar

This comment has been minimized.

Copy link
Member

ccojocar commented Dec 4, 2018

/hold cancel

@ccojocar

This comment has been minimized.

Copy link
Member

ccojocar commented Dec 4, 2018

/test bdd

@agentgonzo

This comment has been minimized.

Copy link
Member

agentgonzo commented Dec 4, 2018

/hold

Show resolved Hide resolved pkg/io/config_store.go Outdated
@agentgonzo

This comment has been minimized.

Copy link
Member

agentgonzo commented Dec 4, 2018

/hold cancel

@ccojocar

This comment has been minimized.

Copy link
Member

ccojocar commented Dec 4, 2018

/hold cancel

@ccojocar

This comment has been minimized.

Copy link
Member

ccojocar commented Dec 4, 2018

/lgtm
/retest

@jenkins-x-bot jenkins-x-bot added the lgtm label Dec 4, 2018

@jenkins-x-bot

This comment has been minimized.

Copy link
Contributor

jenkins-x-bot commented Dec 4, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ccojocar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ccojocar

This comment has been minimized.

Copy link
Member

ccojocar commented Dec 4, 2018

/test bdd

1 similar comment
@rawlingsj

This comment has been minimized.

Copy link
Member

rawlingsj commented Dec 4, 2018

/test bdd

@jenkins-x-bot jenkins-x-bot merged commit 851cb3e into jenkins-x:master Dec 4, 2018

3 checks passed

Hound No violations found. Woof!
serverless-jenkins succeeded
tide In merge pool.
Details

@agentgonzo agentgonzo deleted the agentgonzo:vault-install-secrets-yaml branch Dec 7, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment