Cheat Sheet

[osterman]

what

• Document a cheat sheet for things like where we are storing secrets and how to change them, how to scale up and down or tweak auto-scaling, where our dashboards are, what the various repos you created are for

(requested by @Nuru from PopChest)

[Nuru]

What software are we using for devops and what licenses are they licensed under? This list will be required for a future buyer for sure and possibly for future investors. If there is anything licensed under AGPL, how can we replace it with something with a more favorable license?

How would we bring up a new cluster in another region? What existing files/repos would we clone or modify?

[osterman]

As far as I know, nothing is AGPL. I know this is a toxic license. We try to use mostly APACHE2 or MIT licensed software. Everything Cloud Posse develops is APACHE2, with one exception wp-quip is GNU (due to wordpress requirements).

This is a good question though. We should make sure to call out the software licensing on all tools in the solution.

[Nuru]

While all the questions below should also have generic answers in the generic documentation (e.g. add a metric by adding some piece of info to some configuration file), the cheat sheet should have the specifics (e.g. name and location of configuration file to change, and if it is not in GitHub, how to change it and maintain version history, and how to deploy changes).

How do we manage access control to devops related systems (both in staging and in production):

• Ability to view metrics/graphs/reports
• Ability to add/remove servers or otherwise modify AWS resource provisioning
• Ability to view or change secrets?
• How do we audit access and changes to secrets?
• How do we add custom metrics to graphs?
• How do we add resource scaling triggers based on custom metrics?

[osterman]

@goruha can you add some notes, links, references in this issue to unblock @Nuru. I'll refine them in the documentation, but we can start it here. Specifically, address this question first as it relates to grafana: #20 (comment)

[osterman]

Ability to view or change secrets

@Nuru we need to upgrade the clusters ASAP to use our current "best practice" that we've already rolled out with Joany and PeerStreet. Namely, we've switched to using AWS Parameter Store with KMS encryption. This is all automated using chamber by Segment.io. The way it works at PopChest is using "env files" stored in an encrypted S3 bucket. I don't think it makes sense to document this as it should simply be replaced.

The benefits with chamber are that you can easily view all the parameters in the AWS web console, provided you have permissions.

[osterman]

Ability to add/remove servers or otherwise modify AWS resource provisioning

I'll take a stab at this one today.

[Nuru]

@osterman I'm in favor of using AWS Parameter Store and KMS, and it is fine to switch to that before/instead of documenting the env files (except please note where the env files are/were so we can track whatever secrets were in them should we later discover the files were disclosed) as long as this does not break local/dev/CI environments.

I'm a little wary about using chamber because it does not have a long history of widespread support and adoption (by my definition of those terms), but as long as it is just a convenience and we can manage without it, it is OK to use it if you like.

[osterman]

@Nuru thanks for taking time to explain your position on this. Let me just expand on why we've chosen to back `chamber.

I'm in favor of using AWS Parameter Store and KMS

Great! My gripe with using the envfiles is that it was a snowflake implementation (as are all envfile implementations I've seen). The challenge is how to keep those files secure as a system of record. Mozilla has a project called sops, but it's gnarly by comparison to chamber to implement. Then there's multiple extensions to git which will help encrypt those files using precommit hooks. All in all, didn't like the solutions.

Chamber is a convenience utility. And like you say, it's possible to manage the secrets themselves using the AWS Web Console directly. My experience with AWS is that they make great APIs and rather poor command line tools. That's why there's a rich ecosystem of tools that support it.

Segment.io as a company has raised $108M and been around since 2011. Our philosophy is to cherry pick the best tools from the community to facilitate integration. Most of the tools are by startups because they are the earliest adopters. Segment.io's Chamber meets the cloudposse criteria for this.

There are 600+ stars on the repo. It has an MIT license and regular releases for the past year.

Unfortunately, as with the entire kubernetes ecosystem, it is moving very rapidly. The criteria for widespread support needs to be adjusted relative to the maturity of the community. I have not been able to find any more mature tools that expose secrets stored in AWS SSM (or their new Secrets offering) as environment variables.

Alternatives to consider would be HashiCorp's Vault. This is a tool with widespread support and in use by many major financial institutions. HashiCorp Vault coupled with envconsul would also satisfy our current use-case, but Vault would need to be deployed as a separate cluster at an additional cost and management overhead.

[Nuru]

@osterman The examples above were not meant to be a bullet list of specific documentation requirements as much as a set of examples meant to be broadly covered by the cheat sheet. The cheat sheet can say

• Grafana documentation is <here>
• Where the documentation refers to <this logical file or URL>, use <this actual explicit file or URL> for PopChest staging and use <this actual explicit file or URL> for PopChest production.

and that might be all we need. To you it is probably obvious where everything is but to people with no experience with any of the DevOps tools you installed (which is, I think, our entire team), it is not even clear where to start.

[osterman]

The cheat sheet can say . ... and that might be all we need.

Cool, yes, we'll start with just documenting a roadmap of links with light descriptions of how the pieces fit together. I'll be writing up a few more docs this weekend.

We've also officially relaunched the documentation portal now on Hugo (https://docs.cloudposse.com) with more or less equivalent functionality, but the added benefit of tight github integration for issues.

[osterman]

This PR will serve as the "cheat sheet" (#108). It's more of a getting started guide, but tries to cover all pertinent pages in paragraph form with links out to all the pages. The goal is to communicate how all the pieces fit together.