Update kubeadm docs for 1.6 #2829

jbeda · 2017-03-15T04:42:31Z

It is also built on top of #2779 so ignore that commit.

This change is

deads2k · 2017-03-15T11:43:43Z

docs/admin/accessing-the-api.md

@@ -42,7 +42,7 @@ The input to the authentication step is the entire HTTP request, however, it typ
 just examines the headers and/or client certificate.

 Authentication modules include Client Certificates, Password, and Plain Tokens,
-and JWT Tokens (used for service accounts).
+Bootstrap Tokens and JWT Tokens (used for service accounts).


oxford comma

Fixed in #2779.

deads2k · 2017-03-15T11:45:17Z

docs/admin/authentication.md

+as a "Bootstrap Token".
+
+These tokens are stored as Secrets in the `kube-system` namespace.  As such they
+can be dynamically created and managed.  There is a controller (BootstrapSigner)


I thought it was the tokencleaner.

Good catch! Fixed in #2779

luxas

I'd still like a section at the bottom describing what kind of stability we have on different components in kubeadm:

CLI UX: beta
Config file: alpha
Selfhosting: alpha
The phases command: alpha
Implementation under the hood: alpha

Due to that we're targeting upgrades/HA in future versions and the fact we're using alpha features in core (the bootstrapsigner/tokencleaner and tokenauthmodule)

luxas · 2017-03-15T06:48:37Z

docs/getting-started-guides/kubeadm.md

-Be sure to read the [limitations](#limitations); in particular note that kubeadm doesn't have great support for
-automatically configuring cloud providers. Please refer to the specific cloud provider documentation or
-use another provisioning system.**
+**The kubeadm tool is currently in beta but please try it out and give us


The kubeadm command-line UX is in beta, but please...

We should mention the implementation somehow might change yet (especially when we're targeting HA it will change). And somehow we should point out that the kubeadm alpha subcommands are unstable and that we're using alpha features in core (BS, TC and BootstrapTokenAuth)

That could be a separate section later in the doc indeed with a link to it here

Created a sub-section of "Overview" with more details.

luxas · 2017-03-15T06:50:42Z

docs/getting-started-guides/kubeadm.md


-If you are not constrained, there are some other tools built to give you complete clusters:
+If you are not constrained, there are some other tools built to give you


some other more higher-level tools that aim to give you a complete cluster:

luxas · 2017-03-15T06:51:34Z

docs/getting-started-guides/kubeadm.md

+1. 1GB or more of RAM per machine (any less will leave little room for your
+   apps)
+1. Full network connectivity between all machines in the cluster (public or
+   private network is fine)

 ## Objectives

 * Install a secure Kubernetes cluster on your machines


Are we confident in the secure here? :) I know it wasn't changed here but...

Making it clear that we "aim to create a secure cluster"

Where did you make this change? Can't see it in the diff

luxas · 2017-03-15T06:55:59Z

docs/getting-started-guides/kubeadm.md

-  It runs on all of the machines in your cluster and does things like starting pods and containers.
-* `kubectl`: the command to control the cluster once it's running.
-  You will only need this on the master, but it can be useful to have on the other nodes as well.
+* `docker`: the container runtime, which Kubernetes depends on. v1.12 is


I guess v1.11.2 is still recommended or?

I think that v1.12 is supported so i'm going to list that.

luxas · 2017-03-15T06:57:02Z

docs/getting-started-guides/kubeadm.md

-See the reference doc if you want to read about the different [kubeadm releases](https://github.com/kubernetes/kubeadm/blob/master/CHANGELOG.md)
+**Note:** If you already have kubeadm installed, you should do a `apt-get update &&
+apt-get upgrade` or `yum update` to get the latest version of kubeadm. See the
+reference doc if you want to read about the different [kubeadm


See the kubeadm-specific Release Notes if you want to know more about what has changed between releases

luxas · 2017-03-15T07:34:22Z

docs/getting-started-guides/kubeadm.md


+``` bash
+kubectl describe svc front-end -n sock-shop


a simple kubectl -n sock-shop get svc front-end is sufficent here

BTW -- sock shop demo outputs errors like:

the namespace from the provided object "zipkin" does not match the namespace "sock-shop". You must pass '--namespace=zipkin' to perform this operation.

Might want to pass that on to weave folks.

luxas · 2017-03-15T07:38:09Z

docs/getting-started-guides/kubeadm.md


-### (Optional) Installing a sample application
+Currently, only the pod network flannel is working on multiple architectures.


You can remove the flannel install statement here now and just say that flannel works on amd64, arm, arm64 and ppc64le and weave works on amd64, arm, arm64

luxas · 2017-03-15T17:58:45Z

docs/getting-started-guides/kubeadm.md

+`kubeadm.conf` with the following contents:
+
+``` yaml
+cloudProvider: <cloud provider>


please make this a full example:

kind: MasterConfiguration apiVersion: kubeadm.k8s.io/v1alpha1 cloudProvider: <cloud provider>

and point out that this part (the config file) of kubeadm is alpha

luxas · 2017-03-15T17:59:44Z

docs/getting-started-guides/kubeadm.md


-Currently, only the pod network flannel is working on multiple architectures. You can install it this way:
+You may have trouble in the configuration if you see the following statuses.
+This example is for canal but there may be similar errors for other pod network


Can we have something more generic to not favor anyone?

I just moved this down to make the main flow better. I'm open to reviewing a PR that changes the example. For now I want to get this in without having to test out a bunch of stuff by forcing some errors.

luxas · 2017-03-15T18:00:05Z

docs/getting-started-guides/kubeadm.md

-1. The cluster created here has a single master, with a single `etcd` database running on it.
-   This means that if the master fails, your cluster loses its configuration data and will need to be recreated from scratch.
-   Adding HA support (multiple `etcd` servers, multiple API servers, etc) to `kubeadm` is still a work-in-progress.
+The ```kubectl describe``` comand gives you more details about what went wrong.


only one ` is sufficent

Yeah -- someone added that in there and I thought I got them all.

luxas · 2017-03-15T18:42:51Z

Yeah, just noting that some of my comments here weren't direct comments to @jbeda's changes, but improvements that we could make at the same time we're really reading through and trying to make the docs better.

devin-donnelly · 2017-03-15T23:10:20Z

@jbeda This PR modifies several of the same files as #2779. I think that's going to cause merge conflicts. I've already commented on the changes there; perhaps those files should be backed out of this PR?

jbeda · 2017-03-16T00:35:23Z

@devin-donnelly I based this on that so that I could review them the same. As I manage the rebase I'm keeping things clean and making sure I can resolve. Just look at the last two commits in the set. (actually, just the last one should matter).

jbeda · 2017-03-16T00:46:40Z

@luxas Comments addressed. PTAL

@devin-donnelly This merges in the latest changes in the other PR.

I'm removing the WIP mark as I think this is ready for review.

devin-donnelly · 2017-03-16T04:02:47Z

docs/admin/bootstrap-tokens.md

+information filled out.  The key thing being communicated here is the
+`certificate-authority-data`.  This may be expanded in the future.
+
+The signature is a JWS signature where using a "detached" mode.  To validate the


I think the "where" in this sentence is superfluous.

devin-donnelly · 2017-03-16T04:02:57Z

docs/admin/bootstrap-tokens.md

+`certificate-authority-data`.  This may be expanded in the future.
+
+The signature is a JWS signature where using a "detached" mode.  To validate the
+signature, the user should encode the `kubeconfig` payload according to JWS


the user -> you

devin-donnelly · 2017-03-16T04:03:17Z

docs/admin/bootstrap-tokens.md

+The signature is a JWS signature where using a "detached" mode.  To validate the
+signature, the user should encode the `kubeconfig` payload according to JWS
+rules (base64 encoded while discarding any trailing `=`).  That encoded payload
+is then used to form a whole JWS by inserting it between the 2 dots.  The JWS


"You can verify the JWS..."

devin-donnelly · 2017-03-16T04:03:29Z

docs/admin/bootstrap-tokens.md

+rules (base64 encoded while discarding any trailing `=`).  That encoded payload
+is then used to form a whole JWS by inserting it between the 2 dots.  The JWS
+can be verified using the `HS256` scheme (HMAC-SHA256) with the full token (e.g.
+`07401b.f395accd246ae52d`) as the shared secret.  Users MUST verify that HS256


Users MUST -> You must

Changed but... All caps MUST is a language from RFCs to say that this should not be ignored. People used to implementing protocols know that this is something that you shouldn't ignore.

devin-donnelly · 2017-03-16T04:04:48Z

docs/getting-started-guides/kubeadm.md

+The kubelet is now restarting every few seconds, as it waits in a crashloop for
+kubeadm to tell it what to do.
+
+Note: Disabling SELinux by running `setenforce 0` is required in order to allow


"Note: You must disable SELinux to allow containers to access the host filesystem..."

devin-donnelly · 2017-03-16T04:07:06Z

docs/getting-started-guides/kubeadm.md

+| gcr.io/google_containers/kube-discovery-amd64 | 1.0
+| gcr.io/google_containers/pause-amd64 | 3.0
+
+Right now you can't run `kubeadm init` twice without tearing down the cluster in


Remove "right now" unless we can tell the user something about when this might change.

devin-donnelly · 2017-03-16T04:07:55Z

docs/getting-started-guides/kubeadm.md

-
-A few seconds later, you should notice that running `kubectl get nodes` on the master shows a cluster with as many machines as you created.
+The nodes are where your workloads (containers and pods, etc) run. If you want
+to add any new machines as nodes to your cluster, for each machine: SSH to that


"To add any new machines to your cluster as nodes, you must do the following for each machine"

Reformat the instructions as a numbered list.

devin-donnelly · 2017-03-16T04:08:31Z

docs/getting-started-guides/kubeadm.md


-If there is a firewall, make sure it exposes this port to the internet before you try to access it.
+It takes several minutes to download and start all the containers, watch the


Replace comma with period. Start the resulting second sentence with:

"You can watch the output of..."

devin-donnelly · 2017-03-16T04:09:07Z

docs/getting-started-guides/kubeadm.md


-## Tear down
+Then go to the IP address of your cluster's master node in your browser, and


Comma after "Then".

devin-donnelly · 2017-03-16T04:10:22Z

docs/getting-started-guides/kubeadm.md

-
-This workflow is not yet fully supported, however we hope to make it extremely easy to spin up clusters with cloud providers in the future. 
-(See [this proposal](https://github.com/kubernetes/community/pull/128) for more information) The [Kubelet Dynamic Settings](https://github.com/kubernetes/kubernetes/pull/29459) feature may also help to fully automate this process in the future.
+Enabling specific cloud providers is a common request, this currently requires


Replace comma with period. Second sentence becomes: "However, specific cloud providers require manual configuration and are therefore not supported by kubeadm in version 1.6."

lukemarsden

Great work @jbeda! Please address my comments as you see fit and merge at will. No need for me to take another look.

lukemarsden · 2017-03-17T17:12:38Z

docs/admin/authentication.md

+controller that deletes bootstrap tokens as they expire.
+
+The tokens are of the form `[a-z0-9]{6}.[a-z0-9]{16}`.  The first component is a
+Token ID and the second component is the Token Secret.  ThYou specify the token


ThYou -> You

lukemarsden · 2017-03-17T17:13:06Z

docs/admin/authentication.md

+Authorization: Bearer 781292.db7bc3a58fc5f07e
+```
+
+You must enable the Bootstrap Token Authenticator with the


When must you do this? Does kubeadm do this for you?

To be clear, these docs are written not assuming kubeadm as this can be used by other systems that aren't kubeadm. I'd rather have real docs vs. referring folks to a design doc that may be a little raw.

lukemarsden · 2017-03-17T17:15:00Z

docs/admin/bootstrap-tokens.md

+
+Bootstrap tokens are a simple bearer token that is meant to be used when
+creating new clusters or joining new nodes to an existing cluster.  It was built
+to support [`kubeadm`](/docs/admin/kubeadm/), but can be used in other contexts.


Can you give an example of when they're used in kubeadm?

And an example of how you might expect users to use them in other contexts?

lukemarsden · 2017-03-17T17:16:28Z

docs/admin/accessing-the-api.md

@@ -42,7 +42,7 @@ The input to the authentication step is the entire HTTP request, however, it typ
 just examines the headers and/or client certificate.

 Authentication modules include Client Certificates, Password, and Plain Tokens,
-and JWT Tokens (used for service accounts).
+Bootstrap Tokens, and JWT Tokens (used for service accounts).


What's the difference between Bootstrap Tokens and JWT Tokens? Are JWT tokens the old separate-to-API-server version, whereas Bootstrap Tokens are the new built-in thing? If we're no longer supporting "JWT Tokens" can we just remove the reference to them here?

Bootstrap tokens don't use JWT at all -- they are just random values. The JWT tokens actually are encoded and signed tokens used for authentication. We do use JWS for doing the configmap signing, but that is unrelated and not part of the authentication flow.

lukemarsden · 2017-03-17T17:16:54Z

docs/admin/authentication.md

+
+This feature is currently in **alpha**.
+
+o allow for streamlined bootstrapping for new clusters, Kubernetes includes a


lukemarsden · 2017-03-17T18:08:00Z

docs/getting-started-guides/kubeadm.md


 ``` bash
 scp root@<master ip>:/etc/kubernetes/admin.conf .
 kubectl --kubeconfig ./admin.conf get nodes
 ```

+**Note:** By default GCE instances disable ssh access for root.  First log in to


First -> If you are using GCE, first

lukemarsden · 2017-03-17T18:09:23Z

docs/getting-started-guides/kubeadm.md

+the machine, copy the file someplace that can be accessed and then use [`gcloud
+compute
+copy-files`](https://cloud.google.com/sdk/gcloud/reference/compute/copy-files)
+
 ### (Optional) Connecting to the API Server

 If you want to connect to the API Server for viewing the dashboard (note: the


let's just get rid of the reference to dashboard here, if we're not going to describe how to install it

"If you want to connect to the API Server from outside the cluster you can use"

lukemarsden · 2017-03-17T18:10:13Z

docs/getting-started-guides/kubeadm.md

@@ -428,23 +415,34 @@ running.

 Then go to the IP address of your cluster's master node in your browser, and
 specify the given port. So for example, `http://<master_ip>:<port>`. In the
-example above, this was `30001`, but it is a different port for you.
+example above, this was `30001`, but it may be a different port for you.


actually, it'll always be 30001 now, so we can reword this accordingly

Not if you've installed other services....

lukemarsden · 2017-03-17T18:10:52Z

docs/getting-started-guides/kubeadm.md

-* To undo what  kubeadm did, simply run:
+To undo what kubeadm did, you should first [drain the
+node](https://kubernetes.io/docs/user-guide/kubectl/kubectl_drain/) and make
+sure that the node is empty before turning it down.


"turning it down" -> "shutting it down" (we're not all {ex-,}googlers 😉)

lukemarsden · 2017-03-17T18:11:43Z

docs/getting-started-guides/kubeadm.md

+kubeadm reset
+```
+
+If you wish to start over simply run `kubeadm init` or `kubeadm join`.


with the appropriate arguments as described above

* Word wrap paragraphs * Switch to fenced code blocks for all code samples * Render programs as plain text * Separate out program invocation from program output Much of this is to follow the style guide: https://kubernetes.io/docs/contribute/style-guide/ Signed-off-by: Joe Beda <joe.github@bedafamily.com>

jbeda · 2017-03-18T00:13:41Z

Pushed new changes and separated out the stuff from #2779. It turns out building these on top of each other just confused all reviewers.

luxas

One review round on the docs/admin/kubeadm.md file

luxas · 2017-03-19T08:27:37Z

docs/admin/kubeadm.md

-1. kubeadm generates a self-signed CA using openssl to provision identities
-for each node in the cluster, and for the API server to secure communication
-with clients.
+1. kubeadm generates a self-signed CA using openssl to provision identities for


It's not using openssl and it's generating much more than only a CA and a serving cert

Ref: kubernetes/kubeadm#156 if you want to describe more in detail about what kubeadm does

I'm just reformatting here but happy to update. Once that PR goes in we can (a) turn it in to user facing docs and (b) reference it from here.

luxas · 2017-03-19T08:28:18Z

docs/admin/kubeadm.md


 1. Outputting a kubeconfig file for the kubelet to use to connect to the API
-server, as well as an additional kubeconfig file for administration.
+   server, as well as an additional kubeconfig file for administration.

 1. kubeadm generates Kubernetes resource manifests for the API server,


generates Static Pod manifests for the API Server...

luxas · 2017-03-19T08:32:32Z

docs/admin/kubeadm.md

+   `/etc/kubernetes/manifests`. The kubelet watches this directory for static
+   resources to create on startup. These are the core components of Kubernetes,
+   and once they are up and running we can use `kubectl` to set up or manage any
+   additional components.



in between here kubeadm taints the master unschedulable, sets up RBAC, creates the cluster-info configmap for the token discovery and deploys both kube-proxy and kube-dns

luxas · 2017-03-19T08:32:54Z

docs/admin/kubeadm.md


-1. Use the token to talk to the API server and securely get the root CA
-certificate.
+1. Download root CA infromation from the API server.  Use the token to verify


information

luxas · 2017-03-19T08:34:45Z

docs/admin/kubeadm.md

+- `--kubernetes-version` (default 'latest') the kubernetes version to initialise
+
+The **v1.6** version of kubeadm only supports building clusters that are at
+least **v1.6.0**.  This requirement is due to kubeadm's use of RBAC. With this


the Certificates API, Token Auth module, Token controllers, etc.
There are many reasons

luxas · 2017-03-19T08:37:50Z

docs/admin/kubeadm.md

-WARNING: kubeadm is in alpha and the configuration API syntax will likely change before GA.
+WARNING: While kubeadm command line interface is in beta, the config file is
+still considered alpha.  We are aiming to not break any scripted use of the main
+`kubeadm init` and `kubeadm join` flow.


but the config file UX may still change in non-backwards-compatible ways

luxas · 2017-03-19T08:39:58Z

docs/admin/kubeadm.md

+  - <address|string>
+
+tlsBootstrapToken: <string>
+token: <string>


I would leave token invisible here, if the config file is used, one should really specify the separate tokens.
token is more of a CLI-convenience kind of thing

Gah -- then it should be there. But I'll remove it.

luxas · 2017-03-19T08:40:41Z

docs/admin/kubeadm.md

+a new token.  This token is of the correct form for specifying with the
+`--token` argument to `kubeadm init`.
+
+For the gory details on how the tokens are implemented (including manaing them


luxas · 2017-03-19T08:41:18Z

docs/admin/kubeadm.md


-1.  Generate a token.  This token must have the form  `<6 character string>.<16 character string>`.
+1.  Generate a token.  This token must have the form  `<6 character string>.<16


use of regex instead of . is preferred

luxas · 2017-03-19T08:42:18Z

docs/admin/kubeadm.md

 | `KUBE_HOST_ETCD_PATH` | `/var/lib/etcd` | Local etcd state for Kubernetes cluster |
-| `KUBE_HYPERKUBE_IMAGE` | `` | If set, use a single hyperkube image with this name. If not set, individual images per server component will be used. |
-| `KUBE_DISCOVERY_IMAGE` | `gcr.io/google_containers/kube-discovery-<arch>:1.0` | The bootstrap discovery helper image to use. |
+| `KUBE_HYPERKUBE_IMAGE` | | If set, use a single hyperkube image with this name. If not set, individual images per server component will be used. |
 | `KUBE_ETCD_IMAGE` | `gcr.io/google_containers/etcd-<arch>:2.2.5` | The etcd container image to use. |


Signed-off-by: Joe Beda <joe.github@bedafamily.com>

jbeda · 2017-03-19T17:05:19Z

Addressed all of @luxas's comments. PTAL.

GheRivero · 2017-03-20T07:16:05Z

docs/admin/kubeadm.md

+and handed out to each node with the `--pod-network-cidr` flag. This should be a
+minimum of a /16 so controller-manager is able to assign /24 subnets to each
+node in the cluster. If you are using flannel with [this
+manifest](https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml)


shouldn't we use this the manifest that includes rbac? https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel-rbac.yml

devin-donnelly · 2017-03-20T18:23:14Z

Just waiting on @luxas 's LGTM.

luxas

LGTM

devin-donnelly · 2017-03-20T21:27:53Z

Great, thanks.

jbeda added the do-not-merge DEPRECATED. Indicates that a PR should not merge. Label can only be manually applied/removed. label Mar 15, 2017

jbeda added this to the 1.6 milestone Mar 15, 2017

jbeda requested review from lukemarsden and luxas March 15, 2017 04:42

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 15, 2017

k8s-github-robot assigned deads2k Mar 15, 2017

deads2k reviewed Mar 15, 2017

View reviewed changes

deads2k assigned luxas and unassigned deads2k Mar 15, 2017

jbeda force-pushed the kubeadm-update branch from 2e56b78 to df6dc97 Compare March 15, 2017 15:52

jbeda mentioned this pull request Mar 15, 2017

Dramatically Simplify Kubernetes Cluster Creation (kubeadm umbrella issue). kubernetes/enhancements#11

Closed

18 tasks

luxas suggested changes Mar 15, 2017

View reviewed changes

devin-donnelly added the Tech Review: Open Issues label Mar 15, 2017

jbeda force-pushed the kubeadm-update branch from df6dc97 to 100b0c2 Compare March 16, 2017 00:34

jbeda force-pushed the kubeadm-update branch from 100b0c2 to 41db549 Compare March 16, 2017 00:45

jbeda removed the do-not-merge DEPRECATED. Indicates that a PR should not merge. Label can only be manually applied/removed. label Mar 16, 2017

devin-donnelly suggested changes Mar 16, 2017

View reviewed changes

lukemarsden approved these changes Mar 17, 2017

View reviewed changes

jbeda force-pushed the kubeadm-update branch from 41db549 to 624d327 Compare March 18, 2017 00:12

jbeda mentioned this pull request Mar 18, 2017

Fixed kubeadm taint command for master node #2794

Closed

luxas reviewed Mar 19, 2017

View reviewed changes

Update kubeadm guide and reference pages for 1.6

28c1fc5

Signed-off-by: Joe Beda <joe.github@bedafamily.com>

jbeda force-pushed the kubeadm-update branch from 624d327 to 28c1fc5 Compare March 19, 2017 17:04

GheRivero reviewed Mar 20, 2017

View reviewed changes

devin-donnelly approved these changes Mar 20, 2017

View reviewed changes

devin-donnelly added the Docs LGTM label Mar 20, 2017

luxas approved these changes Mar 20, 2017

View reviewed changes

devin-donnelly added Tech Review LGTM and removed Tech Review: Open Issues labels Mar 20, 2017

devin-donnelly merged commit ab6166b into kubernetes:release-1.6 Mar 20, 2017


		If you are not constrained, there are some other tools built to give you complete clusters:
		If you are not constrained, there are some other tools built to give you


		### (Optional) Installing a sample application
		Currently, only the pod network flannel is working on multiple architectures.


		If there is a firewall, make sure it exposes this port to the internet before you try to access it.
		It takes several minutes to download and start all the containers, watch the


		## Tear down
		Then go to the IP address of your cluster's master node in your browser, and


		This feature is currently in alpha.

		o allow for streamlined bootstrapping for new clusters, Kubernetes includes a


		1. Generate a token. This token must have the form `<6 character string>.<16 character string>`.
		1. Generate a token. This token must have the form `<6 character string>.<16

Update kubeadm docs for 1.6 #2829

Update kubeadm docs for 1.6 #2829

Conversation

jbeda commented Mar 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deads2k Mar 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luxas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luxas commented Mar 15, 2017

devin-donnelly commented Mar 15, 2017

jbeda commented Mar 16, 2017

jbeda commented Mar 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukemarsden left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbeda commented Mar 18, 2017

luxas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbeda commented Mar 19, 2017

Choose a reason for hiding this comment

devin-donnelly commented Mar 20, 2017

luxas left a comment

jbeda commented Mar 15, 2017 •

edited

Loading

deads2k Mar 15, 2017 •

edited

Loading

lukemarsden left a comment •

edited

Loading