Added image-policy proposal #27129

erictune · 2016-06-09T16:04:51Z

Add proposal for image policy.

erictune · 2016-06-09T16:05:47Z

erictune · 2016-06-09T16:08:34Z

@philips it might make sense to have https://github.com/coreos/clair be a backend for this image policy webhook, or something like that? Can you at-mention the right people from Clair?

Also @ericchiang because it is a type of authorization, but intentionally not in RBAC.

erictune · 2016-06-09T16:15:31Z

@roberthbailey

ericchiang · 2016-06-09T16:52:09Z

docs/proposals/image-provenance.md

+- For non-replicated things (size 1 ReplicaSet, PetSet), a single node failure may disable it.
+- a node rolling update will eventually check for liveness of replacements, and would be throttled if
+  in the case when the image was no longer allowed and so replacements could not be started.
+- rapid node restarts will cause existing pod objects to be restared by kubelet.


s/restared/restarted/

ericchiang · 2016-06-09T16:52:12Z

docs/proposals/image-provenance.md

+* Implementing objects in core kubernetes which describe complete policies for what images are approved.
+  * A third-party implementation of an image policy checker could optionally use ThirdPartyResource to store its policy.
+* Kubernetes core code dealing with concepts of image layers, build processes, source repositories, etc.
+  * We expect there will be multiple PaaSes and/or de-facto programming enviroments, each with different takes on


s/enviroments/environments/

erictune · 2016-06-09T17:41:02Z

@jerbia

ericchiang · 2016-06-09T17:55:27Z

cc @gtank @ecordell

smarterclayton · 2016-06-09T18:08:12Z

docs/proposals/image-provenance.md

+ }
+ 
+// ImageReviewSpec is a description of the token authentication request.
+type ImageReviewSpec struct {


Do we need to know the authenticated user / service account that is attempting to perform this action? Who can matter as much as what in this case.

Nm, saw the comment below. Commented there.

smarterclayton · 2016-06-09T18:09:13Z

docs/proposals/image-provenance.md

+	// Error should be empty unless Allowed is false in which case it
+        // may contain a short description of what is wrong.  Kubernetes
+        // may truncate excessively long errors when displaying to the user.
+	Error string


Shouldn't this be Reason and Message just like other status responses?

smarterclayton · 2016-06-09T18:11:14Z

docs/proposals/image-provenance.md

+
+The ReplicaSet, or other controller, is responsible for recognizing when a 403 has happened
+(whether due to user not having permission due to bad image, or some other permission reason)
+and throttling itself and surfacing the error in a way that CLIs and UIs can show to the user.


I'm starting to feel like we're not doing enough to actually solve the usability problem here - the approach you describe is correct (RS needs to surface that info) but we're doing a terrible job of it. For instance... should RS have a condition "CreationForbidden": "true" with a reason "ImageRejected"? Should the deployment also have a condition surfaced?

Right now, we see this all the time in OpenShift (by virtue of security policy) and end users really suffer without this info being surfaced (in quota, in policy, and in crash looping pods).

Yes. #22298

I'm starting to feel like we're not doing enough to actually solve the usability problem here - the approach you describe is correct (RS needs to surface that info) but we're doing a terrible job of it. For instance... should RS have a condition "CreationForbidden": "true" with a reason "ImageRejected"? Should the deployment also have a condition surfaced?

Right now, we see this all the time in OpenShift (by virtue of security policy) and end users really suffer without this info being surfaced (in quota, in policy, and in crash looping pods).

Conditions are one thing that may or may not help here but I believe we already have a mechanism for reporting failures, albeit we are doing a bad job with it: events. Events are nice but fetching exactly those you are interested in is impossible currently. It feels like we should have events as a subresource to all objects or something. #11994 is related.

Found your answer later in the issue. I'm forgetting which proposals I've commented on at this point.

I'd like to generalize this paragraph to be applicable to all controllers. Deplyments and ReplicaSets are not the only ones currently. But I agree with Clayton and Michalis here that we need to make it more obvious and Conditions are one of the possibilities here.

Linked this discussion from #22298

smarterclayton · 2016-06-09T19:16:01Z

docs/proposals/image-provenance.md

+We could have sent the username of the pod creator to the backend.  The username could be used to allow different users to run
+different categories of images.  This would require propagating the username from e.g. Deployment creation, through to
+Pod creation via, e.g. the `Impersonate-User:` header.  This feature is not ready.  When it is, we will re-evaluate
+adding user as a field of `ImagePolicyRequest`.


Isn't this ready now?

API-server respects this header. Controllers do not use it.

Filed #27152

gtank · 2016-06-09T20:34:22Z

docs/proposals/image-provenance.md

+The ImageReviewSpec is used as the key to the cache.
+
+In the case of a cache miss and timeout talking to the backend, the default is to allow Pod creation.
+Keeping services running is more important than a hypotetical threat from an un-verified image.


s/hypotetical/hypothetical

Doesn't this mean any backend DoS lasting more than an hour will result in a fail-open cluster?

Yes. Not sure what else to do, though.

Thinking out loud here: what if you fell back to using any existing "allow" cache as a whitelist if no contact with the backend can be established for a TTL period? This allows things that are already deployed to continue running without failing completely open.

gtank · 2016-06-09T20:40:38Z

docs/proposals/image-provenance.md

+
+## Ubernetes
+
+If two clusters share an image policy backend, then they will have the same policies.


Is the intent to synchronize the admission caches in the federated/ubernetes cases?

gtank · 2016-06-09T20:53:14Z

docs/proposals/image-provenance.md

+We will wait and see how much demand there is for closing this hole. If the community demands a solution,
+we may suggest one of these:
+
+1.  Use a backend that refuses to accept images that are specified with tags, and require users to resolve to IDs


Always forcing users to resolve IDs would allow backends to interact more effectively with content trust measures. With just the tag, there's no binding between what was actually deployed and what the backend sees. cc @ecordell

Resolving IDs before they are put into pods / deployments / rcs is the best
outcome. Doing that resolution in the admission control is fraught (there
should be another admission controller that does the transformation, if you
really want that)

On Thu, Jun 9, 2016 at 4:53 PM, George Tankersley notifications@github.com
wrote:

In docs/proposals/image-provenance.md
#27129 (comment)
:

+## Image tags and IDs
+
+Image tags are like: myrepo/myimage:v1.
+
+Image IDs are like: myrepo/myimage@sha256:beb6bd6a68f114c1dc2ea4b28db81bdf91de202a9014972bec5e4d9171d90ed.
+You can see image IDs with docker images --no-trunc.
+
+The Backend needs to be able to resolve tags to IDs (by talking to the images repo).
+If the Backend resolves tags to IDs, there is some risk that the tag-to-ID mapping will be
+modified after approval by the Backend, but before Kubelet pulls the image. We will not address this
+race condition at this time.
+
+We will wait and see how much demand there is for closing this hole. If the community demands a solution,
+we may suggest one of these:
+
+1. Use a backend that refuses to accept images that are specified with tags, and require users to resolve to IDs

Forcing users to resolve IDs would allow a backend to interact more
effectively with content trust measures. With just the tag, there's no
binding between what was actually deployed and what the backend sees. You
need to somehow supply an account of the data. cc @ecordell
https://github.com/ecordell

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/pull/27129/files/790ab4954039f8d601b925896c01962face3765f#r66519606,
or mute the thread
https://github.com/notifications/unsubscribe/ABG_pykCE46gTf8z3jGttIEODyKkW9UPks5qKH1LgaJpZM4IyI6J
.

Agreed; any signature verification doesn't mean much if you don't verify that the image id matches the content

I just commented on this in https://github.com/kubernetes/kubernetes/pull/27129/files/4a60be831efce93d7e210df47d79e7c18d5d13c2#r71607506

I think it makes sense to either map tag to SHA in kubectl, in CICD, or after the fact in Kubelet. I agree the admission controller is a bad place to map tag to SHA. I think admission is a fine place to require image names that use SHAs.

erictune · 2016-06-13T15:31:33Z

xref: #22888

alex-mohr · 2016-06-21T00:32:49Z

docs/proposals/image-provenance.md

+- a newly created replicaSet will be unable to create Pods.
+- updating a deployment will be safe in the sense that it will detect that the new ReplicaSet is not scaling
+  up and not scale down the old one.
+- an existing replicaSet will not be unable to create Pods that replace ones which are terminated.  If this is due to


nit: "will not be unable to create" -> "will be unable to create"

bgrant0607 · 2016-06-22T19:44:36Z

docs/proposals/image-provenance.md

+
+## Admission controller
+
+An `ImagePolicyWebhook` admission controller will be written.  The admission controller examines all pod objects which are


Is the webhook going to be tried until success? How would it distinguish retryable failure from permanent failure?

The admission controller will admit if the webhook times out.

Will the admin be able to set a policy for a namespace (e.g., fail-open/fail-closed)?

Will this be a generic webhook for any admission plugin? I'd like to see a generic one and it seems like the work here would be about the same.

It will not be a generic webhook. A generic webhook would need a lot more discussion:

a generic webhook needs to touch all objects, not just pods. So it won't have a fixed schema

a generic webhook client needs to ignore kinds it doesn't care about, or the apiserver needs to know which backends care about which kinds

it exposes our whole API to a webhook without giving us (the project) any chance to review or understand how it is being used.

because we don't know which fields of an object are inspected by the backend, caching is not effective. Sending fewer fields allows caching.

sending fewer fields makes it possible to rev the version of the webhook request slower than the version of our internal obejcts (e.g. pod v2 could still use imageReview v1.)

probably lots more reasons.

Added section about this in Alternatives section.

erictune · 2016-07-23T00:33:38Z

@fabioy this is the proposal for image admission controller that we talked about.

philips · 2016-07-23T02:04:20Z

docs/proposals/image-provenance.md

+ - only run images that are scanned to confirm they do no contain vulnerabilities
+ - only run images that use a "required" base image
+ - only run images that contain binaries which were built from peer reviewed, checked-in source
+   by a trusted compiler toolchain.


@erictune isn't the other obvious example images that are signed by a public key in a key list?

Yes. I would do that in conjunction with other controls. I might want to also enforce the whitelist of signing keys at the node level, at which point the check in the apiserver is more a nice-to-have. But you could do it that way.

Adding your suggestion to the doc.

deads2k · 2016-07-25T18:28:56Z

docs/proposals/image-provenance.md

+There will be a cache of decisions in the admission controller. 
+
+If the apiserver cannot reach the webhook backend, it will log a warning and admit the pod.  The rationale here is
+that keeping the system running is most important, and people who care about image provenance deeply will


I think I'd prefer to default the other way or at least offer an option for someone to choose which way they want to change. If we run a bad image and it destroys something, there might not be a way to un-ring the bell.

@liggitt

I think I'd prefer to default the other way or at least offer an option for someone to choose which way they want to change

+1... getting a bad image in should not be as simple as DOSing an admission webhook endpoint

+1 for default deny, with an option to change it if admin wants to do so and is aware of the risk.

I support flag control over deny-on-failure vs allow-on-failure.

Doing that per-namespace, as I think @Q-Lee mentioned above, would not work well with a flag. @Q-Lee do you have a use case for per-namespace?

Interesting question: how to bootstrap image provenance controller running on the cluster itself.

Adding line about the flag.

smarterclayton · 2016-07-25T19:03:23Z

docs/proposals/image-provenance.md

+An `ImagePolicyWebhook` admission controller will be written.  The admission controller examines all pod objects which are
+created or updated.  It can either admit the pod, or reject it.  If it is rejected, the request sees a `403 FORBIDDEN`
+
+The admission controller code will go in `plugin/pkg/admission/imagepolicy`.


I would like for the admission controller to take an interface for "check decision" rather than embedding all the client logic in it. That would allow alternate admission controller implementations to be more easily implemented. As an example, we're trying to build composable chunks of admission logic for policy like this that can be reused in other contexts. Things like authorizer and authentication have succeeded pretty well at this (@liggitt has done some crazy authenticator wrappers that work cleanly). I would like to generally have our policy decision steps behind clean interfaces (in this case, an interface that answers the question about an image and it being accepted and mirrors the ImageReview object) that can be composed later on.

I would like for the admission controller to take an interface for "check decision" rather than embedding all the client logic in it.

You're saying you don't like the webhook mechanism suggested, that you want a generic webhook like I asked above above, or that you want the reference webhook impl to accept what amounts to an admission.Interface?

On Jul 25, 2016, at 3:21 PM, David Eads notifications@github.com wrote:

In docs/proposals/image-provenance.md
#27129 (comment):

reduces latency and allows short outages of the backend to be tolerated.

+Detailed discussion in Ensuring only images are from approved sources are run.
+
+# Implementation
+
+A new admission controller will be added. That will be the only change.
+
+## Admission controller
+
+An ImagePolicyWebhook admission controller will be written. The admission controller examines all pod objects which are
+created or updated. It can either admit the pod, or reject it. If it is rejected, the request sees a 403 FORBIDDEN
+
+The admission controller code will go in plugin/pkg/admission/imagepolicy.

I would like for the admission controller to take an interface for "check
decision" rather than embedding all the client logic in it.

You're saying you don't like the webhook mechanism suggested, that you want
a generic webhook like I asked above above, or that you want the reference
webhook impl to accept what amounts to and admission.Interface?

The latter. Authorizer and Admission interfaces are very successful.
Would like to try to define the equivalents for other policy equally well.

smarterclayton · 2016-07-25T19:05:12Z

docs/proposals/image-provenance.md

+// ImageReviewSpec is a description of the pod creation request.
+type ImageReviewSpec struct {
+	// Containers is a list of a subset of the information in each container of the Pod being created.
+	Containers []ImageReviewContainerSpec


Unfortunately you also need to accept init containers. I would recommend creating a nested struct that is similar to pod template that has the subset of info in it, and make that hierarchal rather than flattened, i.e.:

type ImageReviewPodTemplate struct { Metadata ImageReviewObjectMeta Spec ImageReviewPodSpec }

If we can preserve the same hierarchy as a PodTemplate, that makes automated tools easier in the future (we might take the pod object as unstructured and whitelist the things that are included in a generic fashion).

It also establishes a pattern for future examples of this where subsets of data are returned.

Is it settled where in the PodSpec that init containers will live?

Also, if we do what you said, it couples the versioning of Pods to the versioning of the ImageReview api. Is that desirable?

If is extremely likely init containers will be a peer to containers, barring any sudden revelations.

Possibly not, although creating a new structure to learn is causing
API drift. I would expect this API to evolve to be consistent with
pods in the long term (v1 pods to vX ImagePolicyReview)

I'm of the opinion that instead of mimicking pod layout, we should eliminate the container list altogether. The goal here is to establish a chain of trust for containers, and not to enforce pod-level policies. If you create a pod with 5 containers, then you make 5 requests to the backend.

I'm of the opinion that instead of mimicking pod layout, we should eliminate the container list altogether. The goal here is to establish a chain of trust for containers, and not to enforce pod-level policies. If you create a pod with 5 containers, then you make 5 requests to the backend.

I think that it's likely we'll end up doing both. Given the current power of PSP and it's ability to describe who can request which powers for a pod/container, it seems likely that the decision about whether a particular image is allowed may be affected by the same or a similar policy.

I'm not suggesting that we add that level of complication now, but designing the structure for future expansion seems like a reasonable thing to do and this would be a way to do it.

Also, remote callouts are expensive and if we already have all the data ready, validating it all at once seems pretty reasonable.

Exactly, we have PSP for pod level control. The purpose here is to establish a chain of trust from source code to deployment.

A small scaling factor from pods to containers is nothing to bat an eye at.

soltysh · 2016-07-25T20:21:48Z

docs/proposals/image-provenance.md

+
+The exact nature of "approval" is beyond the scope of Kubernetes, but may include reasons like:
+
+ - only run images that are scanned to confirm they do no contain vulnerabilities


s/do no/do not

soltysh · 2016-07-25T20:32:10Z

docs/proposals/image-provenance.md

+* Block creation of pods that would cause "unapproved" images to run.
+* Make it easy for users or partners to build "image provenance checkers" which check whether images are "approved".
+  *  We expect there will be multiple implementations.
+* Allow users to request an "override" of the policy in a convenient way (subject to the override being allowed).


Is the override be available to all users? Or this will be tied to specific authz?

In one possible implementation, the override is available to all users, but a user who requests the override would be expected to answer to an auditor, sometime after the fact, if she requests the override.

erictune · 2016-07-25T23:49:52Z

I've addressed most comments and updated the docs.

The open issues, as I see it, are:

how to handle init-containers
if the schema should be a subset of PodSpec, so that additional fields fall in the same places.
something about interfaces that I don't quite follow.

smarterclayton · 2016-07-26T00:56:42Z

Interfaces was about making the guts of the admission controller call a policy interface that looks like Authorizer and Admission interfaces, because it imposes discipline and also may lead to experimentation.

philips · 2016-07-27T00:28:26Z

cc the Clair team @Quentin-M @jzelinskie @josephschorr

jzelinskie · 2016-07-27T00:46:02Z

docs/proposals/image-provenance.md

+## Admission controller
+
+An `ImagePolicyWebhook` admission controller will be written.  The admission controller examines all pod objects which are
+created or updated.  It can either admit the pod, or reject it.  If it is rejected, the request sees a `403 FORBIDDEN`


This description is a little vague in the plurality sense. I think this should work over a set of webhooks rather than one. For example, this Pull Request on GitHub has multiple webhooks that must validate it before it is merged.

I kinda understood this as being able to setup multiple, but having it explicit in the proposal is a good idea.

This description is a little vague in the plurality sense. I think this should work over a set of webhooks rather than one. For example, this Pull Request on GitHub has multiple webhooks that must validate it before it is merged.

Seems like we could support a single callout and if someone wanted a union, they could write the union in their particular handler. That keeps our core code out of the business of deciding between the ands, ors, and trumps, which inevitably follow the "give me more than one".

I don't see an issue with making a reference impl for the webhook that can provide a simple union, but I don't think we want to bake multiples into our admission plugin.

I feel convinced.

@deads2k It seems there's precedent in other parts of k8s for doing it the way you have described, so I agree.

ericchiang · 2016-08-03T15:34:21Z

docs/proposals/image-provenance.md

+The WebHook request and response are JSON, and correspond to the following `go` structures:
+
+```go
+// Filename: pkg/apis/authentication.k8s.io/register.go


Are these types intended to live in the authentication.k8s.io API group? Is the package name correct here?

Q-Lee · 2016-08-03T23:55:01Z

docs/proposals/image-provenance.md

+
+// ImageReviewContainerSpec is a description of a container within the pod creation request.
+type ImageReviewContainerSpec struct {
+	Image string


Shouldn't this be Image, ImageHash string?

Images can be specified to docker, and in pod.spec.container[].image as either image:tag or image@SHA:012345679abcdef. So, this field also accepts either format.

It is up to the backend to decide if it accepts image:tag or only accepts image@SHA:012345679abcdef format. There are reasons you might chose to do either way, so this API doesn't have an opinion.

Q-Lee · 2016-08-04T20:57:25Z

This is very close. Let's put this and I'll make the remaining changes in a new, narrower PR.

k8s-github-robot · 2016-08-05T20:03:17Z

PR changed after LGTM, removing LGTM. @erictune @philips @soltysh @Q-Lee

k8s-bot · 2016-08-05T20:25:55Z

GCE e2e build/test passed for commit 9d59ae5.

Added image-policy proposal

googlebot added the cla: yes label Jun 9, 2016

k8s-github-robot assigned bgrant0607 Jun 9, 2016

k8s-github-robot added kind/design size/L release-note-label-needed labels Jun 9, 2016

ericchiang reviewed Jun 9, 2016
View changes

erictune force-pushed the erictune:imgprov branch from a0343bf to 790ab49 Jun 9, 2016

erictune mentioned this pull request Jun 9, 2016

Ensuring only images are from approved sources are run #22888

Closed

smarterclayton reviewed Jun 9, 2016
View changes

gtank reviewed Jun 9, 2016
View changes

dchen1107 mentioned this pull request Jun 10, 2016

Integration test is broken due to swagger-checker breakage #27205

Closed

erictune force-pushed the erictune:imgprov branch from 790ab49 to 1217120 Jun 14, 2016

smarterclayton mentioned this pull request Jun 14, 2016

Add a generic admission controller that uses JSONPath for rules. #27336

Closed

alex-mohr reviewed Jun 21, 2016
View changes

erictune force-pushed the erictune:imgprov branch from 1217120 to 4a60be8 Jun 21, 2016

bgrant0607 reviewed Jun 22, 2016
View changes

bgrant0607 assigned smarterclayton and unassigned bgrant0607 Jun 23, 2016

philips mentioned this pull request Jul 23, 2016

Container Image Policy kubernetes/enhancements#59

Closed

6 of 21 tasks complete

philips reviewed Jul 23, 2016
View changes

deads2k reviewed Jul 25, 2016
View changes

smarterclayton reviewed Jul 25, 2016
View changes

soltysh reviewed Jul 25, 2016
View changes

erictune force-pushed the erictune:imgprov branch from 9136f7c to 2ae1770 Jul 25, 2016

jzelinskie reviewed Jul 27, 2016
View changes

k8s-github-robot added the do-not-merge label Jul 28, 2016

ericchiang reviewed Aug 3, 2016
View changes

erictune force-pushed the erictune:imgprov branch from 2ae1770 to bfd1305 Aug 3, 2016

Q-Lee reviewed Aug 3, 2016
View changes

Q-Lee added the lgtm label Aug 4, 2016

Added image-provenance proposal

Loading status checks…

9d59ae5

erictune force-pushed the erictune:imgprov branch from bfd1305 to 9d59ae5 Aug 5, 2016

k8s-github-robot removed the lgtm label Aug 5, 2016

erictune added the e2e-not-required [deprecated use retest-not-required] label Aug 5, 2016

zhouhaibing089 mentioned this pull request Aug 15, 2016

Image Signing Support #30603

Closed

xingzhou pushed a commit to xingzhou/kubernetes that referenced this pull request Dec 15, 2016

Merge pull request kubernetes#27129 from erictune/imgprov

8a85b8c

Added image-policy proposal

erictune deleted the erictune:imgprov branch Aug 8, 2017


		## Ubernetes

		If two clusters share an image policy backend, then they will have the same policies.


		## Admission controller

		An `ImagePolicyWebhook` admission controller will be written. The admission controller examines all pod objects which are


		The exact nature of "approval" is beyond the scope of Kubernetes, but may include reasons like:

		- only run images that are scanned to confirm they do no contain vulnerabilities

kubernetes / kubernetes

Added image-policy proposal #27129

Added image-policy proposal #27129

Conversation

erictune commented Jun 9, 2016

erictune commented Jun 9, 2016

erictune commented Jun 9, 2016

erictune commented Jun 9, 2016

This comment has been minimized.

ericchiang Jun 9, 2016 • edited Member

This comment has been minimized.

erictune Jun 9, 2016 Author Member

This comment has been minimized.

ericchiang Jun 9, 2016 Member

This comment has been minimized.

erictune Jun 9, 2016 Author Member

erictune commented Jun 9, 2016

ericchiang commented Jun 9, 2016

This comment has been minimized.

smarterclayton Jun 9, 2016 Contributor

This comment has been minimized.

smarterclayton Jun 9, 2016 Contributor

This comment has been minimized.

smarterclayton Jun 9, 2016 Contributor

This comment has been minimized.

erictune Jun 9, 2016 Author Member

This comment has been minimized.

smarterclayton Jun 9, 2016 Contributor

This comment has been minimized.

erictune Jun 9, 2016 Author Member

This comment has been minimized.

kargakis Jun 9, 2016 Member

This comment has been minimized.

smarterclayton Jun 10, 2016 Contributor

This comment has been minimized.

soltysh Jul 20, 2016 Contributor

This comment has been minimized.

erictune Jul 22, 2016 Author Member

This comment has been minimized.

smarterclayton Jun 9, 2016 Contributor

This comment has been minimized.

erictune Jun 9, 2016 Author Member

This comment has been minimized.

erictune Jun 9, 2016 Author Member

This comment has been minimized.

gtank Jun 9, 2016

This comment has been minimized.

gtank Jun 9, 2016

This comment has been minimized.

erictune Jun 9, 2016 Author Member

This comment has been minimized.

gtank Jun 13, 2016 • edited

This comment has been minimized.

gtank Jun 9, 2016 • edited

This comment has been minimized.

erictune Jun 9, 2016 Author Member

This comment has been minimized.

gtank Jun 9, 2016 • edited

This comment has been minimized.

smarterclayton Jun 9, 2016 Contributor

This comment has been minimized.

ecordell Jun 9, 2016 Contributor

This comment has been minimized.

erictune Jul 20, 2016 Author Member

erictune commented Jun 13, 2016

This comment has been minimized.

alex-mohr Jun 21, 2016 Contributor

This comment has been minimized.

erictune Jul 22, 2016 Author Member

This comment has been minimized.

bgrant0607 Jun 22, 2016 Member

This comment has been minimized.

erictune Jul 23, 2016 Author Member

This comment has been minimized.

Q-Lee Jul 23, 2016 Contributor

This comment has been minimized.

deads2k Jul 25, 2016 Contributor

This comment has been minimized.

erictune Jul 25, 2016 Author Member

This comment has been minimized.

ericchiang Jun 9, 2016 •

edited

Member

erictune Jun 9, 2016
Author Member

ericchiang Jun 9, 2016
Member

erictune Jun 9, 2016
Author Member

smarterclayton Jun 9, 2016
Contributor

smarterclayton Jun 9, 2016
Contributor

smarterclayton Jun 9, 2016
Contributor

erictune Jun 9, 2016
Author Member

smarterclayton Jun 9, 2016
Contributor

erictune Jun 9, 2016
Author Member

kargakis Jun 9, 2016
Member

smarterclayton Jun 10, 2016
Contributor

soltysh Jul 20, 2016
Contributor

erictune Jul 22, 2016
Author Member

smarterclayton Jun 9, 2016
Contributor

erictune Jun 9, 2016
Author Member

erictune Jun 9, 2016
Author Member

erictune Jun 9, 2016
Author Member

gtank Jun 13, 2016 •

edited

gtank Jun 9, 2016 •

edited

erictune Jun 9, 2016
Author Member

gtank Jun 9, 2016 •

edited

smarterclayton Jun 9, 2016
Contributor

ecordell Jun 9, 2016
Contributor

erictune Jul 20, 2016
Author Member

alex-mohr Jun 21, 2016
Contributor

erictune Jul 22, 2016
Author Member

bgrant0607 Jun 22, 2016
Member

erictune Jul 23, 2016
Author Member

Q-Lee Jul 23, 2016
Contributor

deads2k Jul 25, 2016
Contributor

erictune Jul 25, 2016
Author Member

erictune Jul 25, 2016
Author Member

philips Jul 23, 2016
Contributor

erictune Jul 25, 2016
Author Member

erictune Jul 25, 2016
Author Member

deads2k Jul 25, 2016
Contributor

liggitt Jul 25, 2016 •

edited

Member

soltysh Jul 25, 2016 •

edited

Contributor

erictune Jul 25, 2016
Author Member

erictune Jul 25, 2016
Author Member

erictune Jul 25, 2016
Author Member

smarterclayton Jul 25, 2016
Contributor

deads2k Jul 25, 2016 •

edited

Contributor

smarterclayton Jul 25, 2016
Contributor

smarterclayton Jul 25, 2016
Contributor

smarterclayton Jul 25, 2016
Contributor

erictune Jul 25, 2016
Author Member

erictune Jul 25, 2016
Author Member

smarterclayton via email Jul 26, 2016
Contributor

smarterclayton Jul 26, 2016
Contributor

Q-Lee Aug 4, 2016
Contributor

deads2k Aug 4, 2016
Contributor

Q-Lee Aug 4, 2016
Contributor

soltysh Jul 25, 2016
Contributor

soltysh Jul 25, 2016
Contributor

erictune Jul 25, 2016
Author Member

soltysh Jul 28, 2016
Contributor

deads2k Jul 28, 2016
Contributor

soltysh Jul 28, 2016
Contributor

jzelinskie Jul 31, 2016 •

edited

ericchiang Aug 3, 2016
Member

erictune Aug 3, 2016
Author Member