Add support for gating node creation on TPM-based remote attestation #23834

Closed
wants to merge 4 commits into
from

Conversation

Projects
None yet
@mjg59

mjg59 commented Apr 4, 2016

This is a straw-man proposal for the moment - I'm interested in wider feedback on design decisions.

This PR adds an admission controller that flags nodes as unscheduleable and untrusted if they fail to meet an admin-defined TPM policy, making it possible to ensure that systems have booted in a trusted configuration before being used to run jobs. TPM measurements are provided by an external TPM daemon that is part of the go-tspi project and should be run on the workers. A config file is defined for the admission controller, which controls whether the system should permit previously unknown TPMs, whether the system should trigger re-attestation of node state on a regular basis and whether the policy should be sourced from the filesystem or from the k8s API.

The questions I'm looking to resolve are:

  1. Is this considered useful within Kubernetes?
  2. If so, should this be maintained as part of Kubernetes or as a primarily out-of-tree admission controller plugin?
  3. If out of tree, should in-tree API objects be defined or should it use third party resources?

This change is Reviewable

mjg59 added some commits Mar 23, 2016

Add TPM dependencies
Pull in the dependencies required to communicate with remote TPMs
Add untrusted flag to nodespec
Add a flag to permit a node to be marked as untrusted, permitting later
diagnostics.
Add TPM admission plugin and support code
Add an admission control plugin for TPMs. This makes it possible to validate
that a worker has booted in a known good configuration before permitting it
to create or update a node.

@googlebot googlebot added the cla: yes label Apr 4, 2016

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Apr 4, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in hack/jenkins/job-configs/kubernetes-jenkins-pull/ instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Apr 4, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in hack/jenkins/job-configs/kubernetes-jenkins-pull/ instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Apr 4, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in hack/jenkins/job-configs/kubernetes-jenkins-pull/ instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Apr 4, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in hack/jenkins/job-configs/kubernetes-jenkins-pull/ instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@@ -1667,6 +1667,9 @@ type NodeSpec struct {
// Unschedulable controls node schedulability of new pods. By default node is schedulable.
Unschedulable bool `json:"unschedulable,omitempty"`
+
+ // Untrusted indicates that the node is not trusted. By default node is trusted.
+ Untrusted bool `json:"untrusted,omitempty"`

This comment has been minimized.

@mjg59

mjg59 Apr 4, 2016

I've deliberately left out the corresponding autogenerated code updates for ease of review - I'll obviously add them if we want to merge this at some point.

@mjg59

mjg59 Apr 4, 2016

I've deliberately left out the corresponding autogenerated code updates for ease of review - I'll obviously add them if we want to merge this at some point.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Apr 4, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in hack/jenkins/job-configs/kubernetes-jenkins-pull/ instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Apr 4, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in hack/jenkins/job-configs/kubernetes-jenkins-pull/ instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Apr 4, 2016

Collaborator

Adding label:do-not-merge because PR changes docs prohibited to auto merge
See http://kubernetes.io/editdocs/ for information about editing docs

Collaborator

k8s-merge-robot commented Apr 4, 2016

Adding label:do-not-merge because PR changes docs prohibited to auto merge
See http://kubernetes.io/editdocs/ for information about editing docs

@k8s-teamcity-mesosphere

This comment has been minimized.

Show comment
Hide comment
@k8s-teamcity-mesosphere

k8s-teamcity-mesosphere Apr 4, 2016

TeamCity OSS :: Kubernetes Mesos :: 4 - Smoke Tests Build 20534 outcome was SUCCESS
Summary: Tests passed: 1, ignored: 267 Build time: 00:05:12

TeamCity OSS :: Kubernetes Mesos :: 4 - Smoke Tests Build 20534 outcome was SUCCESS
Summary: Tests passed: 1, ignored: 267 Build time: 00:05:12

@bgrant0607 bgrant0607 assigned erictune and unassigned davidopp Apr 19, 2016

@bgrant0607

This comment has been minimized.

Show comment
Hide comment
@bgrant0607

This comment has been minimized.

Show comment
Hide comment
Member

bgrant0607 commented Apr 19, 2016

@roberthbailey

This comment has been minimized.

Show comment
Hide comment
@roberthbailey

roberthbailey Apr 19, 2016

Member

FYI, I'll be OOO until next week so I won't be able to look at this until then.

Member

roberthbailey commented Apr 19, 2016

FYI, I'll be OOO until next week so I won't be able to look at this until then.

@mikedanese

This comment has been minimized.

Show comment
Hide comment
@mikedanese

mikedanese Apr 20, 2016

Member

Why is the TPM not checked by the the CSR approval process described in the TLS Bootstrap proposal? It seems like the approval process could very easily challenge the TPM and approve the CSR only if the node passes the challenge. What are the pros of this approach?

Member

mikedanese commented Apr 20, 2016

Why is the TPM not checked by the the CSR approval process described in the TLS Bootstrap proposal? It seems like the approval process could very easily challenge the TPM and approve the CSR only if the node passes the challenge. What are the pros of this approach?

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Apr 20, 2016

Member

The node TLS serving cert approval process is orthogonal to the node being schedulable.

This particular proposal puts a long running operation in the API path. It seems like it would be better modeled as only letting a "node verifier" process mark nodes schedulable, which is a particular example of the need for field-level authorization. If we had that, nodes could be allowed to register themselves but not mark themselves schedulable, and another process could be authorized to mark them schedulable, moving the TPM process out of the API server.

Member

liggitt commented Apr 20, 2016

The node TLS serving cert approval process is orthogonal to the node being schedulable.

This particular proposal puts a long running operation in the API path. It seems like it would be better modeled as only letting a "node verifier" process mark nodes schedulable, which is a particular example of the need for field-level authorization. If we had that, nodes could be allowed to register themselves but not mark themselves schedulable, and another process could be authorized to mark them schedulable, moving the TPM process out of the API server.

@mjg59

This comment has been minimized.

Show comment
Hide comment
@mjg59

mjg59 Apr 20, 2016

@mikedanese Possession of the TLS certificates only indicates that the machine was trusted at the time the TLS certificates were issued - it says nothing about whether the machine is currently in a trustworthy state (eg, an attacker who has a means to reboot the system and is able to inject a different kernel into the deployment mechanism can create an untrusted system that may still have trusted TLS certificates on the filesystem)

mjg59 commented Apr 20, 2016

@mikedanese Possession of the TLS certificates only indicates that the machine was trusted at the time the TLS certificates were issued - it says nothing about whether the machine is currently in a trustworthy state (eg, an attacker who has a means to reboot the system and is able to inject a different kernel into the deployment mechanism can create an untrusted system that may still have trusted TLS certificates on the filesystem)

@mjg59

This comment has been minimized.

Show comment
Hide comment
@mjg59

mjg59 Apr 20, 2016

@liggitt My understanding of the current codebase is that there's no explicit sense in which a node decides to be schedulable - instead, nodes request that they not be schedulable. So what I imagine in your suggestion is:

  1. An admission controller that forcibly sets the unschedulable flag
  2. An independent process that sweeps unschedulable nodes, requests attestation and then clears the unschedulable flag

with appropriate authorization configuration such that only the independent process has permission to clear this flag? Right now there isn't an obviously good way for us to distinguish between nodes that are unschedulable because they haven't been validated yet and nodes that are unschedulable because they asked to be (eg, a master) so we'd need to make that possible.

I'm happy to rework this along those lines if there's broad consensus that keeping this out of process would be preferable.

mjg59 commented Apr 20, 2016

@liggitt My understanding of the current codebase is that there's no explicit sense in which a node decides to be schedulable - instead, nodes request that they not be schedulable. So what I imagine in your suggestion is:

  1. An admission controller that forcibly sets the unschedulable flag
  2. An independent process that sweeps unschedulable nodes, requests attestation and then clears the unschedulable flag

with appropriate authorization configuration such that only the independent process has permission to clear this flag? Right now there isn't an obviously good way for us to distinguish between nodes that are unschedulable because they haven't been validated yet and nodes that are unschedulable because they asked to be (eg, a master) so we'd need to make that possible.

I'm happy to rework this along those lines if there's broad consensus that keeping this out of process would be preferable.

@mikedanese

This comment has been minimized.

Show comment
Hide comment
@mikedanese

mikedanese Apr 20, 2016

Member

Possession of the TLS certificates only indicates that the machine was trusted at the time the TLS certificates were issued - it says nothing about whether the machine is currently in a trustworthy state (eg, an attacker who has a means to reboot the system and is able to inject a different kernel into the deployment mechanism can create an untrusted system that may still have trusted TLS certificates on the filesystem)

This is greatly mitigated by only issuing short lived certs.

The tpm is only checked on creation and update of the Node object. IIUC, I can compromise a node and the certs will continue to work as long the node object is unmodified. What coverage does this add?

Member

mikedanese commented Apr 20, 2016

Possession of the TLS certificates only indicates that the machine was trusted at the time the TLS certificates were issued - it says nothing about whether the machine is currently in a trustworthy state (eg, an attacker who has a means to reboot the system and is able to inject a different kernel into the deployment mechanism can create an untrusted system that may still have trusted TLS certificates on the filesystem)

This is greatly mitigated by only issuing short lived certs.

The tpm is only checked on creation and update of the Node object. IIUC, I can compromise a node and the certs will continue to work as long the node object is unmodified. What coverage does this add?

@liggitt

This comment has been minimized.

Show comment
Hide comment
@liggitt

liggitt Apr 20, 2016

Member

The node's TLS serving certs don't really have anything to do with whether a pod can be scheduled to a node,since the node queries the API for its workloads.

Member

liggitt commented Apr 20, 2016

The node's TLS serving certs don't really have anything to do with whether a pod can be scheduled to a node,since the node queries the API for its workloads.

@bgrant0607

This comment has been minimized.

Show comment
Hide comment
@bgrant0607

bgrant0607 Apr 20, 2016

Member

Re. node schedulability, taints are a better answer than the current unschedulable field:
https://github.com/kubernetes/kubernetes/blob/master/docs/design/taint-toleration-dedicated.md
#17190, #24134

Member

bgrant0607 commented Apr 20, 2016

Re. node schedulability, taints are a better answer than the current unschedulable field:
https://github.com/kubernetes/kubernetes/blob/master/docs/design/taint-toleration-dedicated.md
#17190, #24134

@bgrant0607

This comment has been minimized.

Show comment
Hide comment
@bgrant0607

bgrant0607 Apr 20, 2016

Member

In fact, if taints were used, I don't think we'd need the API field.

Member

bgrant0607 commented Apr 20, 2016

In fact, if taints were used, I don't think we'd need the API field.

@bgrant0607

This comment has been minimized.

Show comment
Hide comment
@bgrant0607

bgrant0607 Apr 20, 2016

Member

Re: "3) If out of tree, should in-tree API objects be defined or should it use third party resources?"

If out of tree and using generic extension mechanisms (ThirdPartyResource, annotations, taints, etc.), it would be easier to prototype.

Member

bgrant0607 commented Apr 20, 2016

Re: "3) If out of tree, should in-tree API objects be defined or should it use third party resources?"

If out of tree and using generic extension mechanisms (ThirdPartyResource, annotations, taints, etc.), it would be easier to prototype.

@mikedanese

This comment has been minimized.

Show comment
Hide comment
@mikedanese

mikedanese Apr 20, 2016

Member

The node's TLS serving certs don't really have anything to do with whether a pod can be scheduled to a node,since the node queries the API for its workloads.

Revoking authorization from a node is a great way to make it unscheduled :)

Ok ignoring the TLS stuff, I can compromise a node in a way that modifies the tpm hash of the machine and the machine will remain "trusted" (i.e. trusted bit set in nodespec) until the node object is touched. What operations in the system should require a strong guarantee (i.e. checked at the time of executing the operation) that the tpm hash is valid? Is it possible using the proposed scheme to make that guarantee? What operations in the system should require a less strong guarantee (i.e. checked sometime in the last minutes) that the tpm hash is valid?

Member

mikedanese commented Apr 20, 2016

The node's TLS serving certs don't really have anything to do with whether a pod can be scheduled to a node,since the node queries the API for its workloads.

Revoking authorization from a node is a great way to make it unscheduled :)

Ok ignoring the TLS stuff, I can compromise a node in a way that modifies the tpm hash of the machine and the machine will remain "trusted" (i.e. trusted bit set in nodespec) until the node object is touched. What operations in the system should require a strong guarantee (i.e. checked at the time of executing the operation) that the tpm hash is valid? Is it possible using the proposed scheme to make that guarantee? What operations in the system should require a less strong guarantee (i.e. checked sometime in the last minutes) that the tpm hash is valid?

@mjg59

This comment has been minimized.

Show comment
Hide comment
@mjg59

mjg59 Apr 21, 2016

@mikedanese TPM state is revalidated periodically, and the node marked as untrusted if validation fails. This is independent of whether the node object has been updated (although in practice the node object is updated frequently due to heartbeats - we don't want to depend on that, though, since a compromised node could simply cease sending them)

mjg59 commented Apr 21, 2016

@mikedanese TPM state is revalidated periodically, and the node marked as untrusted if validation fails. This is independent of whether the node object has been updated (although in practice the node object is updated frequently due to heartbeats - we don't want to depend on that, though, since a compromised node could simply cease sending them)

@mjg59

This comment has been minimized.

Show comment
Hide comment
@mjg59

mjg59 Apr 21, 2016

@bgrant0607 Taints look pretty much perfect for this, thanks! My current prototype is using annotations rather than extending the node type, and I think we'll continue using them for now in order to provide information about which TPM policy validated the node.

mjg59 commented Apr 21, 2016

@bgrant0607 Taints look pretty much perfect for this, thanks! My current prototype is using annotations rather than extending the node type, and I think we'll continue using them for now in order to provide information about which TPM policy validated the node.

@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Apr 21, 2016

Collaborator

@mjg59 PR needs rebase

Collaborator

k8s-merge-robot commented Apr 21, 2016

@mjg59 PR needs rebase

@mjg59

This comment has been minimized.

Show comment
Hide comment
@mjg59

mjg59 Apr 28, 2016

Thought about this some more, and reworking it as an admission controller that defaults nodes to a tainted state along with an external agent that performs TPM attestation and clears the taint seems like a preferable approach. It also has the benefit of allowing other trust agents to be plugged in using the same infrastructure.

mjg59 commented Apr 28, 2016

Thought about this some more, and reworking it as an admission controller that defaults nodes to a tainted state along with an external agent that performs TPM attestation and clears the taint seems like a preferable approach. It also has the benefit of allowing other trust agents to be plugged in using the same infrastructure.

@k8s-merge-robot k8s-merge-robot assigned davidopp and unassigned erictune May 6, 2016

@davidopp

This comment has been minimized.

Show comment
Hide comment
@davidopp

davidopp May 15, 2016

Member

Taints PR is very close to being merged -- #24134

Member

davidopp commented May 15, 2016

Taints PR is very close to being merged -- #24134

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jun 14, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jun 14, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jun 15, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jun 15, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jun 23, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jun 23, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@jessfraz

This comment has been minimized.

Show comment
Hide comment
@jessfraz

jessfraz Jul 7, 2016

Contributor

the taints PR was merged 0:) I can review if you want to update

Contributor

jessfraz commented Jul 7, 2016

the taints PR was merged 0:) I can review if you want to update

@erictune

This comment has been minimized.

Show comment
Hide comment
@erictune

erictune Jul 8, 2016

Member

I don't understand the bigger picture for how this provides security.

If someone gets access to the node's system software and changes it, or otherwise boots a node with non-approved software, then possibly they can:

  • receive traffic which may contain PII via K8s services or ingressess
  • read/write sensitive data from volumes attached to the node
  • use the node's identity on the network to communicate with remote services which use source ip, etc, to identify or block requests.
  • use certs, passwords, or other keys already present on the node to interact with sensitive remote services
  • use Kubernetes certs on the node pull secrets from the apiserver, and then do the previous item.
  • talk to the docker hub and pull and run any images.

Probably I am missing something, but it seem like marking it as "unschedulable" doesn't prevent any of these things from happening.

Member

erictune commented Jul 8, 2016

I don't understand the bigger picture for how this provides security.

If someone gets access to the node's system software and changes it, or otherwise boots a node with non-approved software, then possibly they can:

  • receive traffic which may contain PII via K8s services or ingressess
  • read/write sensitive data from volumes attached to the node
  • use the node's identity on the network to communicate with remote services which use source ip, etc, to identify or block requests.
  • use certs, passwords, or other keys already present on the node to interact with sensitive remote services
  • use Kubernetes certs on the node pull secrets from the apiserver, and then do the previous item.
  • talk to the docker hub and pull and run any images.

Probably I am missing something, but it seem like marking it as "unschedulable" doesn't prevent any of these things from happening.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jul 12, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jul 12, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jul 12, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jul 12, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jul 12, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jul 12, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jul 12, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jul 12, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jul 12, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jul 12, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jul 12, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jul 12, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jul 12, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jul 12, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jul 19, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jul 19, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jul 25, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jul 25, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jul 31, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

k8s-bot commented Jul 31, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@bgrant0607

This comment has been minimized.

Show comment
Hide comment
@bgrant0607

bgrant0607 Jul 31, 2016

Member

@apelisse The messages from k8s-bot asking whether the PR is reasonable to test don't notify anyone specifically. They should notify the assignee. Ref #869

Member

bgrant0607 commented Jul 31, 2016

@apelisse The messages from k8s-bot asking whether the PR is reasonable to test don't notify anyone specifically. They should notify the assignee. Ref #869

@bgrant0607

This comment has been minimized.

Show comment
Hide comment
@bgrant0607

bgrant0607 Jul 31, 2016

Member

ok to test

Member

bgrant0607 commented Jul 31, 2016

ok to test

@k8s-bot

This comment has been minimized.

Show comment
Hide comment
@k8s-bot

k8s-bot Jul 31, 2016

GCE e2e build/test failed for commit 144b3c9.

Please reference the list of currently known flakes when examining this failure. If you request a re-test, you must reference the issue describing the flake.

k8s-bot commented Jul 31, 2016

GCE e2e build/test failed for commit 144b3c9.

Please reference the list of currently known flakes when examining this failure. If you request a re-test, you must reference the issue describing the flake.

@bgrant0607

This comment has been minimized.

Show comment
Hide comment
@bgrant0607

bgrant0607 Jul 31, 2016

Member

@mjg59 I'm going to close this for now. Please reopen if/when you would like to continue with this, though I suggest starting with a proposal first.

Member

bgrant0607 commented Jul 31, 2016

@mjg59 I'm going to close this for now. Please reopen if/when you would like to continue with this, though I suggest starting with a proposal first.

@bgrant0607 bgrant0607 closed this Jul 31, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment