Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

500 response with CronJob in repo #868

Closed
squaremo opened this issue Dec 13, 2017 · 8 comments
Assignees

Comments

@squaremo
Copy link
Member

@squaremo squaremo commented Dec 13, 2017

When we put a CronJob resource in our git repo, fluxd started returning 500s to release requests for another resource, citing only the server could not find the requested resource.

NB this was running Kubernetes 1.7, which doesn't support CronJob; it failed to sync that resource, while otherwise succeeding.

@squaremo

This comment has been minimized.

Copy link
Member Author

@squaremo squaremo commented Dec 14, 2017

That particular error message comes from the kubernetes client, grep tells me.

@squaremo

This comment has been minimized.

Copy link
Member Author

@squaremo squaremo commented Dec 14, 2017

On our dev (where we have the problem):

$ kubectl api-versions
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1beta1
apps/v1beta1
apps/v1beta2
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
batch/v1
batch/v1beta1
certificates.k8s.io/v1beta1
extensions/v1beta1
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1

and in minikube (where I can't reproduce the problem):

$ kubectl api-versions                                                                                                                                                                                             
admissionregistration.k8s.io/v1alpha1               
apiextensions.k8s.io/v1beta1                        
apiregistration.k8s.io/v1beta1                      
apps/v1beta1                                        
apps/v1beta2                                        
authentication.k8s.io/v1                            
authentication.k8s.io/v1beta1                       
authorization.k8s.io/v1                             
authorization.k8s.io/v1beta1                        
autoscaling/v1                                      
autoscaling/v2beta1                                 
batch/v1                                            
batch/v1beta1                                       
batch/v2alpha1                                      
certificates.k8s.io/v1beta1                         
extensions/v1beta1                                  
networking.k8s.io/v1                                
policy/v1beta1                                      
rbac.authorization.k8s.io/v1                        
rbac.authorization.k8s.io/v1alpha1                  
rbac.authorization.k8s.io/v1beta1                   
settings.k8s.io/v1alpha1                            
storage.k8s.io/v1                                   
storage.k8s.io/v1beta1                              
v1                      

The only possibly relevant difference is batch/v2alpha1, and I'm not sure how it's relevant, since the resource in question says it's v1alpha1.

@squaremo

This comment has been minimized.

Copy link
Member Author

@squaremo squaremo commented Dec 14, 2017

Except: https://github.com/weaveworks/flux/blob/master/cluster/kubernetes/resourcekinds.go#L225

/////////////////////////////////////////////////////////////////////////////
// batch/v2alpha1 CronJob

type cronJobKind struct{}
@rade

This comment has been minimized.

Copy link
Contributor

@rade rade commented Dec 14, 2017

There appear to be two distinct problems here:

  1. flux or k8s disliking cronjobs in some circumstances
  2. that dislike preventing releases of other resources

(2) is the more serious issue, by far - no problem with an individual resource should affect operations on others.

@squaremo

This comment has been minimized.

Copy link
Member Author

@squaremo squaremo commented Dec 14, 2017

no problem with an individual resource should affect operations on others.

Yes and no. For some operations we want to know whether we can correctly apply a change, and that usually means looking at everything to make sure there are no duplicates, or whatever.

In this case, I think there's a spot where we're over-estimating the information we need in order to proceed. It's OK to parse all the files to make sure there's a coherently-defined set of controllers (https://github.com/weaveworks/flux/blob/master/release/context.go#L68); less OK is asking for all the resources in question from kubernetes, then filtering down to the single one we care about (a few lines on).

@squaremo

This comment has been minimized.

Copy link
Member Author

@squaremo squaremo commented Dec 15, 2017

that dislike preventing releases of other resources

#869 addresses this problem, by making the release process only ask the cluster about services that are explicitly included in the release. It won't help if the problematic resource is specifically included, in implicitly included with --all.

@squaremo

This comment has been minimized.

Copy link
Member Author

@squaremo squaremo commented Dec 18, 2017

As for the specific problem of dealing with CronJobs: our code uses the batch/v2alpha1 extension to the client, and it seems likely that the API server in dev is just saying "I don't recognise that API extension".

I don't know how one is supposed to move between versions -- either using batch/v1alpha1 instead will just work, or we'll have to ask both APIs (and prepare for idk errors).

@squaremo

This comment has been minimized.

Copy link
Member Author

@squaremo squaremo commented Jan 3, 2018

As for the specific problem of dealing with CronJobs

This bit addressed by #875 (yes, it does respond with resources given either of batch/v2alpha1 or batch/v1beta1).

@squaremo squaremo closed this Jan 3, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.