Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to just wakeup #142

Open
iamatulsingh opened this issue Jun 21, 2022 · 19 comments
Open

How to just wakeup #142

iamatulsingh opened this issue Jun 21, 2022 · 19 comments

Comments

@iamatulsingh
Copy link

iamatulsingh commented Jun 21, 2022

My question is Genuine. Let me tell you one condition.
If I create SleepInfo for weekend and wakeup at weekdays but because of some urgent conditions, I want to make resources up again during weekend. I know I can change the sleepAt (to run for some times at weekend) but again I'll have to change sleepAt like previous (sleep for weekends) configuration and it's really bad if these steps of changing SleepAt can only apply through CICD and that too when someone from an organisation approve it and merge it specially like IAC and after preview can be checked and merge for a environment.

What is the best approach?

@davidebianchi
Copy link
Member

Hi @iamatulsingh, yes this is an interesting feature to add, we could add it to the roadmap.

At the moment, to wake up the namespace we re apply the resources, which reset the replicas to the configured number. But I agree with you that we could have some better interface to handle this.
At first, I thought about to change the SleepInfo configuration (maybe with an annotation or some other config) to tell kube-green to wake up the namespace. But I don't know from your example if this would cover the issue.

Another idea could be an API, but there are some security issues to address. Maybe the more interesting feature could be a dashboard to interact directly with the cluster. So, only someone with the permission to see the kube-green namespace could handle it.

@iamatulsingh
Copy link
Author

iamatulsingh commented Jun 27, 2022

Hi @davidebianchi ,
Thanks for your reply. The way you said to change configuration to make namespace up is good but if someone using Infrastructure as Code to manage kube-green is not that easy specially when code need to get review everytime for the PR. Also making changes to make namespace wakeup and then again change it to the previous one is like more steps.
But this project is amazing and it will be really easy if someday just the wakeup part introduce.

I don't think it's bad at all and any one can change the configuration as you mentione. I'll close this issue.

@iamatulsingh
Copy link
Author

iamatulsingh commented Aug 11, 2022

Sorry to reopen the issue.
I created a SleepInfo as shown in the example and it worked as its described but the problem is when I'm trying to re apply the resources with WakUp time say just after 3 minutes of editing time, It's not waking it up. Everything is still scaled to 0. Am I doing anything wrong?

example code to let everything sleep

apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: working-hours
  namespace: default
spec:
  weekdays: "1-5"
  sleepAt: "13:35"
  wakeUpAt: "08:00"
  timeZone: "Europe/Berlin"
  suspendCronJobs: true
  excludeRef:
    - apiVersion: "apps/v1"
      kind:       Deployment
      name:       api-gateway

This is that I'm trying to wakeup all resources

apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: working-hours
  namespace: default
spec:
  weekdays: "1-5"
  sleepAt: "13:35"
  wakeUpAt: "13:40"
  timeZone: "Europe/Berlin"
  suspendCronJobs: true
  excludeRef:
    - apiVersion: "apps/v1"
      kind:       Deployment
      name:       api-gateway

@iamatulsingh iamatulsingh reopened this Aug 11, 2022
@iamatulsingh
Copy link
Author

Hi @iamatulsingh, yes this is an interesting feature to add, we could add it to the roadmap.

At the moment, to wake up the namespace we re apply the resources, which reset the replicas to the configured number. But I agree with you that we could have some better interface to handle this. At first, I thought about to change the SleepInfo configuration (maybe with an annotation or some other config) to tell kube-green to wake up the namespace. But I don't know from your example if this would cover the issue.

Another idea could be an API, but there are some security issues to address. Maybe the more interesting feature could be a dashboard to interact directly with the cluster. So, only someone with the permission to see the kube-green namespace could handle it.

By this, you meant to re apply resources such as deployments or other things to reset it? I thought to re apply SleepInfo to reset wakup time.

@davidebianchi
Copy link
Member

davidebianchi commented Sep 20, 2022

Hi @iamatulsingh, sorry for the delay in the response.

So, with the re apply of the SleepInfo, the new wake up time is not handled correctly? In the case you described, yes it should wake up at 13:40 (+/- 1 minute).

By this, you meant to re apply resources such as deployments or other things to reset it? I thought to re apply SleepInfo to reset wakup time.

Yes, I meant to reapply the deployment. But reapply the SleepInfo should work too. I try to make some test in the following days

@iamatulsingh
Copy link
Author

Thanks for the reply @davidebianchi . Actually suppose sleepinfo is already in affect but I want to wake it up, reapplying Sleepinfo didn't help.

@victorboissiere
Copy link
Contributor

FYI, it may not help, but it could give insights. On our side, we deploy everything with ArgoCD:

  • we've added a sync window to prevent automatic sync at the same time that deployments were scaled down
  • we've added a job that
    • Remove sleep info config
    • Remove the sync window
    • Trigger ArgoCD sync on resources in the namespace
  • a job with associated alerting not to forget to add it back anytime

It is not optimal, but it allowed us to ship this feature fast.

@AmitBaranes
Copy link

any update here? seems like a must-have feature.

@iamatulsingh
Copy link
Author

FYI, it may not help, but it could give insights. On our side, we deploy everything with ArgoCD:

  • we've added a sync window to prevent automatic sync at the same time that deployments were scaled down

  • we've added a job that

    • Remove sleep info config
    • Remove the sync window
    • Trigger ArgoCD sync on resources in the namespace
  • a job with associated alerting not to forget to add it back anytime

It is not optimal, but it allowed us to ship this feature fast.

@victorboissiere thanks for the insights. It seems a good idea till this feature get added.

@paoloyx
Copy link

paoloyx commented May 29, 2023

@victorboissiere your suggestion looks very helpful, would you mind to better explain the "job" thing? Just because I did not grasp the way you've proceeded. Thank you so much :)

@paoloyx
Copy link

paoloyx commented Nov 20, 2023

+1 for this feature, we're facing a really identical situation, where we need to exit from a sleep for some "urgent" conditions.

I was thinking that your proposal here @#76 could work @davidebianchi i.e. disabling the cluster sleep and leave the SleepInfo resource into the cluster.

If we come up with an agreement on how to proceed it would be not a problem for me to contribute with some PR :)

@davidebianchi
Copy link
Member

So, in your use case is would be ok to change the CRD?
The other way should be to expose an API to invoke, to wake up a specific namespace. But I didn't have time to view the best practice to manage this types of workflow in Kubernetes until now. For sure, it is a future we must add in kube-green!
In the #76, the idea was to disable the kube-green lifecylce and not to wake up the resources (but it could be an idea)

@ziouf
Copy link

ziouf commented Feb 8, 2024

Maybe an annotation on the SleepInfo custom resource (like kube-green/skip: "true" or kube-green/skip-until: "Mon 08:00" or kube-green/force-wakeup: "true") could be watched by kube-green controller and suspend scale down operations until the annotation is removed.

It shouldn't be hard to implement. What do you think about ?

@davidebianchi
Copy link
Member

Hi, for sure it shouldn't be hard to implement. But I think the problem is to have the permissions to change the CRD. But for sure, it could be interesting for the other use cases.

@eldad87
Copy link

eldad87 commented Feb 19, 2024

+1 for this feature 👍

@EvertonSA
Copy link

+1

@siorellana
Copy link

Hi team.
We just implemented this tool and just works fine! Amazing tool and very easy to use.
But now our development teams want to have a way to wake up the environment in case they need to work with it, and I don't find an easy way to do this (except deleting the sleepInfo and deploying the app again).

This feature I think is really powerful, so +1.

@abdulgaffar
Copy link

+1

1 similar comment
@CA1087
Copy link

CA1087 commented May 6, 2024

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests