GitOps impacts on environmental sustainability

[scottrigby]

• WG meeting notes on this topic in this Google Doc
• See https://github.com/cncf/wg-env-sustainability
• Our question in this group is what are and what could be the impact of GitOps on environmental sustainability?
• We have a GitHub team for working on this topic @open-gitops/gitops-environmental-sustainability to research and collaborate on a white paper for https://github.com/open-gitops/documents. This is open to anyone to contribute

[williamcaban]

Some topics for consideration for this:

• Optimizing for power consumption vs optimizing for carbon emissions can be two competing goals for an organization. What should be considered on a GitOps architecture when optimizing for one vs the other?
• Metrics (e.g. pwr consumption of a particular software) and metadata (e.g. power source type, CO2 emissions/SCI score) for optimizing GitOps practices towards sustainability

_W

[nikimanoledaki]

For your first point, I see "optimizing for power consumption" and "optimizing for carbon emissions" as two paths to the same goal. The main tradeoff is the opportunity cost of focusing on one versus the other. We could draw a Venn diagram for solutions that address each or both objectives.

It would be valuable to define, unpack, and provide examples for each optimisation. Maybe that way we will arrive at some that address both.

optimizing for carbon emissions

One example (which we discussed elsewhere but I'd like to note here) is moving workloads to a "greener" region. For example, moving CI to AWS eu-north-1, which runs on a grid that emits fewer emissions. That optimises for carbon, but does not optimise for energy consumption.

[cwstrommer]

I would prefer to focus on GitOps-specific approaches or challenges to environmental sustainability:

• Using your example, switching to a "greener" region alone would not be worth discussing because this can be done even without GitOps. On the other hand, having a declarative list of green regions maintained in SCM AND having a GitOps process for automatically checking whether workloads are optimally distributed among greener regions and reconciling them if they're not would be worth discussing.
• Conversely, it would be worth comparing the energy savings of GitOps (such as the above scenario) to the energy costs of continuous reconciliation, and use that to determine the inflection point where we can say that continuous reconciliation definitively provides better or worse sustainability.

[nikimanoledaki]

I agree with your note on scope. For context, the question of which AWS region is "greener" and based on what data is something that we have been exploring over at the Environmental Sustainability WG. The data is limited so any list of relatively greener AWS regions would require a fair bit of research, including verifying data, comparing regions, and assessing any externalities of moving infra to these regions.

I like the process that we just demonstrated: provide a general use case for optimizing for energy and/or carbon (bearing in mind that a lot of these processes/tools may be novel and evolving quickly) and then provide an example of how to do this in a GitOps way.

Your 2nd point is an excellent idea and example of what to expect to find in this whitepaper. How we would measure things is an (extremely) interesting question.

We are doing a landscape assessment as part of the EnvSus WG, here is the WIP doc. The Software Agent section has some options.

[JaimeMagiera]

Lots of great ideas. I see the Sustainability WG as the one that would come up with the more generalized ideas (rating a power region), and our group demonstrating how to move to that region oin a GitOps fashion. As an example: Describing a process of querying a datastore which contains the sustainability scores of regions and updating your repo with the the Terraform and cluster install/config to act accordingly – all in a GitOps way. A higher resolution example would be node placement based on such scores, as it may not be possible to move entire clusters to a particular region (and somewhat ill-advised for HA).

I too like the question of how we measure.

[nikimanoledaki]

Here are some ideas, suggestions, and questions for the whitepaper.

Mission

As a group of sustainability-focused folks who strive to continuously improve the tooling that we use and promote, we want to critically evaluate GitOps as a practice in terms of its environmental impact.

Title

I suggest editing the topic/question slightly to: “What is and what could be the environmental impact of GitOps?” - open to suggestions :)

Given this question, we would need to define what we mean by “environmental impact”.

Structure

@JaimeMagiera proposed an outline in a previous meeting.

Attendees added some ideas, including:
- 1. Chris Short: increase awareness
- 2. Niklas: Anti-patterns
- 3. Nishad/Niki: models and research

As Jaime also suggested in that meeting, we should make sure that we link things back to the GitOps Principles. My suggestion is that we use these 4 principles as starting points. We can list the pros, cons, and use cases for each one. Once these pros/cons/examples are identified, metrics can follow to validate what we wrote. Recommendations can then be added to reduce the "cons". The “what could be” part of the question is meant for recommendations and improvements for the future.

If folks are happy with this structure, we can be creative with how we brainstorm. I can create a new discussion (like this one) for each Principle so that we can discuss each one with pros/cons/examples/etc. We could also just add more "replies" in this thread to keep the context in one place.

Collaboration

The regular GitOps WG meeting is quite late in the evening for folks in Europe (9-10pm 🙁). We might also want to not wait for bimonthly meetings to progress. For the purpose of this project, we could have an offshoot task group that meets as often as necessary. Is there a precedent that we can follow here?

We could also make some noise in the Environmental Sustainability WG to get some eyes and perspectives from people from that group. There is a lot of overlap in terms of people in the groups anyway, including @JaimeMagiera, @williamcaban, @scottrigby, and me.

Writing

We could start this in a google doc or in Markdown through something like HackMD - thanks @scottrigby for the suggestion! Is there any preference for one or the other?

[JaimeMagiera]

Thanks for accurately summarizing my thoughts. I've been alternately on vacation or very busy at work the past month.

I'm a fan of HackMD, since the docs can be published to a repo.

[nishadmehendale]

I have tried to collate this discussion, added my views and added a breakdown for targeting the Environmental Impact of GitOps in the following ways:

• Comparison with traditional CI/CD
  • I believe for clients/users adopting to GitOps are usually unaware of the impact of existing solutions on Sustainability
  • Due to this I believe a comparison with the traditional CI/CD approach would offer them a good starting point
  • Maybe if we can come up with the tool to do this assessment/use the one’s already out there would be beneficial
• GitOps Providers
  • I am talking about providers such as FluxCD or ArgoCD here
  • Understanding their environmental impact:
    • I think this is being carried out already as part of the EnvSus WG as shared by Niki
    • Can we ask the providers to make it mandatory to provide Metrics (e.g. pwr consumption of a particular software) and metadata (e.g. power source type, CO2 emissions/SCI score) for optimising GitOps practices towards sustainability included as part of their releases?
    • As far as I remember FluxCD allows individual components to be installed and only enables the ones that you need (I confirmed with Niki). ArgoRollouts can be installed additionally if needed. Can we ask to follow this pattern for all GitOps providers? As I see the same request has been raised for crossplane Add ability to disable certain CRDs on installation. crossplane/crossplane#2869
• GitOps Practices
  • Here I think we need to recommend practices that we have derived(or will derive) while implementing a sustainable gitops solution.
    • This can include the following things
      • Structuring repositories and their impact
      • Use of XYZ GItOps provider and its impact
      • Use of certain branching strategies and their impact.

I hope we find this helpful and feel free to correct me where I am wrong.

[scottrigby]

Yeah, I was thinking for topics that could relate to GitOps principles:

1. there are metrics being addressed elsewhere (carbon usage, etc)
2. policies (around those), so they can be declarative
3. the ability through GitOps to turn IT off when not needed

[niklasmtj]

Just a quick thought about 3:
when new commits come in just from "9-5" why not stopping or reduce the reconciliation intervals when "not needed"?

[scottrigby]

@niklasmtj I like how you're thinking! A cool feature could be to support policies to reduce reconciliation intervals to match SLOs if those change on certain days or hours. Conversely – for a production env that should be accessible to end users at any time of day – reconciliation should consistently reflect that too.

Note I mention SLOs because continuous reconciliation isn't only needed when developers are making intended changes in the desired state declarations. Drift can happen outside work hours, at any time, for various reasons (miscellaneous failures, bad actors, etc). So I'd say the usefulness of this feature would depend on the SLOs for the project/environment.

What do you think?

[niklasmtj]

That sounds good! Especially the point about the SLOs. Keeping the reconciliaton "on" in production is a must! But for example staging environments could go in "hibernate mode" outside of the business hours in my opinion. Sure drift can still happen there but the reconiliation could start a couple minutes / hours (depending on the environment) to get everything in sync before the devs start working on these enviroments.

But yeah that is definitely something to be discussed further. I think that it should be a (long term) goal to reduce the amount of power and compute resources to a minimum when it's not needed. Production enviroments excluded from that in the first draft. Thinking about those is a little more complicated I guess.

[nishadmehendale]

Does this mean we update "Continuously Reconciled" principle?

[nikimanoledaki]

Couldn't agree more with the above. Development and staging environments could definitely be a focus here and benefit from things such as:

• less frequent reconciliation by increasing intervals
• cron-like scheduling for reconciliation (e.g. "9-5" as @niklasmtj said, exclude weekends, etc.)
• more frequent use of commands that suspend/resume reconciliation altogether

This would be like "switching off the lights" at the end of the day.

@nishadmehendale I don't think that things change when it comes to production environments but maybe we could elaborate on dev and staging environments if we decide that things can be different depending on the context.

On a slightly different but related note, David Mytton writes an awesome newsletter on cloud sustainability and recently wrote a piece on "Sustainable dev environments in the cloud". It has made me think a lot more about how we can improve our dev processes, CI, etc. :)