[Tracking, Green Reviews WG] Falco as first CNCF project to measure by Green Reviews WG

[guidemetothemoon]

Description

Collaboration between TAG and Falco has been discussed for a while now and Falco maintainers have done an introductory presentation of the tool at one of the recent TAG project meetings. Collaboration request is tracked here: #140

Now that Green Reviews WG has started working on the design doc for implementation of sustainability footprint measurement framework in #182, we would like to go ahead with Falco project as the first CNCF project to apply this measurement framework towards.

This is a tracking issue to gather necessary input from the Falco project maintainers as well as the overall community in order to be able to successfully perform measurement with the respective implementation created by the WG.

Potential points that the input is required for:

• Walkthrough of the technical implementation/architecture of the Falco project, including detailed walkthrough of how eBPF is used;
• We're aiming for using Software Carbon Intensity (SCI) specification for measurement and therefore need to define following aspects in relation to Falco project:
  • What is Functional Unit?
  • What is Software Boundary?
• What kind of relevant metrics is already gathered by Falco? There were some thoughts on looking into ways to better track system resources impact - was there any development in this area since last time? Ref. [UMBRELLA] Falco collaboration with CNCF tag-env-sustainability falcosecurity/falco#2435 (comment)
• Input for creating meaningful load tests for Falco
  ...

Outcome

A document or a section in existing WG document that clearly defines what, why and how to measure the Falco project that we could use as the basis for testing the implementation of sustainability footprint measurement framework.
...

Co-contributor: @nikimanoledaki

[leonardpahlke]

fyi @incertum

[incertum]

Really excited about this next step, thanks @guidemetothemoon!

to gather necessary input from the Falco project maintainers as well as the overall community

Here are some initial requirements that we believe would be helpful. We are looking for feedback so that we can adjust them to something that everyone agrees is good to go.

Hard Requirements

• Linux kernel >= 5.8
• Default kernel configurations.
• Enable libbpf stats (/proc/sys/kernel/bpf_stats_enabled).
• Test Falco's default configuration with each kernel driver {kmod, bpf, modern_bpf}, totaling 3 runs per release.
• Continuous 24/7 testing to observe Falco's long-term performance.
• Green Reviews WG infra will send Falco's own internal metrics (CPU, memory, and counters) and Kepler metrics to The Falco Project.
• Green Reviews WG will support traditional SRE metrics + energy, so infra people can relate intuitively.
• Green Reviews WG will include metrics reflecting server load, nature of workloads. Falco's kernel-side & userspace event counts & tracepoint invocation counts could help.
• Define the performance criteria for Falco that would be considered "green" by the Green Reviews WG, such as the acceptable SCI index, or traditional CPU and memory usage, taking into consideration what Falco does.
• Define tolerance for regression measurements between releases, considering that synthetic workloads evolve over time and Falco's resource utilization depends on the server workload.

Nice-to-Have

• Monitor Falco in 1-2 alternative configurations per release.
• Monitor the impact of Falco on other system applications.

Non-Requirements

• Flexibility in Linux distribution choice.
• Flexibility in infrastructure setup for retrieving results and internal metrics from Falco alongside Kepler metrics.

Walkthrough of the technical implementation/architecture of the Falco project, including detailed walkthrough of how eBPF is used;

Website and repos readmes are the go-to places for information. If the documentation is not clear or complete, we will make it better.

https://falco.org/docs/

https://github.com/falcosecurity/falco
https://github.com/falcosecurity/libs (Falco's underlying libs and kernel drivers)
https://github.com/falcosecurity/rules

What kind of relevant metrics is already gathered by Falco?

https://falco.org/docs/install-operate/production-performance/ (Falco internal metrics)
and https://github.com/falcosecurity/falco/blob/master/falco.yaml string search for metrics: config.

Input for creating meaningful load tests for Falco

Start simple with readily available synthetic workloads, as @catblade suggested, see here. We can become more sophisticated and deliberate over time.

• We're aiming for using Software Carbon Intensity (SCI) specification for measurement and therefore need to define following aspects in relation to Falco project:

  • What is Functional Unit?
  • What is Software Boundary?

We want to learn more so that we can be helpful.

---

On behalf of the Falco maintainers, I would like to tag @FedeDP and @Andreagit97 for their instrumental role in setting up the infrastructure for our new Falco kernel version testing. It sounds similar to the suggested setup here: https://github.com/falcosecurity/kernel-testing.

I would also like to tag @leogr, @LucaGuerra, and @jasondellaluce for their general tracking and making sure that the tests are well-designed and that the results are accurate.

[guidemetothemoon]

Thank you for the feedback, @incertum ! We will start looking into the information you've shared and will reach out to the respective stakeholders that you've tagged here for additional information as we progress.

[andrew-est]

I think using Falco as a first application to demonstrate Green reviews is a great idea. Its complicated enough to be worthwhile but is not so huge as to be too problematic. We need an application like this to be taken seriously.

However I think we need to think of some other demonstration application for another reason. We want to roll out the concept of Green Reviews and Sustainability measuring to many different communities/users. In order to do this I think it would good to have a really simple project that is totally under the control of the Sustainabilty TAG. This "project" would be a hello world level application that had a simple easily defined "Functional Unit" e.g. a HTTP GET. This application would be installed using a very simple Helm Chart etc. We would set up the github actions etc to demonstate the flow of the Green review. The advantage of this over Falco is that it can be kept very simple. The deployment can be a single POD with default configuration paramaters that work out of the box. This can be used to show the "flow" of the testing/reporting for any application. The disavanatage is that it isn't a real application so will not get the sustainability TAG any crediablity which is why I think the demo APP is needed in addition to rather than instead of Falco. This is planning for the sucess of the "Falco" work where we want to roll out to more users and have a simple always working example to show them.

The objective is that if somebody wants to see green reviews working that they can see an end-to-end flow of it working in their own/our environment within a couple of hours. The idea is that we give users the instanst feedback/demo so the they engage more and the don't get put off by the barrier of having to spend an few hours understanding/installing Falco as that has the potential to just be a barrier to entry (Thats not a reflection on Falco as their documenation/instructions are excellent)

Just my thoughts.

[guidemetothemoon]

@incertum @FedeDP @Andreagit97 @leogr @LucaGuerra @jasondellaluce things are moving and we have just received access to the infrastructure on Equinix that we can use to set up and test the measurement workflow we're going to create. In the coming weeks we will start doing doing more concrete tasks and as we progress it would be beneficial to have someone from the Falco maintainers team join our working group meetings.

Since we will be tightly cooperating with Falco going forward I think that having one or multiple maintainers in the meetings would help us collaborate more efficiently and help clarify questions or provide input that may come up on both sides.

Working group meetings take place bi-weekly, every 2nd & 4th Wednesday, 5pm GMT +2:00, 8AM Pacific, 11AM Eastern time. Meeting details: https://github.com/cncf/tag-env-sustainability/blob/main/working-groups/green-reviews/README.md#meetings-and-contact

Will someone from the Falco team be able to join the Green Reviews WG meetings going forward? Next meeting is happening Wednesday, 27th of September and it would be great to have you there!😊

[incertum]

@guidemetothemoon perfect! We will coordinate internally to have the relevant people from our side join (I won't be able to join the 27th meeting because I am taking PTO, but I will join the following meeting).

Would it be possible to define an agenda beforehand so that we can be more specific and effective during the meetings? For example, very broad questions about what Falco is and how it works could be better answered by sharing the relevant official documentation, while the meetings could be used for concrete progress and intermediate results updates, as well as defining next steps.

[leonardpahlke]

@guidemetothemoon @nikimanoledaki, do we have a list of questions and an agenda for the Falco team before the meeting?

[guidemetothemoon]

@incertum I totally understand your point regarding having concrete questions. Unfortunately we didn't manage to get enough time to prepare the questions for the meeting happening tomorrow, but we will use that meeting to get the questions outlined so that we can be ready for the WG meeting happening next time, i.e. on the 11th of October. Does that sound good?
There's no requirement for someone from Falco team to be present in every meeting, but maybe we can have a contact point where we can coordinate which meetings we need your presence at?

[incertum]

Proposal meeting agenda: Wed Oct 11th 8:00am - 9:00am (PDT)

• Review the proposed requirements in [Tracking, Green Reviews WG] Falco as first CNCF project to measure by Green Reviews WG #183 (comment).
• Define a roadmap for Falco CI integration, with concrete goals as follows:
  • Set up communication between the Falco Security organization and the CNCF Green Review server
  • Deploy and run synthetic workloads
  • Deploy and run Falco according to the initial test matrix
  • Retrieve performance metrics (Kepler metrics and Falco's own internal metrics)
• Establish a timeline for completing e2e v1 -- January 2024 may be realistic

Resources:

• https://github.com/cncf-tags/green-reviews-tooling
• wg green reviews project board: https://github.com/orgs/cncf/projects/10/views/12

[guidemetothemoon]

Thank you for adding the proposal for the agenda tomorrow, @incertum!

There are some good points here, but I have extended this a bit more also based on the required clarifications around the infrastructure we have available to build our tooling and respective requirements from the Green Reviews side, as per #182. Also there are clarifications that need to be made around testing and metrics for calculating the sustainability footprint of Falco (for instance functional unit). We may not have enough time to cover it all, but we'll see how it goes and follow-up other point in the subsequent meetings and async.

I have combined your suggested agenda with the active points from our side and added it to the Green Reviews WG Meeting Notes.

[guidemetothemoon]

@andrew-est thank you for sharing your thoughts around this topic.

I do understand your point, but I think that having a blueprint or a demo project that anyone could just pull and test with the tooling we will create is beyond the current scope of the working group. Green Reviews WG has a specific goal which is to measure sustainability footprint of CNCF projects only. Measuring any other projects outside of CNCF landscape is currently out of scope. You can think of it as an internal CNCF initiative to improve the efficiency and sustainability of the projects that CNCF governs that have quite a huge impact in the broader picture since these projects are being adopted by thousands of projects and companies globally and at scale. Due to this goal and current scope of the working group, using the time to create a generic demo project may not be very efficient or useful at this point, since we have existing, production-level CNCF projects that we can collaborate with to ensure that what we're building is correct, realistic and works according to our vision and expectations.

WG members who will be contributing to building Green Reviews tooling may need to familiarize themselves with some basics of the tool we will be measuring, but I don't think that there will be a need to gain a very deep knowledge of the respective tool. Since we have a direct dialogue with the maintainers of the tool we can get help, guidance and even blueprints from their side that we can apply directly in our implementation. For example, scripts for testing the deployment of the tool or a testsuite that we can use to trigger the load on the tool to retrieve necessary measurements.

When it comes to external users that may be interested to test out the Green Reviews tooling I think it's a little bit too early to include it into the scope. If our implementation proves to be successful with a few CNCF projects to begin with, we can consider the next milestone of integrating it for all CNCF-governed projects. A potential milestone after that can be to look into making this implementation re-usable for any cloud native project outside of the CNCF so that a broader community can gain value from it, but that's much further into the future. At that point it could be beneficial to provide some kind of a testing framework that anyone can set up to get hands-on experience and understanding of the tool, as you mentioned. But once again, it's a far more distant milestone that we would need to consider once we get that far.

@andrew-est I hope that this can clarify it a bit more for you, but please let me know if you have any remaining questions or concerns☺️

[nikimanoledaki]

Update

Hi @incertum! 👋 We have made progress with unblocking the cluster and application provisioning. S3 access will unblock the cluster provisioning PR. 🤞

The next step is to gather the deployment manifests for Falco. Were you able to make progress with the repo that will contain these manifests? :)

For now, we are focusing on manual tests so we will initially deploy it with Flux (see this issue) by pointing to the deployment manifest.

• Meeting notes where this was discussed
• Suggested name: https://github.com/falcosecurity/cncf-green-review-testing

[incertum]

Hi @nikimanoledaki thanks for sharing updates ❤️!

I have slated this work for mid-December. However, I still need to coordinate with the other maintainers to create a dedicated repository (e.g. https://github.com/falcosecurity/cncf-green-review-testing).

Since you use Flux it may be easiest to try Flux to deploy Falco as daemonset as well. I'll check it out.

[incertum]

https://github.com/falcosecurity/cncf-green-review-testing is up and if ok going forward we would like to track clarifications and resolve open questions in our dedicated issues https://github.com/falcosecurity/cncf-green-review-testing/issues.

Thank you in advance!

[nikimanoledaki]

Closing this issue since we can track specific work in Falco's repository https://github.com/falcosecurity/cncf-green-review-testing/issues & the WG repository https://github.com/cncf-tags/green-reviews-tooling/tree/main/clusters