Expect FOCUS columns in reporting script and dashboard (was add support for Azure format)

[jnioche]

The test datasets from https://microsoft.github.io/finops-toolkit/open-data can be used as a starting point and enriched with SPRUCE.

The scripts in /reporting will need adjusting accordingly

[jonperron]

@jnioche if you need help on this one, I can help you with it :)

[jnioche]

@jnioche if you need help on this one, I can help you with it :)

sure, @jonperron! thanks

[jnioche]

What might be more robust would be to copy the columns that are CSP specific to those used in FOCUS. We already do something similar for region. We are planning to support FOCUS anyway. Having to support CSP specific columns would not be maintainable in the long run.

[dpol1]

Hi @jnioche,

I did some architectural brainstorming with Claude regarding this SKU pivot, and a really interesting point came up.

I 100% agree on moving to SKUs. It's the only maintainable path for FOCUS. But we should be careful about what we map the SKU to.

If we create a static SkuId -> kgCO2eq mapping, we effectively freeze the Ember grid intensity and PUE into the data. We'd lose our dynamic spatial/temporal precision and have to rebuild the entire lookup table every time CCF or Ember updates.

Alternative idea: A Hardware Profile Registry
What if we map SkuId -> Hardware Profile (vCPUs, RAM, Disk type) instead of final impact?

The flow would be:

• Ingest FOCUS report (SkuId, Region, Quantity).

• Lookup table gives us the Hardware Profile.

• We pass that profile down into our existing Spruce engine to calculate the dynamic impact.

This keeps our dynamic carbon math untouched. It also solves a practical issue: AWS/Azure Pricing APIs don't expose power draw (Watts) anyway, so we still need a clean layer to join against Boavizta/Teads coefficients.

Let me know what you think of this separation! If it sounds like a good path, I'll give a hand script the API scraper to build these base hardware profiles.

[jnioche]

thanks Davide, the discussion here is not so much about how to proceed with handling FOCUS but more about bridging the CSP specific columns to 'neutral' FOCUS ones so that we don't have to handle the complexity of the different formats in our downstream scripts and dashboards

[jnioche]

to answer your point though: see #234

and yes, the idea is very much to only map from SKU to energy user per hour + embodied emissions per hour
and keep the downstream as it is now: generate water and carbon using the existing modules

Have a look at the branch https://github.com/DigitalPebble/spruce/tree/focus-aws - still work in process but will give you an idea

[dpol1]

Ah, got it! I zoomed out a bit too far and jumped straight into the enrichment phase (#234), but I did on purpose since that's where we are heading anyway.

+1 for the immediate step of cleaning up our parsers and dropping the custom x_ CSP columns, bridging to the standard FOCUS schema makes perfect sense.

I'll keep the hardware profile ideas warm for when we tackle the actual SKU-to-impact pipeline.

[dpol1]

to answer your point though: see #234

and yes, the idea is very much to only map from SKU to energy user per hour + embodied emissions per hour and keep the downstream as it is now: generate water and carbon using the existing modules

Have a look at the branch https://github.com/DigitalPebble/spruce/tree/focus-aws - still work in process but will give you an idea

I'll check out the focus-aws branch to get a better feel for it, thx

[jnioche]

+1 for the immediate step of cleaning up our parsers and dropping the custom x_ CSP columns, bridging to the standard FOCUS schema makes perfect sense.

this is not quite what we are doing. the x_ columns are in FOCUS, we don't and won't handle them. What we are discussing now is simply adding another step in the pipelines for AWS and AZURe to create new FOCUS compatible columns for things like cost, user tags etc.. basically the column names we use it the reporting scripts and dashboard.

is this clearer?

[jnioche]

I'll keep the hardware profile ideas warm for when we tackle the actual SKU-to-impact pipeline.

I don't think we want to go down to hardware level - also because in some cases it will be about volumes of storage, networking etc... just SKU to impacts

[jnioche]

I'll create tickets for the generation of FOCUS compatible columns for AWS and AZURE then use those in the reporting resources + documentation