Framework improvements

[joachimvermeir]

Based on one of the Q&A prep session where the 'units' where discussed I have some ideas for improving the framework, probably they need to be split into different items but I'll share it here to start the discussion.

About 'units'

The way the framework currently works indeed lacks a proper handling of units:

• the term 'unit' is not really clear from the discussion in the Q&A and code base
• (units of) input and output parameters are described nowhere
• the resulting output data file of running the impact framework only contains the values of metrics, but not their units
• when passing metrics between steps in the pipeline there's no way to validate or detect that the values are supplied in the right units

Possible improvements:

• Define 'units' properly:
  • e.g. in /src/config/units.yaml it is quite confusing to see an entry called 'carbon'. Carbon is not a unit, carbon is the metric or characteristic (I'm not a native English speaker so not sure what should be the appropriate term) and its unit is gCO2e. Another example is 'count'/'percentage', that's not a unit, that kind of metrics does not have a unit (they are unit-less)
  • Using the SI units where relevant would improve the readability and consistency (e.g. standard unit for time is 's', not 'seconds').
  • Create documentation for the basic units that are part of the base impact framework and perhaps some guidelines on how to define unit names
• I also liked the idea of embedding the description and properties of the input/output parameters in a metadata file in each plugin. This would indeed allow to
  • perform consistency checks:
    • check if output metric's units match the ones of the expected inputs
    • check if the expected inputs are present
    • check if the plugin did not alter/add any other metric in the output than the ones it described in the metadata
  • perform unit conversions
    • automatically for well known standard cases
    • by using a conversion plugin either with built-in conversions or find a generic solution where you can put the conversion factor
  • Include the metric's units in the output file so that it is clear which units are used for the different values
  • Extend the model interface with all the input metrics and their units that are used up till that step and the output metrics and units, so it can be used by the plugins (e.g. a conversion plugin).
  • Given the known metadata one could also imagine 'cleaning' intermediate data. For example, step 2 in the pipeline expects cpu-util as input to convert into something else. If none of the subsequent steps have cpu-util in their input, it is safe to remove that field from the data (memory/file) for any steps after step 2.

Attention points

• If we would implement the 'units' as mentioned above and use plugins to perform conversions we might end up with a situation where a single model/plugin is used multiple times in a pipeline. If this plugin requires configuration that is different each time it appears in the pipeline, how can we configure it correctly (it would need some kind of step ID to allow it to be referenced)
• implementing a validation that a step (model/plugin) does not alter any output except what is described in its metadata, we won't be able to create generic plugins (like unit conversion) that operate on a (not explicityly defined, but at runtime calculated) subset of input fields. An option is to add a property to the plugin's metadata that it is generic (has no predefined inputs/outputs but does a selection at runtime based on the context/metadata)
• For some of the above points, they can be either implemented in the framework and/or in separate plugins or a combination.

Other improvements

• Add some debugging options: right now it's not possible to get the intermediate results of the different steps in the pipelines without commenting out everything after the step you want to debug. For very large pipelines this is not an ideal way of working.
  • Add command line parameter to activate debugging and/or in the config section
  • Debug would create all intermediate result files for each step in the pipeline
  • Optionally also allow output of that result to be in CSV format (columns having all metrics and units, rows contain the values for the different output entries)

[jmcook1186]

Hi @joachimvermeir - thanks for sharing this idea!

There is a lot of valuable insight here that could make a great hackathon project - I'll get into this properly in a couple of hours and aim to give some more detailed feedback later today.

[jmcook1186]

Some more detailed responses below - your text in bold, mine in italic. Overall onboard with the way you're thinking about this and excited to see what comes out of these ideas!

The way the framework currently works indeed lacks a proper handling of units:

- the term 'unit' is not really clear from the discussion in the Q&A and code base

The unit describes how a given parameter is measured, e.g. gCO2eq, seconds etc

- (units of) input and output parameters are described nowhere

they are/should be described in the docs for each model plugin, but it is true they are not defined in a manifest file. There is a file, units.yml where the expected units for each parameter are defined.

- the resulting output data file of running the impact framework only contains the values of metrics, but not their units

yes this is true - right now the output parameter is expected to exist in units.yml and the unit is looked up there - it is not explicitly reported in the output file.

- when passing metrics between steps in the pipeline there's no way to validate or detect that the values are supplied in the right units

yes this is true - the onus is on the user to ensure the right units are passed into each plugin. A plugin that receives some value will naively assume the value to be expressed in the correct unit.

- Define 'units' properly:
- e.g. in /src/config/units.yaml it is quite confusing to see an entry called 'carbon'. Carbon is not a unit, carbon is the metric or characteristic (I'm not a native English speaker so not sure what should be the appropriate term) and its unit is gCO2e. Another example is 'count'/'percentage', that's not a unit, that kind of metrics does not have a unit (they are unit-less)

sure, but carbon is the parameter name, and it has a unit associated with it, defined in the units field. Are you saying the file name is unintuitive? If so, I don't personally see a problem renaming it to metadata.yml or similar.

- Using the SI units where relevant would improve the readability and consistency (e.g. standard unit for time is 's', not 'seconds').

yes, fair point., We have tried to avoid any short-hand or abbreviations across the repository, preferring to be fully expressive wherever possible. Agree that the default unit should be the SI unit for a given parameter wherever possible.

- Create documentation for the basic units that are part of the base impact framework and perhaps some guidelines on how to define unit names

This would be helpful. Right now we are discussing a new approach to defining units that we intend to implement next sprint, most likely providing a way to append to or override the default units.yml via a command or field in the manifest file at runtime. You can follow and/or contribute to the discussion here. When we have agreed an implementation plan and merged it in, good documentation will definitely be needed. This should be ready to document well in advance of the hackathon, so this could form part of a hackathon project.

- I also liked the idea of embedding the description and properties of the input/output parameters in a metadata file in each plugin. This would indeed allow to

• perform consistency checks
• check if output metric's units match the ones of the expected input
• check if the expected inputs are present
• check if the plugin did not alter/add any other metric in the output than the ones it described in the metadata

I'd certainly be interested to explore this idea. Again, I think it will depend on the solution we implement for units.yml but overall if we can provide some basic checks at runtime that each model is receiving its expected parameters that could be a good thing. We've worked on the basis that the framework itself should be minimally opinionated so that individual plugins are free to impose whatever conditions they require. However, I could see the value in some static analysis that checks that the right input values are available to each model in a pipeline (this would require checking the outputs of the preceding models to ensure the data has been enriched correctly at the point of each model's execution). That could be a really useful feature. Part of this could be unit checking (i.e. does the unit for parameter x in the output of model A match the expected unit for parameter x in the inputs for model B and so on).

- perform unit conversions
- automatically for well known standard cases
- by using a conversion plugin either with built-in conversions or find a generic solution where you can put the conversion factor

Agree, mostly covered in my response to the previous point. It would be cool if there was a way to auto-correct units based on the findings from the static analysis - e.g. if the unit for parameter x in the output of model A DOES NOT match the expected unit for parameter x in the inputs for model B then perform some conversion to enforce a correct flow of data through the pipeline. Honestly not sure what the implementation looks like at this stage, just ideating.

- Include the metric's units in the output file so that it is clear which units are used for the different values

No particular problem with this, but it is already captured in units.yml - I guess it's fine to also show in the output file.

- Extend the model interface with all the input metrics and their units that are used up till that step and the output metrics and units, so it can be used by the plugins (e.g. a conversion plugin).

We will probably be reticent to update the plugin interface as it is supposed to be as minimal as possible. That said, we are planning an update to the interface shortly, with the aim of reducing the amount of redundant data being copied into the output file and clarifying the difference between observations and configuration, which is currently somewhat ambiguous.

We're open to discussing any convincing arguments about how else to change the signature, of course, but there really has to be an extremely strong reason to mess with it, as it will require rewriting all existing models. The particular functionality you suggest may well end up being rolled into our planned updates , so I'd say watch the discussion here and here to see if we cover it sufficiently.

- Given the known metadata one could also imagine 'cleaning' intermediate data. For example, step 2 in the pipeline expects cpu-util as input to convert into something else. If none of the subsequent steps have cpu-util in their input, it is safe to remove that field from the data (memory/file) for any steps after step 2.

yes, I like this idea.

- If we would implement the 'units' as mentioned above and use plugins to perform conversions we might end up with a situation where a single model/plugin is used multiple times in a pipeline. If this plugin requires configuration that is different each time it appears in the pipeline, how can we configure it correctly (it would need some kind of step ID to allow it to be referenced)

Interesting... I haven't really considered multiple invocations in a single pipeline. I suppose this could be especially relevant for plugins that do some manipulation or correction moreso than plugins that do calculations... I'll think on this a bit more, but gut reaction is that it might be worth exploring.

- implementing a validation that a step (model/plugin) does not alter any output except what is described in its metadata, we won't be able to create generic plugins (like unit conversion) that operate on a (not explicityly defined, but at runtime calculated) subset of input fields. An option is to add a property to the plugin's metadata that it is generic (has no predefined inputs/outputs but does a selection at runtime based on the context/metadata)

ok - agree in principle!

- For some of the above points, they can be either implemented in the framework and/or in separate plugins or a combination.

yes!

[jawache]

Thanks, @joachimvermeir. There is a lot here, as @jmcook1186 says; we might benefit from breaking this up into different replies in this discussion topic. I will do that to break up the conversation a little bit. You've got some very refined thinking here @joachimvermeir. Not many of us have thought in such detail about this issue, exciting to see.

[jawache]

RE: Naming

I do agree that units.yml is confusing. Perhaps it should be parameters.yml?

We called it units.yml, but I suppose that wasn't the intention; we just named it poorly.

The intention is to have a clear standard for parameter names. Since we want plugin authors to be able to trust that when there is "CPU-util," for instance, that is going to be expressed in "percentage" rather than "0-1," which is sometimes how it's expressed. These inconsistencies are where we had problems in the past and why we started creating the units.yml standard, E.g., carbon is expressed in gCO2e; one thing that's caught many people by surprise is that some tools and services will express carbon as lbsCO2e instead! As you can imagine, it has caused many problems, so we forced everything to use gCO2e instead.

[joachimvermeir]

@jawache the filename could be parameters, metrics, characteristics of something similar.

[joachimvermeir]

@jawache
For the percentage, if you indeed want to show value 90 instead of 0.9 it makes sense to link a unit to it (unit would be 'percent', not percentage), but then again percentage is just another way of displaying a value. Why then choose percentage, and not 'one per mil' or a ratio (4/5) or even hexadecimal representation? Does it make sense to apply formatting/display on these numeric values or should this be applied later on when you visualize the result of your calculation in a dashboard? All intermediate values/calculations are for computers, not for humans to read/process.

[joachimvermeir]

@jawache
In the current units.yaml you can find unit 'count', that is also very strange, I would expect no unit for these parameters:

total-resources:
description: total resources available
unit: count
aggregation: none

[jawache]

percent sounds good to me!

and I agree count doesn't feel right, you should expect to see an integer for that parameter.

The intention is to give information to plugin authors about what to expect when they see this field, perhaps type is a better term that expresses the intention of that field?

[jawache]

Btw you might want to read this proposal I submitted for a proper naming convention for the whatever-we-will-call-it.yml
Green-Software-Foundation/if#410

[jawache]

RE: Unit Conversion

Our thinking is for there to be a healthy ecosystem where writing plugins is simple then each parameter needs to be unique, and the units cannot change. Each plugin author should not worry about what units a parameter is expressed in. If you want to express the same parameter in a different unit, the parameter used needs to be different, so if you wanted to express carbon as lbs, you would have to export a parameter called carbon-lbs and not overload the carbon parameter name thus forcing every other plugin author to have to be wary regarding what carbon means.

If that's the case, then there can be a plugin that performs conversions, but so as not to cause confusion downstream, those conversations would need to produce a different parameter. So there can be a conversion plugin that say converts carbon to carbon-lbs (or anything to anything-unit).

To speak to your point on multiple of the same plugins in the same pipeline, I've thought of this before and the way you would do that right now would be to configure the same plugin multiple times in the initialization section. The name of the plugin that you use in a pipeline is just an id that refers to an instantiation of a plugin. I think it's nice for the plugin names to be descriptive of the task they are performing.

initialize:
  models:
    - name: carbon-to-lbs
      model: GenericConversionModel
      path: "path-to-your-model"
      config:
         parameter-from: carbon
         unit-from: gCO2e
         parameter-to: carbon-lbs
         unit-to: lbsCO2e
    - name: energy-to-joules
      model: GenericConversionModel
      path: "path-to-your-model"
      config:
         parameter-from: energy
         unit-from: kWh
         parameter-to: energy-joules
         unit-to: joules

Then in your pipeline, you can perform conversions like so:

  pipeline:
    - teads-curve
    - sci-e
    - sci-m
    - sci-o
    - carbon-to-lbs
    - energy-to-joules

(FYI, as I wrote the above, I realized that sci-e, m, and o don't meet my statement that plugin names need to be descriptive!)

[jawache]

@joachimvermeir and @heaversm so "post-hackathon" my personal views on this have aligned more towards @joachimvermeir's original proposal.

Seeing the way people have struggled to connect the outputs of one plugin to the inputs of another, my personal challenges in renaming all our internal plugins to adopt the standard proposed here Green-Software-Foundation/if#410 then thinking through how best to resolve this in a way that is a "standard-for-everyone" has honestly been much more challenging than I initially thought. Perhaps we just let this play out in a more freeform way and then if there is a call for standardization from the community later on we can weigh in and try to help.

We're going to start an epic discussion in the repo about inputs and outputs more broadly, I was going to propose something along the lines of what @joachimvermeir was proposing above.

Plugin Metadata returning Inputs/Outputs

• Either plugins return metadata about the inputs and outputs in the metadata object (downsides: this means you have to instantiate the code to return the metadata, plusides: this offers some dynamism since then the params can depend somewhat on the global config)
• Alternatively perhaps move metadata to a static file in some known place in the repo, that way you can do offline static analysis.

Something like this perhaps?

inputs:
  parameter-1: 
   aggregation: sum
   description: a description of this param
   unit: What unit is this value expected in, g, Kg etc...
  parameter-2: 
   aggregation: sum
   description: a description of this param
   unit: What unit is this value expected in, g, Kg etc...
outputs:
  parameter-1: 
   aggregation: sum
   description: a description of this param
   unit: What unit is this value expected in, g, Kg etc...

Plugin Configuration in Initialize Block

initialize:
  models:
    - name: name-of-your-plugin
      model: GenericConversionModel
      path: "path-to-your-model"
      config:
      mapping:
           the-parameter-name-you-expect: the-parameter-name-being-passed-in

When you config a plugin, if the parameters being passed in are the same as the plugin expects then you need to do nothing. If the name of the parameter is different from what the plugin expects then it maps the input to the name it expects.

I suspect the config will need to be more detailed to aloow for defaults, unit conversions things like that but roughly, some config in the metadata to return inputs/outputs, some way to map actual inputs from the pipeline to the expected input in the plugin.

Thoughts?

[jmcook1186]

Glad to see this back in discussion - pretty much echo'ing @jawache but here's my 2c nonetheless:

There is definitely a tension between enabling permissionless innovation on top of IF and also imposing standards at the level of plugin parameters. We've gradually softened the requirement to conform to what is now params.ts to provide more flexibility, but I'm inclined to remove it altogether and instead implement a scheme where units are defined in each plugin's initialize block and then some analysis is done on the pipeline at runtime to ensure there are no discontinuities in the units passed from one plugin to the next - in short agree with @joachimvermeir and @jawache suggestions above.

We do currently rely on params.ts to configure the aggregation feature - we need to define somewhere which method to use when a given parameter is aggregated over time/components. This could also happen in the initialize block, though.

In terms of imposing standards, I think the right way is to provide detailed docs that make the expected units for plugin params exceptionally clear, publish prebuilt pipelines that people can iterate on, and write validation code that warns users when something doesn't look right, but not to actually enforce adherence to a set of standards programmatically.

[heaversm]

I'd be fine with whatever approach you choose, the main things of importance to me are the ability to, wherever things get defined, have a dependable location to go to to find what you can expect as an input / output, with its units, regardless of whether you choose to standardize those units or leave it up to the plugin developer - and then some mechanism in place to try to keep plugin developers to the standard so that you don't get a bunch of plugins where this information is missing, or scattered somewhere in the readme or code comments or elsewhere.

[jawache]

@heaversm agreed, given what you just wrote and how I think this might be implemented, I think the only solution is that the output manifest file needs to have a node which describes the outputs. So basically append the units info to the manifest so the manifest has everything embedded in it. The data, but also the explanation of what each field means, it's unit, perhaps even which plugin outputted that unit.

[heaversm]

Yes! This sounds ideal to me

[jawache]

RE: Debugging

That's an excellent point; something I've thought of before is a CLI param that exports each step in the pipeline as a separate file so we can see how the pipeline evolves the data. More generally, we need a better developer experience, which is a great step.

What other improvements could we make to the developer experience?

[josh-swerdlow]

Some ideas from atwoosnam and my ideation session on the carbon hack (none were our final submission).

On Joachims point for making the units of observations attached as metadata, we thought to make an intellisense plugin to extract relevant unit/descriptive text from local plugin descriptions, parameters, unit files and associate it with the manifest keywords. This was a compromise between changing the manifest to add this metadata (seemed onerous to us and not within the scope of hackathon) and leaving it as is. It's effectively automating the manual process one needs to currently do (looking at the units, plugin, etc files to understand what it does or what it is).

[josh-swerdlow]

We had discussed creating a plugin that will auto-install and link any missing plugins for you by searching at least the if-official and if-unofficial repositories for you. That was a bit manual. I'm unsure how automated this already is, but I've seen some posts about it. I may have just no figured out how to do this.

[josh-swerdlow]

Another idea we discussed was to create a "bookend" plugin that rather than write a standard editable manifest file, might create a much more rich annotated manifest file meant to be sent to friends to check your work. As such, it should have all the info one needs to verify your calculations (at least dimensionally) etc. We also thought it may help to make this file non-writable so you don't accidentally edit it as we all have while using this tool.

[josh-swerdlow]

Like Joachim mentioned, we also had concerned over the ability for one to send any numeric input to any plugin without guaranteeing that the operation is dimensionally correct. We foresaw a lot of users sending each other manifests files only to find out they tried to convert something that dimensionally makes no sense and then had to debug it. Andrew and I discussed making a custom class of assertions for zod to check units. I haven't used Zod, but since Zod is the current parameter validation library used we thought that would be the relevant entry point.

[josh-swerdlow]

Had a lot of similar ideas come up while ideation for the carbon hack. Posted relevant ones in the appropriate thread. I would be interested in circling back on some of these post-hack. Especially on the dimensional consistency of conversions.