Emissions CSV export

[aaccensi]

Context

As part of the emissions project we want to be able to export the emissions data in a CSV. The existing ConfiguredCSVSerializer serializes graph data to CSV based on a YAML config in ETSource, but it only supported simple query-based columns. The emissions export requires iterating over nodes in a group and reading attributes directly from each node, including support for both the energy and molecules graphs.

Implemented changes

Expanded ConfiguredCSVSerializer to support two new column types:

• node_group: A hidden column whose value names a node group. The config row is expanded into one CSV row per node in that group, combining nodes from both the energy and molecules graphs.
• node_attribute: Evaluates a Ruby expression against node_api for each expanded node. Supports an optional transform: expression (with value bound to the result) for post-processing, e.g. unit conversion or conditional mapping.

Other changes:

• Added an optional period: parameter (:present or :future) to the serializer to select which graph to use for node expansion. Existing column types (present, future, unit, query) are unaffected.
• Fixed label: being silently ignored for non-query column types.
• Added emissions_present and emissions_future CSV endpoints backed by a new emissions_csv.yml config in ETSource.

Related

This effort is in 3 places but only 2 of them have PRs (to merge back into the general emissions branch)

• etengine: Emissions CSV export #1753
• etsource: Emissions CSV export etsource#3443
• etmodel: No PR, created emissions branch that didn't exist yet

Checklist

• I have tested these changes
• I have updated documentation as needed
• I have tagged the relevant people for review

[noracato]

Very elegant solution with the transform attribute. Nice find!

Specs look good as well! Great work

[kndehaan]

@noracato @aaccensi we updated the direct_emissions_csv.yml on ETSource to contain all the relevant groups. Robin also updated the spec fixture file on ETEngine. Does this spec file indeed need to be updated as well or should we undo this commit?

[noracato]

ETEngine should generally only be touched by devs. The specs are fast internal tests to see if the code is behaving correctly under different cases and circumstances. I will revert the commit for you.