Emissions

[noracato]

1. Multi-Index CSV Support

Introduces a new CSVDocument::MultiIndex class for handling CSV files with multiple index columns (e.g., sector, subsector, use, ghg). The data is flattened into a hash for efficient lookups by ETEngine.

Example structure:

etm_sector,etm_subsector,use,ghg,unit,value
Energy,Electricity production,energetic,co2,kg,123.4

2. EMISSIONS() Runtime Function

New GQL function for querying emission data from datasets:

EMISSIONS(agriculture_non_specified, non_energetic, co2)
# => returns emission value

3. Multi-Line Array Parsing

Extended the ActiveDocument parser to support multi-line arrays in .ad files, enabling cleaner formatting of long array definitions:

- attribute = [
  - value1
  - value2
  - value3
]

4. Dataset Emissions Accessor

Added dataset.emissions method for accessing emissions data with 4-index-column support.

5. EnergyNode Attribute

Added co2_utilisation_per_mj Float attribute to EnergyNode for tracking CO2 utilisation.

Goes with:

• Emissions etengine#1754
• Emissions #189
• Add direct emissions method with data export etsource#3444
• Add direct emissions method with data export etmodel#4719
• https://github.com/quintel/etdataset/pull/1087

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.99%. Comparing base (923f382) to head (90fb02b).
⚠️ Report is 6 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #189      +/-   ##
==========================================
+ Coverage   92.78%   92.99%   +0.20%     
==========================================
  Files          91       92       +1     
  Lines        2635     2712      +77     
==========================================
+ Hits         2445     2522      +77     
  Misses        190      190              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[louispt1]

Based on the altered csv structure, I've updated the branch slightly - I will explain further noline @noracato but basically the input csvs were split into two separate csvs and the column structure was altered slightly.

[noracato]

This completely changes the design of the module. Let's indeed discuss the redesign, now it became a bit of a mashup of our two things.

[louispt1]

Can you review this now too @noracato? I can't add you as a reviewer

[louispt1]

Note: I updated the semaphore configuration to use ubuntu 2404 - can I remove the checks for Ruby 2.7.5 or is something still relying on that older version? Do you know this @aaccensi / @noracato ?

[aaccensi]

Note: I updated the semaphore configuration to use ubuntu 2404 - can I remove the checks for Ruby 2.7.5 or is something still relying on that older version? Do you know this @aaccensi / @noracato ?

Looks like the emissions branch was created long ago, this was done in master a couple months back, @louispt1 I suggest to use that file: https://github.com/quintel/atlas/blob/master/.semaphore/semaphore.yml

(Should we not merge/rebase or similar?)

[louispt1]

Note: I updated the semaphore configuration to use ubuntu 2404 - can I remove the checks for Ruby 2.7.5 or is something still relying on that older version? Do you know this @aaccensi / @noracato ?

Looks like the emissions branch was created long ago, this was done in master a couple months back, @louispt1 I suggest to use that file: https://github.com/quintel/atlas/blob/master/.semaphore/semaphore.yml

(Should we not merge/rebase or similar?)

Great point! Have rebased and all is well 👍

[noracato]

Suggested change

	def first_raw_value
	lines.first.to_s.match(LINE)&.captures&.last&.strip
	end

I think the idea of the module is good, but can use some refactoring. For instance taking first_raw_value out as a method that is being reused.

Also interesting is try if you could do something like this for the value method:

def value
  validate!

  lines = lines&.flatten if multi_line_array?
  parse_single_line_value
end

[louispt1]

I had a go at simplifying the approach, not in quite the way outlined here - let me know what you think!