Allow mapping distinct names for MESSAGE commodity → MACRO sector

[khaeru]

This functionality is implied to work but, per the added test on the initial commits on this branch, does not. See e.g. here:

RuntimeError: 0-price found in MESSAGE variable PRICE_COMMODITY for commodity "FOO" in node "Westeros".

This occurs because of code like:

message_ix/message_ix/macro.py

Lines 473 to 475 in 6dee23a

	model_price.rename(
	columns={"lvl": "value", "commodity": "sector"}, inplace=True
	)

Data originating from MESSAGE's PRICE_COMMODITY is sent to MACRO's price_MESSAGE, with the commodity dimension renamed to sector, but the mapping provided by the user in the config data is never applied.

This code ultimately dates back to #223, and was not altered in #327. Tests were never added for this functionality.

The PR fixes the bug and makes other improvements:

• Document each atomic function in the calculations; closes message_ix.macro lacks minimal documentation #315.
• Use genno to structure the functions, as suggested at

  message_ix/message_ix/macro.py

  Lines 323 to 324 in cec4092

  	(This is done through method chaining, the bottom of which is aconst()
  	# NB this means it could be rewritten using reporting)

• Allow the user to omit e.g. price_ref. In this case, as is common in actual use, they are back-extrapolated from data (e.g. PRICE_COMMODITY) in the scenario which is being calibrated.
• Accept either path to a file with calibration data, or a dictionary, allowing some of the data to be constructed as needed without writing to file; see e.g. Miscellaneous enhancements for 2023-W24 message-ix-models#104.
• Correct typos in the MACRO GAMS code.

Housekeeping:

• Activate ixmp.utils.sphinx_linkcode_github; items in the documentation are now linked directly to their source on GitHub.

How to review

• Build and view the updated docs.
• Skim the updated test code.
• (Optional) Run existing downstream/user code which uses Scenario.add_macro; confirm that it continues to work as expected.

PR checklist

• Continuous integration checks all ✅
• Add or expand tests; coverage checks both ✅
  • Update and re-enable the multiregion tests.
• Add, expand, or update documentation.
• Update release notes.

[codecov]

Codecov Report

Merging #719 (de613c8) into main (cec4092) will increase coverage by 0.0%.
The diff coverage is 97.6%.

@@          Coverage Diff          @@
##            main    #719   +/-   ##
=====================================
  Coverage   94.4%   94.4%           
=====================================
  Files         43      43           
  Lines       3448    3434   -14     
=====================================
- Hits        3257    3245   -12     
+ Misses       191     189    -2     

Impacted Files	Coverage Δ
message_ix/reporting/computations.py	100.0% <ø> (ø)
message_ix/macro.py	96.6% <96.3%> (+0.1%)	⬆️
message_ix/core.py	95.1% <100.0%> (+<0.1%)	⬆️
message_ix/tests/test_macro.py	100.0% <100.0%> (+0.5%)	⬆️

[glatterf42]

The formulation of match seems odd to me here, so just to make sure: is the raised error message really referencing something like columns.gdp_calibrate.year? The MissingKeyError in genno seems to be phrased differently, at least.

[khaeru]

Hi, yes. This exception is raised by the function .macro._validate_data()—in full it is something like "Missing expected columns for gdp_calibration: {'year'}". pytest.raises(…, match=…) is treated as a regular expression, so the .* matches a bunch of characters in the middle.

[glatterf42]

Apart from that one clarifying question, the new tests look good to me, very streamlined.

[glatterf42]

To build the docs locally, I ran

sphinx-build -T -E -d _build/doctrees-readthedocs -D language=en . _build/html

as suggested in our own docs. This command produces a few warnings, which might be related to your changes:

Cannot locate code for 'message_ix.models.DEFAULT_CPLEX_OPTIONS' or parent class/module             
Cannot locate code for 'message_ix.models.MESSAGE_ITEMS' or parent class/module
Cannot locate code for 'message_ix.macro.MACRO_ITEMS' or parent class/module

The local docs render slightly differently on e.g. https://docs.messageix.org/en/latest/api.html, where the initial list does not have any bullet points in my local version. The same is true for https://docs.messageix.org/en/latest/macro.html, where the nested structure is also lost for the table-of-content-list. But this does not seem to be specific to this PR because it happens on all other pages as well. Apart from that, the docs-part is looking good to me.

[khaeru]

This command produces a few warnings, which might be related to your changes

Yes, that's related to the Sphinx extension ixmp.utils.sphinx_linkcode_github. In general that extension tries to identify the specific lines of a file on which an object (class, function, etc.) is defined, and link directly to them. But it fails for module-level constants, because Python's internals do not provide that information or it's harder to get at. See here.