Warnings in recursive-aggregation

[pjuergens]

When using the recursive aggregation with one of the two lines

data_IIASA.aggregate('Capacity', recursive='skip-validate', append=True)
data_IIASA.aggregate('Capacity', recursive=True, append=True)

I most of the time get a (or lots of) warning(s)

WARNING: Filtered IamDataFrame is empty!

either from pyam.core or from pyam.utils.

Functionally it's not a problem, but it's quite annoying as it fills up the output and makes it hard to see any other - potentially important - print statements. So it would be nice to either hide the warning completely when using recursive aggregation or to generally add a keyword in the aggregate-function to hide these warnings.

Or, one could look deeper into it and see if the recursive-aggregation is doing unnecessary aggregations and eliminate them.

[danielhuppmann]

Thanks for reporting this issue @pjuergens - my hunch is that the offending line is

pyam/pyam/_aggregate.py

Line 79 in 0989fdd

_data_self = _df.filter(variable=var_list)._data

which (correctly) will return this warning if none of var_list exists in _df.

Luckily, there is a simple decorator in pyam that allows you to change the log-level when executing some code - see this in use in the same file, in the _aggregate_region() function.

pyam/pyam/_aggregate.py

Line 148 in 0989fdd

with adjust_log_level(logger):

Adding that to the offending line in the recursive aggregation should clean up the log.

[pjuergens]

It seems to be indeed this line - however decorating the line with with adjust_log_level(logger) doesn't remove the warnings.

Do I need to specify the logger-instance? I'm a bit confused as the usage should be exactly the same like in Line 148

[danielhuppmann]

Now that I'm looking at it, I'm wondering if this even works as intended for the aggregate_region() function...

Indeed, the logger referenced here is the one for this file (see line 19 __name__). So you could try to get the logger for pyam.core and set that to WARNING.

with adjust_log_level(logging.getLogger("pyam.core")):

Or even nicer, update the function adjust_log_level() to take a logger or a string, and do the logging.getLogger() inside the pyam.logging utility. So that the function in the aggregation module could be called with

with adjust_log_level("pyam.core"):