Fuel_Costs_TP Expression speedup

[bmaluenda]

The Fuel_Costs_TP Expression in the fuel_cost module is taking significant time to be constructed. By building and using a new Set, this process can be sped up.

For the same input set used in my last commit (23 lz, 720 tp, 190 proj, 32 tx lines) building the Fuel_Costs_TP Expression was taking 7,76 s and the whole load_inputs function was at 71,04 s.
With these changes, the Expression construction takes 1,64 s and the load_inputs function takes 68,63 s.
Model instantiation is sped up only slightly (because the Expression akes less time to build, but a new Set has to be constructed), but I believe it may become more significant in larger models, specially with more timepoints.

[josiahjohnston]

A 5% speedup isn't bad :)
You can commit as-is if you like.

It might be slightly faster/cleaner to avoid defining DISPATCHED_PROJECTS_FUEL_IN_TP in dispatch.py, because it is only used in one operation in fuel_cost.py. You could instead define a custom set in fuel_cost that isn't a model component. I posted one way of doing that in the fuel_cost_speedup_discussion branch. I don't have profiling scripts set up to easily compare performance or try alternative ways of tuning that code, but in theory, having a local set could have less overhead than adding a new Set model component in a core module.

[mfripp]

I agree with Josiah that it's nice to keep down the number of sets if possible. DISPATCHED_PROJECTS_FUEL_IN_TP will probably only be used within the fuel_costs module, so it may not need to be added as a formal set (if it is added, maybe it should be renamed as TP_ACTIVE_PROJECTS_FUELS). On the other hand, the code in https://github.com/switch-model/switch/tree/fuel_cost_speedup_discussion is not as efficient as it could be, because it filters through all the projects active in each timepoint, which will include a lot of non-fuel projects.

Another option would be to use code like this to define Fuel_Costs_TP (in fuel_cost.py):

def Fuel_Costs_TP_rule(m, t):
    if not hasattr(m, 'Fuel_Costs_TP_dict'):
        # cache all Fuel_Cost_TP values in a dictionary (created in one pass)
        m.Fuel_Costs_TP_dict = {t2: 0.0 for t2 in m.TIMEPOINTS}
        for (proj, t2, f) in m.PROJ_FUEL_DISPATCH_POINTS:
            if (m.proj_load_zone[proj], f, m.tp_period[t2]) in m.FUEL_AVAILABILITY:
                m.Fuel_Costs_TP_dict[t2] += (
                    m.ProjFuelUseRate[proj, t2, f] 
                    * m.fuel_cost[m.proj_load_zone[proj], f, m.tp_period[t2]]
                )
    # return a result from the dictionary;
    # this uses pop() so all memory used by the dictionary will eventually be released
    return m.Fuel_Costs_TP_dict.pop(t)
mod.Fuel_Costs_TP = Expression(mod.TIMEPOINTS, rule=Fuel_Costs_TP_rule)

I think this code will be about as fast as you can get. It seems to work fine with run_tests.py and some other spot checks, but I don't have a slow model that uses fuel_cost.py, so I can't say how it affects performance.

[josiahjohnston]

That looks great!
Benjamin, would you mind checking the speed of Matthias's code snippet?

[bmaluenda]

I ran a medium example a couple of times and whilst the original code takes between 4.3 and 4.8 s to construct the Fuel_Costs_TP expression, this new snippet reduces that time to between 0.85 and 0.95 s. Outputs were identical, so everything is working as expected.

It is true that defining new sets is something that should be avoided. I will commit this new version and if everything looks ok, I'll merge it in.

[josiahjohnston]

The code you posted has an indent error. Other than that, it looks good to me. Please squash it to a single commit with a clean message when you push the changes.

Thanks! :)

[josiahjohnston]

Looks good to me 👍