implement `future.apply` in `forecast` as it is in `model`

[davidtedfordholt]

Using fable at any kind of scale gets painful if computation of forecasts isn't also parallelized. I've got a possible implementation I'll PR tomorrow.

[mitchelloharawild]

A PR for this would be great, thanks!

[mitchelloharawild]

My suggestion for this would be to drop the use of mutate_at() and replace it with a internal util function which iterates over the mable with parallel support. There is a common pattern throughout the package where functions applied to a mable need to operate over each model, and a more general (and parallelisable) helper function for this would be great (for example, generate() should also be possible in parallel).

[davidtedfordholt]

I've got to pull a couple of errant commits out, but I'll send it along momentarily. I reversed the order of the pivot_longer and changed the mutate_at to mapply to mirror the future.apply implementation.

[davidtedfordholt]

I'm happy to make a util function that does the iteration piece. It would essentially start with something that essentially at the point of having been pivot_longer'ed and iterates across with either mapply or future_mapply, given that they take the same arguments (other than the future-specific added ones). Take everything that's passed in additionally and place it in MoreArgs, then select the method based on the availability of future.

[davidtedfordholt]

This is an 80-core machine trying to churn through forecasts for the models it fit between 5:45pm and 6pm. It's still at it, 4 hours later. This is a pretty strong motivator, as I'll be explaining the bill later. :)

[mitchelloharawild]

I think it could be safer / more efficient if the internal util would also handle the pivot_longer bit to avoid possible variable name conflicts that exist from the pivoting process.

[davidtedfordholt]

Makes sense, and shouldn't be any tougher. It won't pivot_wider to return to the previous structure, because forecast() specifically needs for it not to do so. I can't think of an instance where it would need to, but I can add in a param to return it to the original structure, if you'd like.

[mitchelloharawild]

An example of returning to the previous structure is when the operation returns a <mable>.

For example, refit(<mable>, <tsibble>) returns a <mable> of the same structure. The same applies for stream(<mable>, <tsibble>)

Adding/extending the apply-util for this can be done later.

[davidtedfordholt]

Makes sense. Should be straightforward enough.

[davidtedfordholt]

To make modeling and forecasting more memory efficient for tsibbles with many keys, and this should probably go for any fabletools process, it might make sense to shrink the environment of each worker in the future_mapply calls. This means we'll need to figure out what packages like fable.prophet might be attached that will need to be available. Obviously, if we can generally rely on the idea that the namespace will begin with fable. then it's pretty easy. I don't know if that's something you've thought about and feel strongly about, but it seems reasonable enough.

Otherwise, we might need to crawl across the functions within the call to find all of their namespaces and add them that way, which seems a bit more fraught with peril.

[snowman-87]

Just following along as I use fable for millions of time series. If this is in the newest version of the Github release I'd be happy to help test, otherwise just saying I appreciate the work here!

[mitchelloharawild]

On further thought and a bug report, converting to longer before parallelising isn't appropriate because the column structure defines the way in which the forecasts are handled (such as reconciliation). I'll have a go at adding the conditionally parallel mapply function.