Use own trend model

[pjebs]

In the paper Forecasting at Scale by Sean J Taylor and Benjamin Letham, it says that prophet decomposes the timeseries using equation:

y(t) = g(t) + s(t) + h(t), where g is the trend model.

Is it possible to plug in our own trend model and use prophet only for s and h?

[pjebs]

If not possible, consider it a feature request.

[seanjtaylor]

It's not possible at present. Do you have a specific trend model in mind? In general the trend model would have to be added to Stan and so we'd need to know what kinds of models the user would require in advance.

[bletham]

#340 is the open issue for supporting custom Stan models, which is what would make this possible. Both of these block on #501, which hopefully will be possible with the latest Stan release.

[pjebs]

I have trend data (series of (t, y_hat)) that is pumped out by a black-boxed machine learning algorithm.

[bletham]

In that case you could try adding the output of the machine learning algorithm as an extra regressor. I think that's about the best that could be done in this situation currently.

[pjebs]

How would I then "disable" prophet from adding it's g(t)?

[bletham]

You can set n_changepoints to 0, which will make the trend a straight line and would likely be sufficient - if there isn't a trend in the residuals between y and the ML trend, then I'd expect it to fit a slope of 0.

You can't currently entirely turn off the built-in trend (#614), so if you did end up needing to do that you'd have to modify the stan code to fix k=0.

[pjebs]

Will it be a realistic potential feature to supply a series of data for the trend model?

[bletham]

Potentially, but I think my first priority would be #340, at which point it would be a minor modification of the Stan model to use an externally estimated trend. We could then at that point see if allowing this by modifying the Stan model is sufficient or if it is worth creating a built-in model for handling that.

[pjebs]

If this feature request was accepted, what kind of timeframe will it be implemented?

[bletham]

I'd be thinking ~6 months given that there is a substantial refactor involved, and that it will first require some testing of the new Stan feature that should allow moving the model predictions to Stan.

[pjebs]

Since the issue is closed, does it mean this is no longer on the wishlist?

[bletham]

I had closed it because I think #340 is how this will actually be accomplished (make it easy to customize the Stan model, including making changes to the trend), but we can leave it open to be sure that it is actually made possible that way.

[pjebs]

I was wondering what is the new estimate to make this feature possible?

[bletham]

The Stan feature for doing predictions in Stan does not yet have an interface in rstan and pystan, so we haven't been able to use it. We've been trying to work around it and get predictions happening in Stan another way in #865, but that's also getting blocked by some perf issues in rstan. Our plan now is to wait for the next rstan release and see after that what the best path forward will be.

[pjebs]

It's there any new news on this issue?

[bletham]

Making this a built-in feature has been deprioritized since we've decided not to pursue #501 (which was expected to make this clean and easy).

Instead of having this built-in, I favor just having some documentation that lays out the changes you'd need to make in order to add a custom trend to your local checkout of the package. It's actually not a terrible amount of work - #1466 adds a new trend, and I think provides a great roadmap for adding a trend. If you're interested to give it a try, look that PR over and let me know if you have any questions.

I'd also be interested in getting a better sense of what type of trend you're looking to implement / what is blocked by this. In particular if you were able to post a plot of the time series that needs a different trend model that'd be helpful!

[rquintino]

Hi everyone, not sure how related this is, but would there be a quick way of switching to prev timeseries value/persistent instead of default trend? Wonder if for short term horizons this could be competitive .

Or at least what should the inverse calc to nullify trend component replacing by last value (removed of other remaining effects is possible). Something like this :) (not an expert!)
thx!

[bletham]

Interesting... I actually just did something really similar. Basically what you can do is override the trend calculation to add another changepoint at the end of the timeseries that sets the slope to 0 and intercept to the last value:

import pandas as pd
import numpy as np
from fbprophet import Prophet

class ProphetLastValue(Prophet):
        
    @staticmethod
    def piecewise_linear(t, deltas, k, m, changepoint_ts):
        """
        This is the method Prophet uses for computing the piecewise linear trend.
        Here it is changed to keep the trend flat at the last value of the time series.
        
        Parameters
        ----------
        t: np.array of times on which the function is evaluated.
        deltas: np.array of rate changes at each changepoint.
        k: Float initial rate.
        m: Float initial offset.
        changepoint_ts: np.array of changepoint times.
        
        Returns
        -------
        Vector trend(t).
        """
        # Intercept changes
        gammas = -changepoint_ts * deltas
        # Get cumulative slope and intercept at each t
        k_t = k * np.ones_like(t)
        m_t = m * np.ones_like(t)
        for s, t_s in enumerate(changepoint_ts):
            indx = t >= t_s
            k_t[indx] += deltas[s]
            m_t[indx] += gammas[s]
        
        # Future dates have t > 1, due to how t is scaled.
        # Add an additional delta to have a flat trend at t==1.
        if max(t) > 1:
            indx_future = np.argmax(t >= 1)
            end_trend = float(k_t[indx_future] + m_t[indx_future])
            k_t[indx_future:] -= k_t[indx_future]  # Set slope to 0
            m_t[indx_future:] -= (m_t[indx_future] - end_trend)
        return k_t * t + m_t


df = pd.read_csv('../examples/example_retail_sales.csv')
m = ProphetLastValue(seasonality_mode='multiplicative').fit(df)
future = m.make_future_dataframe(periods=120, freq='MS')
forecast = m.predict(future)
m.plot(forecast)

The result of running the above code is this, which I believe is what you were interested in:

Doing this was a lot easier than a fully custom trend, because it affected only the trend computation in the future, and thus didn't affect model fitting. That allowed me to make the small change above to how the trend is computed in the Py, and I didn't have to make the changes to the Stan code that would have been required for anything that also affects model fitting.

[pjebs]

Instead of having this built-in, I favor just having some documentation that lays out the changes you'd need to make in order to add a custom trend to your local checkout of the package. It's actually not a terrible amount of work - #1466 adds a new trend, and I think provides a great roadmap for adding a trend. If you're interested to give it a try, look that PR over and let me know if you have any questions.

When you say "changes you'd have to make", do you mean we have to maintain our own prophet library and make adjustments in the fork?

I'd also be interested in getting a better sense of what type of trend you're looking to implement / what is blocked by this. In particular if you were able to post a plot of the time series that needs a different trend model that'd be helpful!

The trends were coming out of a machine-learning black-box algorithm that just produces the trend data. I was hoping to use prophet to add on the seasonal affects etc. I don't have any data because it was a project that was non-urgent at the time (but getting closer to being urgent), and I was hoping to implement prophet.

[bletham]

@pjebs I think there are two different types of custom trends that are getting a bit mixed up in the discussion here. One is to have a trend model that replaces the built-in piecewise-linear/piecewise-logistic trend, and is jointly fit along with the seasonalities in the same way the built-in trends are. That is, the trend model has parameters that will be fit by Prophet. In order to do this, things have to be changed inside the Stan model. #1466 provides a good example of how this can be done. This is sort of adjustment I think for the time being would be made in a fork.

What you've described is a little different; there is a trend provided by an external model that we wish to use in the place of the built-in trend. The difference is that the trend parameters do not need to be fit by Prophet. This is quite a bit simpler. With additive seasonality, this can actually be done now. Above I had proposed doing this by including the external trend as an external regressor. But, as noted above, this wasn't entirely satisfactory because there was no way to prevent Prophet from adding extra trend on top of it until #614 was done. Well, as of the last version push (0.7.1), #614 is now done in Python! So what you would want to do is:

• Use the built-in flat trend (https://facebook.github.io/prophet/docs/additional_topics.html#flat-trend-and-custom-trends)
• Add the external trend as a regressor (https://facebook.github.io/prophet/docs/seasonality,_holiday_effects,_and_regressors.html#additional-regressors) . Include future trend values when making the prediction. You might want to turn off standardization in the regressor - I'm not totally sure.

The reason this would work is because with additive seasonality, the Prophet model is

y(t) = trend(t) + seasonality(t) + beta * regressor(t) + noise

so here we would be setting trend(t) to be flat, and then beta * regressor(t) would be taken the place of the trend.

With multiplicative seasonality, the external trend would have to be used as the trend in the Stan model (and not a regressor), so it would require modification there.

[rquintino]

@bletham huge thx for the ProphetLastValue snippet, seems really what we were looking for, let us test asap and share feedback. We have been struggling with trend issues for short term forecasts (mostly kind of "lagging" behind), so will be really interesting to give this a try.
again, thank you and all the team!

[bletham]

@rquintino Cool, hope its useful. I did just notice that I had left a bug in the handling of the max(t)<=1 case, so I just updated the code snippet above to fix it.