[ML] SIGFPE in CSeasonal::propagateForwards() in 6.8.1

[droberts195]

A SIGFPE crashed an ML job in 6.8.1. This is the failure message:

Fatal error: 'si_signo 8, si_code: 1, si_errno: 0, address: 0x7f64eccb39b1, library: /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/../lib/libMlMaths.so, base: 0x7f64ec95c000, normalized address: 0x3579b1', version: 6.8.1 (build 6e3432237cefa4)

It looks like the function where the SIGFPE occurred was ml::maths::CTimeSeriesDecompositionDetail::CComponents::CSeasonal::propagateForwards(long, long):

$ objdump -T libMlMaths.so | grep '^00000000003579' | sort
0000000000357930 g    DF .text  000000000000016e  Base        _ZN2ml5maths30CTimeSeriesDecompositionDetail11CComponents9CSeasonal17propagateForwardsEll
$ c++filt _ZN2ml5maths30CTimeSeriesDecompositionDetail11CComponents9CSeasonal17propagateForwardsEll
ml::maths::CTimeSeriesDecompositionDetail::CComponents::CSeasonal::propagateForwards(long, long)

The relevant code looks like this in 6.8.1:

void CTimeSeriesDecompositionDetail::CComponents::CSeasonal::propagateForwards(core_t::TTime start,
                                                                               core_t::TTime end) {
    for (std::size_t i = 0u; i < m_Components.size(); ++i) {
        core_t::TTime period{m_Components[i].time().period()};
        core_t::TTime a{CIntegerTools::floor(start, period)};
        core_t::TTime b{CIntegerTools::floor(end, period)};
        if (b > a) {
            double time{static_cast<double>(b - a) /
                        static_cast<double>(CTools::truncate(period, DAY, WEEK))};
            m_Components[i].propagateForwardsByTime(time);
            m_PredictionErrors[i].age(std::exp(-m_Components[i].decayRate() * time));
        }
    }
}

It's still the same in latest 6.8. It was refactored in version 7.3.0 in #496. We should probably backport a bare minimum change to 6.8 that at least logs an error instead of crashing with a SIGFPE.

[droberts195]

The SIGFPE could have been in one of the functions called from CSeasonal::propagateForwards() if it was inlined.

"Floating point error" is a bit confusing here, as the si_code: 1 is FPE_INTDIV: integer division by zero.

So presumably this did occur in one of the called functions, which was inlined.

[droberts195]

Most likely cause is CIntegerTools::floor being called with alignment 0. That indicates another problem earlier on, namely that period is 0. This wouldn't happen in the latest code as period is clamped to be between a day and a week before CIntegerTools::floor is called, but it is still a risk in 6.8. Also, it's still possible that whatever bug caused period to be 0 still exists in the latest code, and that could cause problems elsewhere in the analysis.

[tveasey]

Having period = 0 shouldn't be possible and is extremely odd. The only possibility I can think of is a restore issue. However, it is possible to make CIntegerTools::floor defensive and better than having a SIGFPE. I would like to continue to investigate this issue, so won't close this issue at present.

[droberts195]

Having period = 0 shouldn't be possible and is extremely odd

We got to the bottom of this. The cluster where this happened had a mix of 6.8.1 and 6.8.10 nodes. Due to #821 6.8.1 cannot load state from 6.8.10. In 7.0 and above an older node would never be asked to load state produced by a newer node (due to elastic/elasticsearch@c10b2ea), but in 6.x there is no protection against an older node being asked to run a job whose model state was produced on a newer node. In 6.x it's necessary to upgrade all ML nodes to the newer version to recover if this problem occurs.