Documentation for s in UnivariateSpline is confusing

[cancan101]

From the docs:

w : (N,) array_like, optional
Weights for spline fitting. Must be positive. If None (default), weights are all equal.

s : float or None, optional
Positive smoothing factor used to choose the number of knots. Number of knots will be increased until the smoothing condition is satisfied:
If None (default), s = len(w) which should be a good value if 1/w[i] is an estimate of the standard deviation of y[i]. If 0, spline will interpolate through all data points.

I interpret that to mean that if w is not specified, then s=0 should be the same as s=None but that does not seem to be the case.

[josef-pkt]

I read this as s = nobs = len(data) in this case
implied weights if w is not specified are w = np.ones(len(data))

guessing based on part of docs in description

[cancan101]

@josef-pkt how do you think s will differ when you set it to None vs 0 when w=None?

[ev-br]

TBH I fail to read it as s=0 is equivalent to s=None. It is definitely not the same. Compare the plot at
http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.interpolate.UnivariateSpline.html
(s=None, default) to http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.interpolate.InterpolatedUnivariateSpline.html#scipy.interpolate.InterpolatedUnivariateSpline --- here s=0.

[cancan101]

I know they are not the same. My question is what is actually happening.

[ev-br]

The missing piece of documentation is maybe that UnivariateSpline does not do interpolation, it does fitting unless you force interpolation by using s=0.

Regarding what is actually happening, uhm well, here are Fortan level docs: https://github.com/scipy/scipy/blob/master/scipy/interpolate/fitpack/curfit.f#L141

Maybe some of this information could/should be added to python level docs.

EDIT: https://github.com/scipy/scipy/blob/master/scipy/interpolate/src/fitpack.pyf#L263
It seems that the default is w = np.ones_like(x) and s=len(x).

[josef-pkt]

if w is None and s is None: s = number_of_observations is my interpretation of the docstring (which sounds too large to me)

[argriffing]

My question is what is actually happening.

This has already been answered in this thread, but I agree that the interface is confusing. You might think if you don't want weights or smoothing then you just don't provide w or s, but in fact this will use default (equal) weights and default (nonzero) smoothing. So if you want to disable smoothing then you have to set it to zero explicitly.

[josef-pkt]

correction: s=len(x) is not too large because s is sum of squares and not mean.
From the docstring this would mean it's designed for standardized variables where variance (ss / nobs) is one

[josef-pkt]

but I thought/think s is residual sum of squares not sum of squares of y.

[ev-br]

@cancan101 viewing this as a documentation issue, what would you suggest to add/remove/improve?
I don't think we should change the behavior of the UnivariateSpline (as, erm, quirky it is)

[cancan101]

Perhaps mention that s=0 results in an interpolating spline whereas the default of s=None yields a smoothing spline that uses all of the points?