Setting sample_weight in Poisson regression example

[thomasjpfan]

Describe the issue linked to the documentation

In the Poisson regression and non-normal loss example, we set the sample weight to the exposure, when we divided the count data by the exposure. We had this discussion regarding this here: https://github.com/scikit-learn/scikit-learn/pull/14300/files#r386066958

When looking at the reference paper (page 16) the example was based on, it handles this by using an offset:

glm(formula = ClaimNb ~ VehPowerGLM + VehAgeGLM + DrivAgeGLM +
	BonusMalusGLM + VehBrand + VehGas + DensityGLM + Region +
 	AreaGLM, family = poisson(), data = learn, offset = log(Exposure))

Which I think is the same as:

where l is the exposure. In our example, the target has been already divided by the exposure. If we want to match the narrative by the paper, is the sample_weight required?

Edit: I guess we are treating 4 event in 8 years to have a higher weight than 1 event in 2 years.

CC @lorentzenchr @rth

[lorentzenchr]

If we want to match the narrative by the paper, is the sample_weight required?

Yes, it is required because modelling counts with exposure as offset is equivalent to modelling frequency = counts/exposure with exposure as sample weight and scikit-learn does not (yet?) provide the possibility to include offsets.

I guess we are treating 4 event in 8 years to have a higher weight than 1 event in 2 years.

Exactly 4 times more weight:smirk:

More background

Hard to argue without some notation, so...

Assumptions

A GLM assumes an expectation E[Y_i] = mu_i = h(x_i beta) and a variance Var[Y_i] = phi/w_i * v(mu_i). The second assumption about the variance stems from assuming that Y follows an EDM, but the variance assumption is enough for estimation of the expectation.

Notation:

• i indicates observation i
• x_i is the feature vector (row of feature matrix) for observation i
• beta are the coefficients (aka weights, but misleading term here)
• h is the inverse link function
• w_i is a sample weight
• v is the variance function
• phi is a dispersion parameter.

Estimation

It is important to note that the derivative of the inverse link h'(x beta) and the variance both enter the estimation (score) equation while phi drops out:
sum_i x_i h'(x_i beta) / Var[Y_i] * (y_i - E[Y_i]) = 0.

Log-Poisson

For the Poisson model with log-link we have Var[Y] = E[Y] and h(x)=exp(x). It just happens that modelling Y=counts with offset=log(w) (without sample weight) is equivalent to modelling Z=counts/w with sample weight w (plug into estimation equation to see this). This is usually not the case for other combinations of mean-variance relation (EDM) and link function.

Difference between sample weight and offset

What is then the difference between sample weight and offset?

• A sample weight (or exposure) is a variable that is additive and can be used to build meaningful weighted averages of the target Y, e.g. z_2 = (w_1 * y_1 + w_2 * y_2)/(w_1 + w_2). More precisely, it should fulfill the reproductive property of an EDM. This means that the expectation E[Y] should be proportional to and the variance Var[Z] = Var[Y/w] should be inverse to the weight, compare to GLM assumptions above. Good candidates are time, volume and exposure measures.
• Offsets are a way to add a priori known effects of features to the linear prediction x beta. Examples are percentage of loss covered in insurance (anybody with a good example?).

Personal note

I much prefer the use of sample weights as it also works for other EDMs than log-Poisson, e.g. consider a logit model with grouped/aggregated data=(observed success rate, number of trails). This fits perfectly with sample weights, but good luck to formulate this with offsets:smirk:
Though, the use of time as offset in a Poisson model can be motivated by considering a Poisson process.

[lorentzenchr]

@thomasjpfan Can we close this issue?

[thomasjpfan]

Yup lets close