BUG: Require sample weights to sum to less than 1 when replace = True

[microslaw]

• closes BUG: DataFrame.sample weights not required to sum to less than 1 #61516
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[rhshadrach]

Looking good, but the condition needs adjusted. Will also need a note in the whatsnew for 3.0 under the Other section.

[rhshadrach]

This is not the correct condition. See the formula in the OP of the linked issue.

[yuanx749]

Suggested change

	When replace = True will not allow weights that add up to less
	than 1, to avoid biased results.
	When replace = False will not allow ``(n * max(weights) / sum(weights)) > 1``,
	to avoid biased results.

[microslaw]

1. I've added this as an exception, but I've also considered doing this as a warning, since the issue is

   • easy to encounter - one case occured in user_guide
   • hard to explain within a short error message - requires specific equation and some level of understanding of probability
   • has minimal effect on majority of use cases - as long as the variance isn't huge, the bias this error introduces is minimal, and for very specific cases e.g. the village sampling from the op warning should suffice

2. I also deviated a bit from original issue in that i'm allowing max(w) * n / sum (w) == 1. As far as I understand, it doesn't introduce any bias, and it allows some specific uses, e.g. df.sample(n=1,weights=[1,0])

3. If this is implemented as an exception, I believe that there is no way for the numpy exception "Fewer non-zero entries in p than size" to occur, as:

   • w - list of weights
   • n - number of non-zero weights in w
   • size - size of the sample to be drawn
   • len - length of w

   I'm assuming that

   • the weights are normalized, so sum(w) = 1, as it doesn't change anything and makes the math easier
   • the weights are equal, since if any of them is larger than the others, max(w) and max(w) * size will increase
   • therefore w = [1/n, 1/n, ..., 1/n] or any combination like w = [1/n, 0, 1/n, 0, ..., 1/n] where n is the number of non-zero weights
   • for the "Fewer non-zero entries in p than size" to occur, size > n
   • for the "Invalid weights: If replace=False, " to occur, size * max(w) / sum(w) >= 1
   • size * max(w) / sum(w) >= 1
   • size * max(w) >= 1, as sum(w) = 1
   • size * 1/n >= 1, as max(w) = 1/n
   • size >= n

[rhshadrach]

Looking good!

[rhshadrach]

• Avoid O(n) computation if it is not needed.
• Combine with the if block above; no need to check if weights is not None twice.
• Suggestion on a more clear error message.

Suggested change

	if weights is not None:
	is_max_weight_dominating = size * weights.max() > 1
	if is_max_weight_dominating and not replace:
	raise ValueError(
	"Invalid weights: If `replace`=False, "
	"total unit probabilities have to be less than 1"
	)
	if not replace and size * weights.max() > 1:
	raise ValueError(
	"Weighted sampling cannot be achieved with replace=False. Either "
	"set replace=True or use smaller weights. See the docstring of sample "
	"for details."
	)

[rhshadrach]

Can you add "See the Notes below for more details." and then add a bit more detail in the notes section. I can give it a shot if you'd prefer.

[rhshadrach]

Can you make these two separate tests.