not_empty_in makes unchecked assumptions about its input

[asmeurer]

I noticed this when doing #16344.

In not_empty_in, there is this code:

sympy/sympy/calculus/util.py

Lines 294 to 303 in f06edf4

	if isinstance(finset_intersection, FiniteSet):
	finite_set = finset_intersection
	_sets = S.Reals
	else:
	finite_set = finset_intersection.args[1]
	_sets = finset_intersection.args[0]
	if not isinstance(finite_set, FiniteSet):
	raise ValueError('A FiniteSet must be given, not %s: %s' %
	(type(finite_set), finite_set))

The code assumes that any set that isn't a finite set is an intersection with two arguments, a finite set and another set. It also assumes that the finite set will be the second argument of the intersection.

If the algorithm can only work in this case, that is fine, but it should check the input to make sure that the object actually is what it thinks it is. Otherwise, it will produce wrong results.

[kangzhiq]

Hello,

I would like to work on this issue.

but it should check the input to make sure that the object actually is what it thinks it is

By saying this, do you mean that we should check if the input finset_intersection has args[1] and arg[0] as finite set and other set?

if not isinstance(finite_set, FiniteSet): 
     raise ValueError('A FiniteSet must be given, not %s: %s' % 
                      (type(finite_set), finite_set))

This if clause is one the checks, isn't-it? Could you please guide me in this?
Thank you in advance!

[asmeurer]

That is one type check, but there are other assumptions, like assuming that finset_intersection is an intersection and assuming it has exactly 2 args. Also, it should be able to handle the case where the finite set is first as well. Intersection sorts the sets by the infimum, so they can be in either order.

[kangzhiq]

@asmeurer OK! I will come back to this issue later, I am preparing the proposal for GSoC 2019.