PERF: allow to skip validation/sanitization in DataFrame._from_arrays

[jorisvandenbossche]

For cases where you know to have valid data (eg you just created them yourself, or they are already validated), it can be useful to skip the validation checks when creating a DataFrame from arrays.

Use case is for example #32825

From investigating #32196 (comment)

@rth this gives another 20% improvement on the dataframe creation part. Together with #32856, it gives a bit more than a 2x improvement on the dataframe creation part (once the sparse arrays are created)

[jorisvandenbossche]

In [1]: arrays = [pd.arrays.SparseArray(np.random.randint(0, 2, 1000), dtype="float64") for _ in range(10000)] 
   ...: index = pd.Index(range(len(arrays[0])))   
   ...: columns = pd.Index(range(len(arrays)))

In [2]: %timeit pd.DataFrame._from_arrays(arrays, index=index, columns=columns)   
119 ms ± 3.52 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [3]: %timeit pd.DataFrame._from_arrays(arrays, index=index, columns=columns, verify_integrity=False)    
98.1 ms ± 713 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

[rth]

Very nice, thanks @jorisvandenbossche ! It's good that is applies to extension arrays in general not just sparse frames. Though I guess the use case of a very large number of columns (>10k) is less common outside of sparse.

[rth]

Actually my comment was more about #32856, got confused in your multiple performance improvement PRs :) This is nice too!

[rth]

By "actual arrays" do you mean numpy ndarray or pandas arrays? Might be worth specifying.

[jorisvandenbossche]

One of both. Basically the array as how it is stored in a block. Will mention it needs to be one of both.

[jorisvandenbossche]

I think your comment applies to both ;)

[simonjayhawkins]

Nice docstring. 😄

[TomAugspurger]

What order do we want to do this and #32825 in?

Seems like we merge this first and then update #32825 to pass verify_integrity=False?

[jorisvandenbossche]

Doesn't matter too much, can be added either here or there.
And I would also like to add verify_integrity to the benchmark that I am adding in #32856

[TomAugspurger]

SGTM. I think this can go in since CI is passing :)