TST: skip tests of in overlay symmetric_difference under pandas 1.3.3

[martinfleis]

overlay based on symmetric_difference started failing on pandas master because we used an outer join based on columns full of NaNs. Refactoring the implementation to avoid it.

[martinfleis]

New pandas is out breaking even more environments.

@jorisvandenbossche can we get this in to make CI useful again?

[jorisvandenbossche]

Hmm, that's a pity I didn't look into it earlier, as this is a regression in pandas as far as I can see, and unfortunately it ended up in the 1.3.x release (and not just in master). See pandas-dev/pandas#43550

Now, I think this patch is a nice clean-up anyway :) (not fully sure why we were using an outer merge on keys we knew had nothing in common, so it's basically a concat on the other axis)

[jorisvandenbossche]

Now, I think this patch is a nice clean-up anyway :) (not fully sure why we were using an outer merge on keys we knew had nothing in common, so it's basically a concat on the other axis)

Actually, that doesn't handle duplicate column names .. (column names present in both left and right dataframe: with concat they are combined in a single column, with merge they are kept as separate columns, so that's probably the reason we were using merge and not concat ..)

[martinfleis]

Actually, that doesn't handle duplicate column names

True. I didn't check that and our CI is not doing that either.

I guess I will scrap this PR and skip the test for pandas 1.3.3 assuming it will get fixed there?

[jorisvandenbossche]

Yes, skipping the test for pandas 1.3.3 might be best for now (it should already be fixed on master). Hopefully pandas can do a 1.3.4 relatively soon.

[jorisvandenbossche]

And we should also add a test with a case that would be different between merge vs concat

[martinfleis]

I've reverted the code change back, added xfail marks where needed and changed the existing test of duplicated columns to run with all how options. That would catch the bug I had here before.

[jorisvandenbossche]

I pushed two changes:

• xfailing the how fixture, which avoids having to add the if pandas_133 ... check to each test that uses it
• I pinned to pandas 1.3.2 in one of the "latest" envs, so we have at least one build with recent pandas where we still test overlay (and that also avoids having to temporarily add the doctest skip)

[jorisvandenbossche]

OK, all passing so merging we can get our CI back green :) Thanks!