BUG: dataframe setitem error on size incompatible assignment

[Mxchaeltrxn]

• closes BUG: Unexpected behavior when assigning multi-dimensional array to DataFrame column #40827
• tests added / passed
• Ensure all linting tests pass, see here for how to run them

I've labeled the error something silly but that can be fixed later.

This pull request replaces this older pr #41008

I've put all my tests inside one file for easier review.

Issues

I'm unsure how to stop this test from failing test_pivot_table_doctest_case Could you help with this?

This test (test_invalid_colormap) also fails but when I rerun it individually, it passes ...

I also had an issue with flake8 and black not working together ... in my most recent commit so I just committed it with --no-verify.

[Mxchaeltrxn]

I may have worked something out so don't review this yet.

[Mxchaeltrxn]

I've gotten test_pivot_table_doctest_case to pass.

[jreback]

don't add a new test file. rather somewhere in tests/frame/indexing

[jreback]

probably better to have .should_store return False in this case and have it hit the other path? or is that incorreect

[Mxchaeltrxn]

If I understand you correctly, I think it should stay as it is because the below test case fails:

def test_setitem_size_incompatible_ndarray2(arr):
    data = DataFrame(
        [[1, "A", 1.0], [2, "B", 2.0], [3, "C", 3.0], [4, "D", 4.0]],
        columns=["A", "A", "B"],
    )
    msg = "Errored123"
    with pytest.raises(ValueError, match=msg):
        # data is of shape (4,2) but we are assigning it to shape (4,4)
        data["A"] = np.random.randn(4, 4)

It fails because it goes down the other path (where blk.should_store is false) and no error is thrown despite assigning the column(s) to data of a different shape. I haven't handled this yet because I wanted to make sure the logic was ok so far (sorry I should have mentioned this earlier).

[Mxchaeltrxn]

I think that .should_store is probably not the best description of what it does, (based on my understanding of the method).

[Mxchaeltrxn]

@jreback Thanks for the review. I will move the tests and rename them later. How do I make sure that I'm not duplicating tests ...?

[jreback]

@jreback Thanks for the review. I will move the tests and rename them later. How do I make sure that I'm not duplicating tests ...?

IIUC, you are essentially narrowing a case, so either try to find the tests that exercise that part of the code (might) be hard, or simply add in the logical place; by-definition you won't be duplicating.

[Mxchaeltrxn]

@jreback Sorry I've been a bit busy. I'll return to this on the weekend when I have some time.

[Mxchaeltrxn]

@jreback Hey so I've been trying to work on this but I've been getting an error. Do you have any ideas?
Steps to reproduce:

1. git fetch upstream
2. git merge upstream/master
3. conda env update -n pandas-dev --file environment.yml --prune
4. python setup.py build_ext -j 4

I get this strange error:

  File "setup.py", line 249
    f"{extension}-source file '{sourcefile}' not found.\n"
                                                         ^
SyntaxError: invalid syntax

I'm not really sure why this is happening when it wasn't before. I'm using the new M1 chip on the mac if that matters.

Also, if I get this issue in the future, where should I be putting it? I don't think should be raising a Github issue for development build errors.

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

[Mxchaeltrxn]

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

I will have a look and see if I can get this repository running in the next few days.

[Mxchaeltrxn]

Closing as I do not want to work on this anymore.