REGR: ValueError: setting an array element with a sequence. when assigning an array.array using .loc #46268

pat-schmitt · 2022-03-08T16:49:56Z

Pandas version checks

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np
from array import array

df = pd.DataFrame({"A": [2,4,6,8]})

cenlon = array('d')
cenlon.append(45.345343)

# Problem
df.loc[0, 'CenLon'] = cenlon

# Solution 1
# df.loc[0, 'CenLon'] = np.array(cenlon)

# Solution 2
# df['CenLon'] = ''
# df.loc[0, 'CenLon'] = cenlon

Issue Description

If the Example above is running in a jupyter notebook, the notebook gets stuck and no error message is shown. The error message only appears in the console where the jupyter notebook is running. From the notebook inside it looks like the cell is still computing. The Example also includes two solutions to it, but a decent error message still would be nice.

Expected Behavior

It would be great if the error message would be displayed directly in the notebook or the provided datatype should be supported again.
As background info: this datatype you will get when working with geopandas (https://geopandas.org/en/stable/index.html) and shaply (https://shapely.readthedocs.io/en/stable/manual.html) and you want to extract the centre coordinates of a geometry. Further, I think this was supported in a previous pandas version as the error just appeared on an older notebook (if wanted I could try to find out up to which version or you decide it is not supported anymore then I will let shaply and geopandas know).

Installed Versions

INSTALLED VERSIONS ------------------ commit : 06d2301 python : 3.9.10.final.0 python-bits : 64 OS : Linux OS-release : 5.14.0-1024-oem Version : #26-Ubuntu SMP Thu Feb 17 14:35:50 UTC 2022 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.4.1
numpy : 1.22.3
pytz : 2021.3
dateutil : 2.8.2
pip : 22.0.4
setuptools : 59.8.0
Cython : 0.29.28
pytest : 7.0.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.3
IPython : 8.1.1
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
fsspec : 2022.02.0
gcsfs : None
matplotlib : 3.5.1
numba : None
numexpr : 2.8.0
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.8.0
sqlalchemy : None
tables : 3.7.0
tabulate : None
xarray : 2022.3.0
xlrd : None
xlwt : None
zstandard : None

The text was updated successfully, but these errors were encountered:

phofl · 2022-03-10T11:47:49Z

I get an error on master. Could you please try on master too?

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
ValueError: could not convert string to float: array('d', [45.345343])

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
/tmp/ipykernel_37761/2199181826.py in <module>
      8 
      9 # Problem
---> 10 df.loc[0, 'CenLon'] = cenlon

~/PycharmProjects/pandas/pandas/core/indexing.py in __setitem__(self, key, value)
    797 
    798         iloc = self if self.name == "iloc" else self.obj.iloc
--> 799         iloc._setitem_with_indexer(indexer, value, self.name)
    800 
    801     def _validate_key(self, key, axis: int):

~/PycharmProjects/pandas/pandas/core/indexing.py in _setitem_with_indexer(self, indexer, value, name)
   1732                             indexer, self.obj.axes
   1733                         )
-> 1734                         self._setitem_with_indexer(new_indexer, value, name)
   1735 
   1736                         return

~/PycharmProjects/pandas/pandas/core/indexing.py in _setitem_with_indexer(self, indexer, value, name)
   1777         if take_split_path:
   1778             # We have to operate column-wise
-> 1779             self._setitem_with_indexer_split_path(indexer, value, name)
   1780         else:
   1781             self._setitem_single_block(indexer, value, name)

~/PycharmProjects/pandas/pandas/core/indexing.py in _setitem_with_indexer_split_path(self, indexer, value, name)
   1843             elif self._is_scalar_access(indexer):
   1844                 # We are setting nested data
-> 1845                 self._setitem_single_column(indexer[1], value, pi)
   1846 
   1847             elif len(ilocs) == len(value):

~/PycharmProjects/pandas/pandas/core/indexing.py in _setitem_single_column(self, loc, value, plane_indexer)
   1974 
   1975             orig_values = ser._values
-> 1976             ser._mgr = ser._mgr.setitem((pi,), value)
   1977 
   1978             if ser._values is orig_values:

~/PycharmProjects/pandas/pandas/core/internals/managers.py in setitem(self, indexer, value)
    338             raise ValueError(f"Cannot set values with ndim > {self.ndim}")
    339 
--> 340         return self.apply("setitem", indexer=indexer, value=value)
    341 
    342     def putmask(self, mask, new, align: bool = True):

~/PycharmProjects/pandas/pandas/core/internals/managers.py in apply(self, f, align_keys, ignore_failures, **kwargs)
    302                     applied = b.apply(f, **kwargs)
    303                 else:
--> 304                     applied = getattr(b, f)(**kwargs)
    305             except (TypeError, NotImplementedError):
    306                 if not ignore_failures:

~/PycharmProjects/pandas/pandas/core/internals/blocks.py in setitem(self, indexer, value)
    950                     #  test_iloc_setitem_custom_object
    951                     casted = setitem_datetimelike_compat(values, len(vi), casted)
--> 952             values[indexer] = casted
    953         return self
    954 

ValueError: setting an array element with a sequence.

pat-schmitt · 2022-03-10T15:59:47Z

Hi, thanks for the quick reply.

I just tried out with master (version '1.5.0.dev0+501.g0453fdf4b') and it is still the same behaviour. In the jupyter notebook the cell seems to be busy and no error message is printed out. Only when looking at the console where the jupyter notebook is running I see the following error:

[IPKernelApp] ERROR | Exception in message handler:
ValueError: could not convert string to float: array('d', [45.345343])

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3361, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "/tmp/ipykernel_22879/3431533376.py", line 11, in <cell line: 11>
    df.loc[0, 'CenLon'] = cenlon
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/pandas/core/indexing.py", line 799, in __setitem__
    iloc._setitem_with_indexer(indexer, value, self.name)
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/pandas/core/indexing.py", line 1734, in _setitem_with_indexer
    self._setitem_with_indexer(new_indexer, value, name)
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/pandas/core/indexing.py", line 1779, in _setitem_with_indexer
    self._setitem_with_indexer_split_path(indexer, value, name)
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/pandas/core/indexing.py", line 1845, in _setitem_with_indexer_split_path
    self._setitem_single_column(indexer[1], value, pi)
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/pandas/core/indexing.py", line 1976, in _setitem_single_column
    ser._mgr = ser._mgr.setitem((pi,), value)
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 340, in setitem
    return self.apply("setitem", indexer=indexer, value=value)
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 304, in apply
    applied = getattr(b, f)(**kwargs)
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/pandas/core/internals/blocks.py", line 952, in setitem
    values[indexer] = casted
ValueError: setting an array element with a sequence.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 1979, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'ValueError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3301, in run_ast_nodes
    if await self.run_code(code, result, async_=asy):
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3378, in run_code
    self.showtraceback(running_compiled_code=True)
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 1981, in showtraceback
    stb = self.InteractiveTB.structured_traceback(etype,
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/ultratb.py", line 1105, in structured_traceback
    return FormattedTB.structured_traceback(
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/ultratb.py", line 999, in structured_traceback
    return VerboseTB.structured_traceback(
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/ultratb.py", line 871, in structured_traceback
    formatted_exceptions += self.format_exception_as_a_whole(etype, evalue, etb, lines_of_context,
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/ultratb.py", line 775, in format_exception_as_a_whole
    assert etb is not None
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 1979, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'AssertionError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2900, in _run_cell
    return runner(coro)
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
    coro.send(None)
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3098, in run_cell_async
    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3320, in run_ast_nodes
    self.showtraceback()
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 1981, in showtraceback
    stb = self.InteractiveTB.structured_traceback(etype,
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/ultratb.py", line 1105, in structured_traceback
    return FormattedTB.structured_traceback(
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/ultratb.py", line 999, in structured_traceback
    return VerboseTB.structured_traceback(
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/ultratb.py", line 871, in structured_traceback
    formatted_exceptions += self.format_exception_as_a_whole(etype, evalue, etb, lines_of_context,
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/ultratb.py", line 775, in format_exception_as_a_whole
    assert etb is not None
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 1979, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'AssertionError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 367, in dispatch_shell
    await result
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 662, in execute_request
    reply_content = await reply_content
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/ipykernel/ipkernel.py", line 360, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/ipykernel/zmqshell.py", line 532, in run_cell
    return super().run_cell(*args, **kwargs)
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2854, in run_cell
    result = self._run_cell(
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2905, in _run_cell
    self.showtraceback(running_compiled_code=True)
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 1981, in showtraceback
    stb = self.InteractiveTB.structured_traceback(etype,
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/ultratb.py", line 1105, in structured_traceback
    return FormattedTB.structured_traceback(
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/ultratb.py", line 999, in structured_traceback
    return VerboseTB.structured_traceback(
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/ultratb.py", line 871, in structured_traceback
    formatted_exceptions += self.format_exception_as_a_whole(etype, evalue, etb, lines_of_context,
  File "/home/patrick/mambaforge/envs/pandas_dev/lib/python3.9/site-packages/IPython/core/ultratb.py", line 775, in format_exception_as_a_whole
    assert etb is not None
AssertionError

So the ValueError is the same, but not printed out in the notebook, and also some more exceptions occur.

Thanks for your help!

simonjayhawkins · 2022-04-05T12:00:00Z

Thanks @pat-schmitt for the report

Further, I think this was supported in a previous pandas version

looks like it was working in 1.3.5 so labelling as a regression for now pending further investigation.

simonjayhawkins · 2022-04-05T13:18:17Z

looks like it was working in 1.3.5 so labelling as a regression for now pending further investigation.

first bad commit: [018590c] BUG: enable test_numpy tests with ArrayManager (#42780)

cc @jbrockmendel

simonjayhawkins · 2022-06-22T15:31:45Z

moving to 1.4.4

jorisvandenbossche · 2022-08-12T13:23:02Z

BTW, the fact that a notebook is hanging and you see that long traceback in IPython, that was a bug in IPython that should be fixed in the latest version (ipython/ipython#12467)

That still leaves the regression or behaviour change in pandas. I started investigating this and have an PR at #48057

pat-schmitt added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Mar 8, 2022

pat-schmitt mentioned this issue Mar 8, 2022

Notebook / Pandas bug when assigning cenlon/cenlat OGGM/tutorials#56

Closed

phofl added the Needs Info Clarification about behavior needed to assess issue label Mar 10, 2022

mroeschke removed the Needs Triage Issue that has not been reviewed by a pandas team member label Mar 17, 2022

simonjayhawkins added Regression Functionality that used to work in a prior pandas version and removed Bug Needs Info Clarification about behavior needed to assess issue labels Apr 5, 2022

simonjayhawkins added this to the 1.4.3 milestone Apr 5, 2022

simonjayhawkins added the Compat pandas objects compatability with Numpy or Python functions label Apr 5, 2022

simonjayhawkins added a commit to simonjayhawkins/pandas that referenced this issue Apr 5, 2022

code sample for pandas-dev#46268

a77bbf3

simonjayhawkins changed the title ~~BUG: Unexpected jupyter notebook behavior when assigning an array.array variable~~ REGR: ValueError: setting an array element with a sequence. when assigning an array.array using .loc Jun 22, 2022

simonjayhawkins added the Indexing Related to indexing on series/frames, not to indexes themselves label Jun 22, 2022

simonjayhawkins modified the milestones: 1.4.3, 1.4.4 Jun 22, 2022

jorisvandenbossche mentioned this issue Aug 12, 2022

REGR: fix regression in scalar setitem with setting a length-1 array-like #48057

Merged

4 tasks

mroeschke closed this as completed in #48057 Aug 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

REGR: ValueError: setting an array element with a sequence. when assigning an array.array using .loc #46268

REGR: ValueError: setting an array element with a sequence. when assigning an array.array using .loc #46268

pat-schmitt commented Mar 8, 2022 •

edited

Loading

phofl commented Mar 10, 2022 •

edited

Loading

pat-schmitt commented Mar 10, 2022

simonjayhawkins commented Apr 5, 2022

simonjayhawkins commented Apr 5, 2022

simonjayhawkins commented Jun 22, 2022

jorisvandenbossche commented Aug 12, 2022

REGR: ValueError: setting an array element with a sequence. when assigning an array.array using .loc #46268

REGR: ValueError: setting an array element with a sequence. when assigning an array.array using .loc #46268

Comments

pat-schmitt commented Mar 8, 2022 • edited Loading

Pandas version checks

Reproducible Example

Issue Description

Expected Behavior

Installed Versions

phofl commented Mar 10, 2022 • edited Loading

pat-schmitt commented Mar 10, 2022

simonjayhawkins commented Apr 5, 2022

simonjayhawkins commented Apr 5, 2022

simonjayhawkins commented Jun 22, 2022

jorisvandenbossche commented Aug 12, 2022

pat-schmitt commented Mar 8, 2022 •

edited

Loading

phofl commented Mar 10, 2022 •

edited

Loading