Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

iris.util.broadcast_to_shape does not work with lazy data #5295

Closed
bouweandela opened this issue May 2, 2023 · 3 comments · Fixed by #5307
Closed

iris.util.broadcast_to_shape does not work with lazy data #5295

bouweandela opened this issue May 2, 2023 · 3 comments · Fixed by #5307
Assignees

Comments

@bouweandela
Copy link
Member

🐛 Bug Report

How To Reproduce

Steps to reproduce the behaviour:

Run the following code:

>>> import dask.array as da
>>> from iris.util import broadcast_to_shape
>>> a = da.array([[1, 2, 3], [4, 5, 6]])
>>> b = broadcast_to_shape(a, (5, 2, 6, 3), (1, 3))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bandela/mambaforge/envs/test/lib/python3.11/site-packages/iris/util.py", line 98, in broadcast_to_shape
    array_view = _broadcast_helper(array)
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bandela/mambaforge/envs/test/lib/python3.11/site-packages/iris/util.py", line 95, in _broadcast_helper
    strides[dim] = a.strides[idim]
                   ^^^^^^^^^
AttributeError: 'Array' object has no attribute 'strides'

Expected behaviour

The function documentation mentions that "This function maintains laziness when called; it does not realise data.", so I would expect the function to return a Dask array instead of crash when following the example from the docstring with a dask array as input.

Screenshots

Environment

  • OS & Version: Ubuntu 23.04
  • Iris Version: 3.5
$ mamba env export 

gives

name: test
channels:
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - antlr-python-runtime=4.7.2=py311h38be061_1003
  - brotli=1.0.9=h166bdaf_8
  - brotli-bin=1.0.9=h166bdaf_8
  - brotlipy=0.7.0=py311hd4cff14_1005
  - bzip2=1.0.8=h7f98852_4
  - c-ares=1.18.1=h7f98852_0
  - ca-certificates=2022.12.7=ha878542_0
  - cartopy=0.21.1=py311hd88b842_1
  - certifi=2022.12.7=pyhd8ed1ab_0
  - cf-units=3.2.0=py311hcb2cf0a_0
  - cffi=1.15.1=py311h409f033_3
  - cftime=1.6.2=py311h4c7f6c3_1
  - charset-normalizer=3.1.0=pyhd8ed1ab_0
  - click=8.1.3=unix_pyhd8ed1ab_2
  - cloudpickle=2.2.1=pyhd8ed1ab_0
  - contourpy=1.0.7=py311ha3edf6b_0
  - cryptography=40.0.2=py311h9b4c7bb_0
  - curl=8.0.1=h588be90_0
  - cycler=0.11.0=pyhd8ed1ab_0
  - dask-core=2023.4.1=pyhd8ed1ab_0
  - expat=2.5.0=hcb278e6_1
  - fonttools=4.39.3=py311h2582759_0
  - freetype=2.12.1=hca18f0e_1
  - fsspec=2023.4.0=pyh1a96a4e_0
  - geos=3.11.2=hcb278e6_0
  - hdf4=4.2.15=h9772cbc_5
  - hdf5=1.12.2=nompi_h4df4325_101
  - icu=72.1=hcb278e6_0
  - idna=3.4=pyhd8ed1ab_0
  - importlib-metadata=6.6.0=pyha770c72_0
  - importlib_metadata=6.6.0=hd8ed1ab_0
  - iris=3.5.0=pyhd8ed1ab_0
  - jinja2=3.1.2=pyhd8ed1ab_1
  - jpeg=9e=h0b41bf4_3
  - keyutils=1.6.1=h166bdaf_0
  - kiwisolver=1.4.4=py311h4dd048b_1
  - krb5=1.20.1=h81ceb04_0
  - lcms2=2.15=hfd0df8a_0
  - ld_impl_linux-64=2.40=h41732ed_0
  - lerc=4.0.0=h27087fc_0
  - libaec=1.0.6=hcb278e6_1
  - libblas=3.9.0=16_linux64_openblas
  - libbrotlicommon=1.0.9=h166bdaf_8
  - libbrotlidec=1.0.9=h166bdaf_8
  - libbrotlienc=1.0.9=h166bdaf_8
  - libcblas=3.9.0=16_linux64_openblas
  - libcurl=8.0.1=h588be90_0
  - libdeflate=1.17=h0b41bf4_0
  - libedit=3.1.20191231=he28a2e2_2
  - libev=4.33=h516909a_1
  - libexpat=2.5.0=hcb278e6_1
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=12.2.0=h65d4601_19
  - libgfortran-ng=12.2.0=h69a702a_19
  - libgfortran5=12.2.0=h337968e_19
  - libgomp=12.2.0=h65d4601_19
  - libiconv=1.17=h166bdaf_0
  - liblapack=3.9.0=16_linux64_openblas
  - libnetcdf=4.8.1=nompi_h261ec11_106
  - libnghttp2=1.52.0=h61bc06f_0
  - libnsl=2.0.0=h7f98852_0
  - libopenblas=0.3.21=pthreads_h78a6416_3
  - libpng=1.6.39=h753d276_0
  - libsqlite=3.40.0=h753d276_1
  - libssh2=1.10.0=hf14f497_3
  - libstdcxx-ng=12.2.0=h46fd767_19
  - libtiff=4.5.0=h6adf6a1_2
  - libuuid=2.38.1=h0b41bf4_0
  - libwebp-base=1.3.0=h0b41bf4_0
  - libxcb=1.13=h7f98852_1004
  - libxml2=2.10.4=hfdac1af_0
  - libzip=1.9.2=hc929e4a_1
  - libzlib=1.2.13=h166bdaf_4
  - locket=1.0.0=pyhd8ed1ab_0
  - markupsafe=2.1.2=py311h2582759_0
  - matplotlib-base=3.7.1=py311h8597a09_0
  - munkres=1.1.4=pyh9f0ad1d_0
  - ncurses=6.3=h27087fc_1
  - netcdf4=1.6.2=nompi_py311hc6fcf29_100
  - numpy=1.24.3=py311h64a7726_0
  - openjpeg=2.5.0=hfec8fc6_2
  - openssl=3.1.0=hd590300_2
  - packaging=23.1=pyhd8ed1ab_0
  - partd=1.4.0=pyhd8ed1ab_0
  - pillow=9.4.0=py311h50def17_1
  - pip=23.1.2=pyhd8ed1ab_0
  - platformdirs=3.5.0=pyhd8ed1ab_0
  - pooch=1.7.0=pyha770c72_3
  - proj=9.2.0=h8ffa02c_0
  - pthread-stubs=0.4=h36c2ea0_1001
  - pycparser=2.21=pyhd8ed1ab_0
  - pyopenssl=23.1.1=pyhd8ed1ab_0
  - pyparsing=3.0.9=pyhd8ed1ab_0
  - pyproj=3.5.0=py311h1850bce_1
  - pyshp=2.3.1=pyhd8ed1ab_0
  - pysocks=1.7.1=pyha2e5f31_6
  - python=3.11.3=h2755cc3_0_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python-xxhash=3.2.0=py311h2582759_0
  - python_abi=3.11=3_cp311
  - pyyaml=6.0=py311hd4cff14_5
  - readline=8.2=h8228510_1
  - requests=2.29.0=pyhd8ed1ab_0
  - scipy=1.10.1=py311h8e6699e_0
  - setuptools=67.7.2=pyhd8ed1ab_0
  - shapely=2.0.1=py311h54d622a_1
  - six=1.16.0=pyh6c4a22f_0
  - sqlite=3.40.0=h4ff8645_1
  - tk=8.6.12=h27826a3_0
  - toolz=0.12.0=pyhd8ed1ab_0
  - typing-extensions=4.5.0=hd8ed1ab_0
  - typing_extensions=4.5.0=pyha770c72_0
  - tzdata=2023c=h71feb2d_0
  - udunits2=2.2.28=hc3e0081_0
  - urllib3=1.26.15=pyhd8ed1ab_0
  - wheel=0.40.0=pyhd8ed1ab_0
  - xorg-libxau=1.0.9=h7f98852_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xxhash=0.8.1=h0b41bf4_0
  - xz=5.2.6=h166bdaf_0
  - yaml=0.2.5=h7f98852_2
  - zipp=3.15.0=pyhd8ed1ab_0
  - zlib=1.2.13=h166bdaf_4
  - zstd=1.5.2=h3eb15da_6

Additional context

Click to expand this section...
Please add additional verbose information in this section e.g., code, output, tracebacks, screenshots etc
@trexfeathers
Copy link
Contributor

@ESadek-MO do you have the code that you used to test laziness on this function?

@ESadek-MO
Copy link
Contributor

Unfortunately, due to a lack of foresight, I do not. There's every chance some of these feel through the gap and were incorrectly documented.
I'm currently off, but will try and replicate the test when I return.

@rcomer
Copy link
Member

rcomer commented May 3, 2023

This function is properly old, and I think might even predate numpy.broadcast_to. It definitely predates Iris' use of dask. I suggest the best thing here would be to simplify the code to use both numpy.broadcast_to and dask.array.broadcast_to.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants