Typing for open_dataset/array/mfdataset and to_netcdf/zarr #6612

headtr1ck · 2022-05-15T18:09:15Z

Closes Type hints for xr.open_dataset #5382

Mypy is not able to compute the overloads of to_netcdf properly (too many Unions).
I had to add some # type: ignore

I am not sure about the types of

concat_dim
combine_attrs: the Callable part is still with Anys

for more information, see https://pre-commit.ci

…yping

for more information, see https://pre-commit.ci

…yping

for more information, see https://pre-commit.ci

max-sixty

Herculean effort on this!

Could we add some -> None to some of the methods in test_backends.py so we can test the annotations? Particularly for these methods "at the edge", they're unlikely to be called by many methods within xarray, and so really the only way to test them is to add annotations to tests.

Thanks a lot @headtr1ck !

xarray/backends/api.py

xarray/core/dataset.py

max-sixty · 2022-05-15T23:40:24Z

I think merging main may fix the benchmark errors. (If you enable permissions for maintainers to modify the PR then GH offers a button to do this, FWIW)

headtr1ck · 2022-05-16T11:08:02Z

I think merging main may fix the benchmark errors. (If you enable permissions for maintainers to modify the PR then GH offers a button to do this, FWIW)

It shows as enabled for me?

for more information, see https://pre-commit.ci

…yping

for more information, see https://pre-commit.ci

…yping

for more information, see https://pre-commit.ci

…yping

for more information, see https://pre-commit.ci

…yping

for more information, see https://pre-commit.ci

…yping

max-sixty

Excellent, this is a great upgrade to the typing.

There are a couple of small errors, we can def merge when those are solved. Thank you @headtr1ck !

xarray/backends/api.py

xarray/core/dataarray.py

headtr1ck · 2022-05-17T18:32:27Z

Puh, 1k LOC diff, that was quite the rabbit hole...

There is still one open point:
The concat_dim arg is not clear to me what the type is actually allowed.
The Index is only there because the docstring mentions it, in the code it is not checked so this might be actually wrong.
Someone with a better understanding of this part should check this type!

max-sixty · 2022-05-17T19:31:55Z

Puh, 1k LOC diff, that was quite the rabbit hole...

Yes!

Re the concat_dim — it looks quite reasonable, though I'm not the expert. I would vote to merge before we get conflicts as I recognize it must have taken a lot of work to get here. I will also add a label and ask someone to have a look at our next dev meeting, though I suspect through time we'll find any improvements through bug reports & tests.

Thank you very much @headtr1ck !

* main: concatenate docs style (pydata#6621) Typing for open_dataset/array/mfdataset and to_netcdf/zarr (pydata#6612) {full,zeros,ones}_like typing (pydata#6611)

commit 398f1b6 Author: dcherian <deepak@cherian.net> Date: Fri May 20 08:47:56 2022 -0600 Backward compatibility dask commit bde40e4 Merge: 0783df3 4cae8d0 Author: dcherian <deepak@cherian.net> Date: Fri May 20 07:54:48 2022 -0600 Merge branch 'main' into dask-datetime-to-numeric * main: concatenate docs style (pydata#6621) Typing for open_dataset/array/mfdataset and to_netcdf/zarr (pydata#6612) {full,zeros,ones}_like typing (pydata#6611) commit 0783df3 Merge: 5cff4f1 8de7061 Author: dcherian <deepak@cherian.net> Date: Sun May 15 21:03:50 2022 -0600 Merge branch 'main' into dask-datetime-to-numeric * main: (24 commits) Fix overflow issue in decode_cf_datetime for dtypes <= np.uint32 (pydata#6598) Enable flox in GroupBy and resample (pydata#5734) Add setuptools as dependency in ASV benchmark CI (pydata#6609) change polyval dim ordering (pydata#6601) re-add timedelta support for polyval (pydata#6599) Minor Dataset.map docstr clarification (pydata#6595) New inline_array kwarg for open_dataset (pydata#6566) Fix polyval overloads (pydata#6593) Restore old MultiIndex dropping behaviour (pydata#6592) [docs] add Dataset.assign_coords example (pydata#6336) (pydata#6558) Fix zarr append dtype checks (pydata#6476) Add missing space in exception message (pydata#6590) Doc Link to accessors list in extending-xarray.rst (pydata#6587) Fix Dataset/DataArray.isel with drop=True and scalar DataArray indexes (pydata#6579) Add some warnings about rechunking to the docs (pydata#6569) [pre-commit.ci] pre-commit autoupdate (pydata#6584) terminology.rst: fix link to Unidata's "netcdf_dataset_components" (pydata#6583) Allow string formatting of scalar DataArrays (pydata#5981) Fix mypy issues & reenable in tests (pydata#6581) polyval: Use Horner's algorithm + support chunked inputs (pydata#6548) ... commit 5cff4f1 Merge: dfe200d 6144c61 Author: Maximilian Roos <5635139+max-sixty@users.noreply.github.com> Date: Sun May 1 15:16:33 2022 -0700 Merge branch 'main' into dask-datetime-to-numeric commit dfe200d Author: dcherian <deepak@cherian.net> Date: Sun May 1 11:04:03 2022 -0600 Minor cleanup commit 35ed378 Author: dcherian <deepak@cherian.net> Date: Sun May 1 10:57:36 2022 -0600 Support dask arrays in datetime_to_numeric

* main: (95 commits) Use `zarr` to validate attrs when writing to zarr (pydata#6636) Add pre-commit hook to check CITATION.cff (pydata#6658) Fix kwargs used for extrapolation in docs (pydata#6639) Fix notebooks' HTML links (pydata#6655) Doc index update (pydata#6530) CFTime support for polyval (pydata#6624) Support dask arrays in datetime_to_numeric (pydata#6556) [pre-commit.ci] pre-commit autoupdate (pydata#6654) 0-padded month. (pydata#6653) [test-upstream] import `cleanup` fixture from `distributed` (pydata#6650) Allow all interp methods in typing (pydata#6647) Typing support for custom backends (pydata#6651) Improved DataArray typing (pydata#6637) Adjust code comments & types from pydata#6638 (pydata#6642) Typing of `str` and `dt` accessors (pydata#6641) Feature/to dict encoding (pydata#6635) fix {full,zeros,ones}_like overloads (pydata#6630) Mypy badge (pydata#6626) concatenate docs style (pydata#6621) Typing for open_dataset/array/mfdataset and to_netcdf/zarr (pydata#6612) ...

headtr1ck and others added 13 commits May 15, 2022 18:14

type filename and chunks

cb5ce9a

type open_dataset, open_dataarray, open_mfdataset

e0a3c46

type to_netcdf

4a02e87

add return doc to Dataset.to_netcdf

27daca4

[pre-commit.ci] auto fixes from pre-commit.com hooks

664e35a

for more information, see https://pre-commit.ci

fix import error

3102f73

Merge branch 'open_typing' of github.com:headtr1ck/xarray into open_t…

46ca5c2

…yping

[pre-commit.ci] auto fixes from pre-commit.com hooks

2007cd5

for more information, see https://pre-commit.ci

replace tuple[x] by Tuple[x] for py3.8

c7dd188

Merge branch 'open_typing' of github.com:headtr1ck/xarray into open_t…

c4f1e88

…yping

fix some merge errors

befd121

add overloads to to_zarr

86c5536

[pre-commit.ci] auto fixes from pre-commit.com hooks

e5b7e48

for more information, see https://pre-commit.ci

headtr1ck changed the title ~~Typing for open_dataset/array/mfdataset and to_netcdf~~ Typing for open_dataset/array/mfdataset and to_netcdf/zarr May 15, 2022

max-sixty approved these changes May 15, 2022

View reviewed changes

max-sixty reviewed May 15, 2022

View reviewed changes

xarray/core/dataset.py Show resolved Hide resolved

headtr1ck and others added 12 commits May 16, 2022 19:16

fix absolute import

a60e7a6

CamelCase type vars

6f7c875

move some literal type to core.types

6f454ee

add JoinOptions to core.types

e0bc3cb

Merge branch 'main' into open_typing

6ec0f2e

[pre-commit.ci] auto fixes from pre-commit.com hooks

2fa1934

for more information, see https://pre-commit.ci

add some blank lines under bullet lists in docs

dd1dd00

add comments to overloads

0f1a1df

Merge branch 'open_typing' of github.com:headtr1ck/xarray into open_t…

6800356

…yping

some more typing

0d08118

fix absolute import

c6aa646

[pre-commit.ci] auto fixes from pre-commit.com hooks

9376d36

for more information, see https://pre-commit.ci

headtr1ck and others added 20 commits May 17, 2022 10:53

fix DataArray import

820cb11

[pre-commit.ci] auto fixes from pre-commit.com hooks

768ba79

for more information, see https://pre-commit.ci

fix _dataset_concat arg name

ebf1e7d

Merge branch 'open_typing' of github.com:headtr1ck/xarray into open_t…

fd1d14a

…yping

fix DataArray not imported

d64c04a

remove xr import in Dataset

c5c3125

[pre-commit.ci] auto fixes from pre-commit.com hooks

d6b7cc2

for more information, see https://pre-commit.ci

some more typing

7367248

replace some Sequence by Iterable

c30ae55

Merge branch 'open_typing' of github.com:headtr1ck/xarray into open_t…

fb8ac8d

…yping

fix wrong default in docstring

68b3d58

fix docstring indentation

5166bc4

fix overloads and type some tests

9543dea

[pre-commit.ci] auto fixes from pre-commit.com hooks

b2d02a7

for more information, see https://pre-commit.ci

fix open_mfdataset typing

502f7c1

Merge branch 'open_typing' of github.com:headtr1ck/xarray into open_t…

2084bdb

…yping

minor update of docstring

f2afb6f

[pre-commit.ci] auto fixes from pre-commit.com hooks

d8e39a0

for more information, see https://pre-commit.ci

Merge branch 'open_typing' of github.com:headtr1ck/xarray into open_t…

4b4ca57

…yping

remove uneccesary import

5043f85

max-sixty approved these changes May 17, 2022

View reviewed changes

xarray/backends/api.py Show resolved Hide resolved

xarray/backends/api.py Outdated Show resolved Hide resolved

xarray/core/dataarray.py Outdated Show resolved Hide resolved

fix overloads of to_netcdf

fd400e5

minor docstring update

e277727

max-sixty merged commit 6b1d97a into pydata:main May 17, 2022

headtr1ck deleted the open_typing branch May 19, 2022 16:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Typing for open_dataset/array/mfdataset and to_netcdf/zarr #6612

Typing for open_dataset/array/mfdataset and to_netcdf/zarr #6612

headtr1ck commented May 15, 2022

max-sixty left a comment

max-sixty commented May 15, 2022

headtr1ck commented May 16, 2022

max-sixty left a comment

headtr1ck commented May 17, 2022

max-sixty commented May 17, 2022

Typing for open_dataset/array/mfdataset and to_netcdf/zarr #6612

Typing for open_dataset/array/mfdataset and to_netcdf/zarr #6612

Conversation

headtr1ck commented May 15, 2022

max-sixty left a comment

Choose a reason for hiding this comment

max-sixty commented May 15, 2022

headtr1ck commented May 16, 2022

max-sixty left a comment

Choose a reason for hiding this comment

headtr1ck commented May 17, 2022

max-sixty commented May 17, 2022