opendap / dap4 support for pydap backend

[Mikejmnez]

There have been some (revitalizing) upgrades over the last year to pydap (with some more to come), including authentication, pydap data model (dap2 vs dap4). This Draft PR aims to

• DOC: Update info on how to authenticate (handled by requests, which recovers credentials )
• DOC: updates links to new (regularly updated) pydap documentation.
• DOC: How to define opendap protocol via URLs.
• DOC: Some info on caching sessions to avoid repeatedly downloading same metadata/data from server. At least point to pydap documentation. (Performance upgrade)
• Removes slashes in variable names (part of dap4 protocol, which allows for hierarchical variables in dataset)
• Support for DataTree.
• Improves testing of backend.
• removes (largely unused) output_grid option. It is now set to False as default in pydap.
• Add deprecation warning of output_grid as an input parameter. It will always be False as default (and there is no need for it to be True, as it only adds unnecessary logic).

This PR enables, for example (check test opendap dap4 url)

URL = "dap4://test.opendap.org/opendap/dap4/SimpleGroup.nc4.h5"
ds = xr.open_datatree(URL, engine='pydap')

[Mikejmnez]

Apologies for all the added labels / contributors. Didn't have much time to work on this pr this week, and so I had to rebase and merge the main branch to this one. Changed files are correct though.

[Mikejmnez]

Ready for review :)

[dcherian]

Thanks @Mikejmnez

The test_cmp_local_file failure is real.

[dcherian]

don't we need this any more?

[Mikejmnez]

I set the output_grid=False by default on pydap, so that should not be necessary. But I'll double check (All the tests passed on the xarray-tests environment)

pytest -v xarray/tests/test_backends.py --run-network-tests

[Mikejmnez]

I updated/upgraded to a newer pydap version on min_all_dep.yml. The failing test test_cmp_local_file is gone as expected. Meaning this is correct. I am also removing output_grid as an argument. It is not really used. So I added a deprecation warning.

[Mikejmnez]

It looks like the failing for ubuntu-latest py3.10 min-all-deps has pydap pinned to 3.4 , which is at least 1 year old (and would explain the errors). The latest conda version is 3.5.5.

[dcherian]

We like to keep compatibility for ~ a year. I could see relaxing this in special cases. Can we do 3.5.0? That was released in August so ~ 8 months ago.

[Mikejmnez]

We like to keep compatibility for ~ a year. I could see relaxing this in special cases. Can we do 3.5.0? That was released in August so ~ 8 months ago.

Cool. I think it is totally doable. There might be one or two features that may be newer than that, I might just have to add a try / except case once or twice for backwards compatibility.

[Mikejmnez]

I set the min version to pydap 3.5.0, which has almost everything except a few methods implemented in 3.5.1. Let me know if there is any other change needed aside from updating/merging main into this branch :) .

[dcherian]

Thanks! I think we can merge. Can you add a note to whats-new mentioning the version bump under Breaking Changes please? It'd be nice to add a note under New Features too described all these great improvements.

[Mikejmnez]

Awesome! Will do shortly

[dcherian]

Sorry one last test failure on the flaky env (which I think is the only one where pydap tests are run)

ValueError: conflicting sizes for dimension 'lat': length 1 on 'subgroup1_var' and length 2 on {'lat': 'subgroup1_var'}

[Mikejmnez]

'lat': 'subgroup1

Hmmm - there might be an issue with that test dataset. I'll look into this

[Mikejmnez]

I don't really know what is going on... I am having a hard time replicating the error. I installed the environment ci/requirements/environment.yml and ran:

pytest xarray/tests/test_backends_datatree.py -v --run-flaky --run-network-tests -W default 

And the test highlighted here passes.

There was, however, another test that failed but only once:

____________________________ TestPyDAPDatatreeIO.test_inherited_coords _________
        with xr.open_dataset(url, engine=self.engine, group="/SimpleGroup") as expected:
/Users/jimenezm/xarray/xarray/tests/test_backends_datatree.py:505: AssertionError

When I run the same testing command, without making any change, all tests pass no matter how many times I try. I repeatedly try to replicate these errors and cannot. I have pydap 3.5.5, xarray installed in dev mode, and python 3.13.3 installed.

For example according to the failed test, the following should fail (python 3.13.3 pydap=3.5.5):

I am kind of confused... Any suggestion @dcherian ?

[Mikejmnez]

Ok! I think likely there was something off with the remote dataset, and it got updated. flaky passes all tests. I see Minimum Version Policy fails because of Pydap=3.5.0 (and I think that will be ok).

I looked at the RuntimeError for the docs/readthedocs build which failed, and I see:

RuntimeError: Unexpected warning in `/home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/10182/doc/user-guide/computation.rst` line 644

Exception occurred:
  File "/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/10182/lib/python3.12/site-packages/IPython/sphinxext/ipython_directive.py", line 603, in process_input
    raise RuntimeError(
RuntimeError: Unexpected warning in `/home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/10182/doc/user-guide/computation.rst` line 644

However, that environment does not even use pydap. I think that is it.

@dcherian let me know if there is any other thing needed.

Have a good weekend!

[dcherian]

Nice, thanks for your work here. Sorry it was a bit annoying at the end.