Remove dask dependency?

[TomNicholas]

Is dask still required for this package? As far as I can tell it's not, but it's still listed as a required dependency. It would be helpful to clarify that.

[alxmrs]

Dask isn't a required package, per se. However, since I require the input ds to be chunked, and dask seems like the default ChunkManager, it almost seems like an essential dependency right now. For instance, when I remove dask altogether in #80, I hit this error in every test (see below). Should I make dask a test-only dependency? But, if this always happens when we chunk a dataset, then should dask be a proper dependency (until this changes)? I don't want to confuse users of this package.

_________________ SqlErrorHandlingTestCase.test_syntax_errors __________________

self = <xarray_sql.sql_test.SqlErrorHandlingTestCase testMethod=test_syntax_errors>

    def setUp(self):
      """Set up dataset for error testing."""
      self.air = xr.tutorial.open_dataset('air_temperature')
      self.air_small = self.air.isel(
          time=slice(0, 10), lat=slice(0, 5), lon=slice(0, 8)
>     ).chunk({'time': 5})
        ^^^^^^^^^^^^^^^^^^

xarray_sql/sql_test.py:747: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.venv/lib/python3.13/site-packages/xarray/core/dataset.py:2524: in chunk
    chunkmanager = guess_chunkmanager(chunked_array_type)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

manager = 'dask'

    def guess_chunkmanager(
        manager: str | ChunkManagerEntrypoint[Any] | None,
    ) -> ChunkManagerEntrypoint[Any]:
        """
        Get namespace of chunk-handling methods, guessing from what's available.
    
        If the name of a specific ChunkManager is given (e.g. "dask"), then use that.
        Else use whatever is installed, defaulting to dask if there are multiple options.
        """
    
        available_chunkmanagers = list_chunkmanagers()
    
        if manager is None:
            if len(available_chunkmanagers) == 1:
                # use the only option available
                manager = next(iter(available_chunkmanagers.keys()))
            else:
                # use the one in options (default dask)
                manager = OPTIONS["chunk_manager"]
    
        if isinstance(manager, str):
            if manager not in available_chunkmanagers and manager in KNOWN_CHUNKMANAGERS:
>               raise ImportError(
                    f"chunk manager {manager!r} is not available."
                    f" Please make sure {KNOWN_CHUNKMANAGERS[manager]!r} is installed"
                    " and importable."
E                   ImportError: chunk manager 'dask' is not available. Please make sure 'dask' is installed and importable.

.venv/lib/python3.13/site-packages/xarray/namedarray/parallelcompat.py:116: ImportError
=============================== warnings summary ===============================
xarray_sql/df_test.py::ExplodeTest::test_cardinality
  <frozen importlib._bootstrap>:488: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

[TomNicholas]

But is dask actually being used for anything? Zarr has chunks - is there some way to thread this through without having dask as an intermediary?

Where is the code that actually turns the dask-chunked Dataset into the type of chunks that xarray-sql needs?

[alxmrs]

To be fair, I get failures in my test code only from lines that look like this:

  def setUp(self) -> None:
    self.air = xr.tutorial.open_dataset('air_temperature')
    self.chunks = {'time': 240}
    self.air = self.air.chunk(self.chunks)  # <----------- this is where we get this error

Honestly, I would give me a lot of piece of mind if you (the author of the chunk manager) "blessed" moving dask as a test-only dependency, since this library doesn't use this method for it's chunking strategy at all.

[TomNicholas]

I think that should be possible, but I want to better understand how this library actually uses the chunks first.

I require the input ds to be chunked

Are you basically just using the knowledge that there are (on-disk) chunks to know that lazy-loading a subset of the data won't trigger loading all of it into memory at once?

[TomNicholas]

Like, you're not explicitly passing the dask tasks to another layer or anything like that?

[alxmrs]

Check out this block slices implementation, i.e. map_blocks from xarray: https://github.com/alxmrs/xarray-sql/blob/main/xarray_sql/df.py This is the way we make use of chunks, which is to say, no, we don’t mess with dask layers here.

…

On Thu, Oct 2, 2025 at 7:18 PM Tom Nicholas ***@***.***> wrote: *TomNicholas* left a comment (alxmrs/xarray-sql#79) <#79 (comment)> Like, you're not explicitly passing the dask tasks to another layer or anything like that? — Reply to this email directly, view it on GitHub <#79 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARXAB5IJ5Q5K6ZSCRBX7M33VVM6XAVCNFSM6AAAAACH2FUSTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGNRSGI2DONRUGA> . You are receiving this because you commented.Message ID: ***@***.***>