Skip to content

danielballan/reader-intake-adapter

master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 

This proof of concept aims to demonstrate that if a library implements this API:

class MyReader:
    container = '...'  # fully-qualified class name of type(self.read())

    def read(self):
        """
        Return any type that reader_adapter knows about.

        This currently includes dask.array.core.Array and
        dask.dataframe.core.DataFrame, but could grow to include numpy.ndarray,
        pandas.DataFrame, and xarray types.
        """
        ...

    def close(self):
        ...

we can automatically build a fully-compliant intake DataSource around it. (The cache and persist functionality are not filled in to this proof of concept yet, but I believe there is a path to add that as well, and we should do so.)

Currently, libraries that want to implement a DataSource must do one of the following:

  • Accept an intake dependency, subclass DataSource, and define _get_schema, _get_partition, read, to_dask, and (sometimes) read_partition.
  • Avoid the intake dependency and implement the DataSource API from scratch. (I am not aware of any packages that do this, but it's conceivable.)

This proposal has two aims:

  • Make it easier to write and maintain a plugin for a given format by moving logic that is generic for a given container type out of the format-specific plugin.
  • Provide a path for existing libraries like PIMS, imageio, tifffile, area-detector-handlers, and others to potentially add intake-compatible plugins and entrypoints without taking on an intake dependency.

As is, this shows that without any changes to intake any package could define a Reader as above, import reader_adapter (a zero-dependency project that could theorhetically go up on PyPI), and use

MyDataSource = reader_adapter.adapt(MyReader, 'MyDataSource')

to auto-build a complete DataSource out of their Reader and declare that as an 'intake.drivers' entrypoints. This is what my_tiff_package and my_fwf_package do. The example scripts tiff_example.py and fwf_example.py show that these work.

We could make this even more convenient for package authors by making one change to intake itself. If intake were to incorporate this reader_adapter code and define a third entrypoint intake.readers (alongside the more generic intake.drivers and the recently-added intake.catalogs) then intake could do the reader-to-DataSource conversion itself transparently. The package author would only need to define the simple Reader API and declare it as an 'intake.readers' entrypoint.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages