WIP - Document the way to write reader and writer plugins (#91)

* 📚 add custom plugin writer

* This is an auto-commit, updating project meta data, such as changelog.rst, contributors.rst

* 📚 update writer plugin

* 📚 update plugin writer and reader documentation

* 📚 telling the location of the example codes

Co-authored-by: chfw <chfw@users.noreply.github.com>

.moban.d/docs/source/index.rst.jj2

@@ -91,6 +91,7 @@ get_data(.., library='pyexcel-ods')
    csvz
    sqlalchemy
    django
+   options
    extensions
 
 

docs/source/conf.py

@@ -26,16 +26,16 @@
 copyright = '2015-2020 Onni Software Ltd.'
 author = 'chfw'
 # The short X.Y version
-version = '0.6.0'
+version = '0.5.20'
 # The full version, including alpha/beta/rc tags
-release = '0.5.20'
+release = '0.6.0'
 
 # -- General configuration ---------------------------------------------------
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = [    'sphinx.ext.autodoc',    'sphinx.ext.doctest',    'sphinx.ext.intersphinx',    'sphinx.ext.viewcode',]
+extensions = [    'sphinx.ext.autosummary',    'sphinx.ext.doctest',    'sphinx.ext.intersphinx',    'sphinx.ext.viewcode',    'sphinx.ext.autodoc',]
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']

docs/source/extensions.rst

@@ -1,42 +1,129 @@
 Extend pyexcel-io Tutorial
 ================================================================================
 
-pyexcel-io itself comes with csv support.
+You are welcome extend pyexcel-io to read and write more tabular formats. In
+github repo, you will find two examples in `examples` folder. This section
+explains its implementations to help you write yours.
+
+.. note::
+
+   No longer, you will need to do explicit imports for pyexcel-io extensions.
+   Instead, you install them and manage them via pip.
 
 Reader
 --------------------------------------------------------------------------------
 
 Suppose we have a yaml file, containing a dictionary where the values are
-two dimensional array. The task is write reader plugin to pyexcel-io so that
-we can use get_data() to read it out.
-
-Example yaml data::
+two dimensional array. The task is to write a reader plugin to pyexcel-io so that
+we can use get_data() to read yaml file out.
 
 .. literalinclude:: ../../examples/test.yaml
   :language: yaml
-
-Example code::
 
-.. literalinclude:: ../../examples/custom_yeaml_reader.py
+**Implement IReader**
+
+First, let's impolement reader interface as below. Three implementations are required:
+
+1. `content_array` attribute, is expected to be a list of `NamedContent`
+2. `read_sheet` function, read sheet content by its index.
+3. `close` function, to clean up any file handle
+
+.. literalinclude:: ../../examples/custom_yaml_reader.py
   :language: python
+  :lines: 19-33
+
+**Implement ISheet**
 
+`YourSingleSheet` makes this simple task complex in order to show case its inner
+workings. Two abstract functions require implementation:
+
+1. `row_iterator`: should return a row: either content arry or content index as long as
+                   `column_iterator` understands
+
+2. `column_iterator`: should return cell values one by one.
+
+.. literalinclude:: ../../examples/custom_yaml_reader.py
+  :language: python
+  :lines: 8-16
+
+
+**Plug in pyexcel-io**
+
+Last thing is to register with pyexcel-io about your new reader. `relative_plugin_class_path`
+meant reference from current module, how to refer to `YourReader`. `locations` meant
+the physical presence of the data source: "file", "memory" or "content". "file" means
+files on physical disk. "memory" means a file stream. "content" means a string buffer.
+`stream_type` meant the type of the stream: binary for BytesIO and text for StringIO.
+
+.. literalinclude:: ../../examples/custom_yaml_reader.py
+  :language: python
+  :lines: 36-41
+
+
+**Test your reader **
+
+Let's run the following code and see if it works.
+
+.. literalinclude:: ../../examples/custom_yaml_reader.py
+  :language: python
+  :lines: 43-45
 
 Writer
 --------------------------------------------------------------------------------
 
+Now for the writer, let's write a pyexcel-io writer that write a dictionary of
+two dimentaional arrays back into a yaml file seen above.
 
+** Implement IWriter **
 
-Working with xls, xlsx, and ods formats
-================================================================================
+Two abstract functions are required:
 
-.. note::
+1. `create_sheet` creates a native sheet by sheet name, that understands how to code up the native sheet. Interestingly, it returns your sheet.
+2. `close` function closes file handle if any.
+
+.. literalinclude:: ../../examples/custom_yaml_writer.py
+  :language: python
+  :lines: 18-30
+
+** Implement ISheetWriter **
+
+It is imagined that you will have your own sheet writer. You simply need to figure
+out how to write a row. Row by row write action was already written by `ISheetWrier`.
+
+
+.. literalinclude:: ../../examples/custom_yaml_writer.py
+  :language: python
+  :lines: 7-14
+
+**Plug in pyexcel-io**
+
+Like the reader plugin, we register a writer.
+
+.. literalinclude:: ../../examples/custom_yaml_writer.py
+  :language: python
+  :lines: 33-38
+
+**Test It**
+
+Let's run the following code and please examine `mytest.yaml` yourself.
+
+.. literalinclude:: ../../examples/custom_yaml_writer.py
+  :language: python
+  :lines: 40-46
 
-   No longer, you will need to do explicit imports for pyexcel-io extensions.
-   Instead, you install them and manage them via pip.
 
-Work with physical file
+
+Other pyexcel-io plugins
 -----------------------------------------------------------------------------
 
+Get xls support
+
+.. code-block::
+
+
+   $ pip install pyexcel-xls
+
+
 Here's what is needed::
 
     >>> from pyexcel_io import save_data
@@ -71,7 +158,6 @@ And you can also get the data back::
 
 The same applies to :meth:`pyexcel_io.get_data`.
 
-
 Other formats
 -----------------------------------------------------------------------------
 

docs/source/index.rst

@@ -193,6 +193,7 @@ get_data(.., library='pyexcel-ods')
    csvz
    sqlalchemy
    django
+   options
    extensions
 
 

examples/custom_yaml_writer.py

@@ -0,0 +1,46 @@
+import yaml
+from pyexcel_io import save_data
+from pyexcel_io.plugins import IOPluginInfoChainV2
+from pyexcel_io.plugin_api import IWriter, ISheetWriter
+
+
+class MySheetWriter(ISheetWriter):
+    def __init__(self, sheet_reference):
+        self.native_sheet = sheet_reference
+
+    def write_row(self, data_row):
+        self.native_sheet.append(data_row)
+
+    def close(self):
+        pass
+
+
+class MyWriter(IWriter):
+    def __init__(self, file_name, file_type, **keywords):
+        self.file_name = file_name
+        self.content = {}
+
+    def create_sheet(self, name):
+        array = []
+        self.content[name] = array
+        return MySheetWriter(array)
+
+    def close(self):
+        with open(self.file_name, "w") as f:
+            f.write(yaml.dump(self.content, default_flow_style=False))
+
+
+IOPluginInfoChainV2(__name__).add_a_writer(
+    relative_plugin_class_path="MyWriter",
+    locations=["file"],
+    file_types=["yaml"],
+    stream_type="text",
+)
+
+if __name__ == "__main__":
+    data_dict = {
+        "sheet 1": [[1, 3, 4], [2, 4, 9]],
+        "sheet 2": [["B", "C", "D"]],
+    }
+
+    save_data("mytest.yaml", data_dict)

pyexcel-io.yml

@@ -2,9 +2,9 @@ overrides: "pyexcel.yaml"
 project: "pyexcel-io"
 name: pyexcel-io
 nick_name: io
-version: 0.6.0
+version: 0.5.20
 current_version: 0.6.0
-release: 0.5.20
+release: 0.6.0
 copyright_year: 2015-2020
 moban_command: false
 is_on_conda: true
@@ -31,6 +31,12 @@ keywords:
   - csvz
   - django
   - sqlalchemy
+sphinx_extensions:
+  - sphinx.ext.autosummary
+  - sphinx.ext.doctest
+  - sphinx.ext.intersphinx
+  - sphinx.ext.viewcode
+  - sphinx.ext.autodoc
 description: A python library to read and write structured data in csv, zipped csv format and to/from databases
 python_requires: ">=3.6"
 min_python_version: "3.6"

pyexcel_io/plugin_api/abstract_reader.py

@@ -9,10 +9,13 @@ class IReader(object):
     """
 
     def read_sheet(self, sheet_index) -> ISheet:
-        raise NotImplementedError("")
+        raise NotImplementedError("Read the sheet by index")
 
     def sheet_names(self):
         return [content.name for content in self.content_array]
 
     def __len__(self):
         return len(self.content_array)
+
+    def close(self):
+        raise NotImplementedError("Close the file")

pyexcel_io/plugin_api/abstract_sheet.py

@@ -1,9 +1,9 @@
 class ISheet(object):
     def row_iterator(self):
-        raise NotImplementedError("")
+        raise NotImplementedError("iterate each row")
 
     def column_iterator(self, row):
-        raise NotImplementedError("")
+        raise NotImplementedError("iterate each column at a given row")
 
 
 class ISheetWriter(object):
@@ -16,3 +16,6 @@ def write_array(self, table):
         """
         for row in table:
             self.write_row(row)
+
+    def close(self):
+        raise NotImplementedError("How would you close your file")

setup.py

@@ -40,7 +40,7 @@
     "format and to/from databases"
 )
 URL = "https://github.com/pyexcel/pyexcel-io"
-DOWNLOAD_URL = "%s/archive/0.5.20.tar.gz" % URL
+DOWNLOAD_URL = "%s/archive/0.6.0.tar.gz" % URL
 FILES = ["README.rst", "CHANGELOG.rst"]
 KEYWORDS = [
     "python",
@@ -85,8 +85,8 @@
 }
 # You do not need to read beyond this line
 PUBLISH_COMMAND = "{0} setup.py sdist bdist_wheel upload -r pypi".format(sys.executable)
-GS_COMMAND = ("gs pyexcel-io v0.5.20 " +
-              "Find 0.5.20 in changelog for more details")
+GS_COMMAND = ("gs pyexcel-io v0.6.0 " +
+              "Find 0.6.0 in changelog for more details")
 NO_GS_MESSAGE = ("Automatic github release is disabled. " +
                  "Please install gease to enable it.")
 UPLOAD_FAILED_MSG = (

tests/test_plugin_api.py

@@ -24,6 +24,10 @@ def setUp(self):
     def test_write_row(self):
         self.isheet_writer.write_row([1, 2])
 
+    @raises(NotImplementedError)
+    def test_close(self):
+        self.isheet_writer.close()
+
 
 class TestIReader:
     def setUp(self):
@@ -33,6 +37,10 @@ def setUp(self):
     def test_read_sheet(self):
         self.ireader.read_sheet(1)
 
+    @raises(NotImplementedError)
+    def test_close(self):
+        self.ireader.close()
+
 
 class TestIWriter:
     def setUp(self):