Custom importer Improvements

.github/workflows/testing.yaml

@@ -1,6 +1,7 @@
 name: Testing
 
 on:
+  workflow_dispatch:
   push:
     branches: [ develop, release ]
     paths:
@@ -10,7 +11,6 @@ on:
     - 'poetry.lock'
     - 'pyproject.toml'
   pull_request:
-    branches: [ develop ]
     paths:
     - 'jrnl/**'
     - 'features/**'
@@ -29,6 +29,7 @@ jobs:
       matrix:
         python-version: [ 3.7, 3.8, 3.9, 3.10-dev ]
         os: [ ubuntu-latest, macos-latest, windows-latest ]
+        has_plugins: [ 'no plugins', 'plugins' ]
 
     steps:
     - run: git config --global core.autocrlf false
@@ -74,6 +75,12 @@ jobs:
 
         echo 'DEPS_INSTALLED=true' >> $GITHUB_ENV
 
+      # this is temporary until we implement a 'given' step
+      # to conditionally install plugins during bdd tests
+    - name: Install plugins
+      if: ${{ matrix.has_plugins == 'plugins' }}
+      run: poetry run pip install tests/external_plugins_src
+
     - name: Code formatting (Black)
       if: ${{ matrix.python-version != '3.10-dev' && env.DEPS_INSTALLED == 'true' }}
       run: |

docs/formats.md

@@ -13,7 +13,10 @@ used alone (e.g. `jrnl --format json`) to display all entries from the selected
 
 This page shows examples of all the built-in formats, but since `jrnl` supports adding
 more formats through plugins, you may have more available on your system. Please see
-`jrnl --help` for a list of which formats are available on your system.
+`jrnl --version` for a list of which formats are available on your system. Note
+that plugins can also override built-in formats, so review your installed
+plugins if your output does not match what is listed here. You can also [write
+your own plugins](./plugins.md) to create custom formats.
 
 Any of these formats can be used interchangeably, and are only grouped into "display",
 "data", and "report" formats below for convenience.

docs/plugins.md

@@ -0,0 +1,181 @@
+<!-- Copyright (C) 2012-2021 jrnl contributors
+     License: https://www.gnu.org/licenses/gpl-3.0.html -->
+
+# Extending jrnl
+
+*jrnl* can be extended with custom importers and exporters.
+
+Note that custom importers and exporters can be given the same name as a
+built-in importer or exporter to override it.
+
+Custom Importers and Exporters are traditional Python packages, and are
+installed (into *jrnl*) simply by installing them so they are available to the
+Python interpreter that is running *jrnl*.
+
+Exporter are also used as "formatters" when entries are written to the command
+line.
+
+## Rational
+
+I added this feature because *jrnl* was overall working well for me, but I
+found myself maintaining a private fork so I could have a slightly customized
+export format. Implementing (import and) export plugins was seen as a way to
+maintain my custom exporter without the need to maintaining my private fork.
+
+This implementation tries to keep plugins as light as possible, and as free of
+boilerplate code as reasonable. As well, internal importers and exporters are
+implemented in almost exactly the same way as custom importers and exporters,
+and so it is hoped that plugins can be moved from "contributed" to "internal"
+easily, or that internal plugins can serve as a base and/or a demonstration for
+external plugins.
+
+-- @MinchinWeb, May 2021
+
+## Entry Class
+
+Both the Importers and the Exporters work on the `Entry` class. Below is a
+(selective) description of the class, it's properties and functions:
+
+- **Entry** (class) at `jrnl.Entry.Entry`.
+    - **title** (string): a single line that represents a entry's title.
+    - **date** (datetime.datetime): the date and time assigned to an entry.
+    - **body** (string): the main body of the entry. Can be basically any
+      length. *jrnl* assumes no particular structure here.
+    - **starred** (boolean): is an entry starred? Presumably, starred entries
+      are of particular importance.
+    - **tags** (list of strings): the tags attached to an entry. Each tag
+      includes the pre-facing "tag symbol".
+    - **\_\_init\_\_(journal, date=None, text="", starred=False)**: contractor
+      method
+        - **journal** (*jrnl.Journal.Journal*): a link to an existing Journal
+          class. Mainly used to access it's configuration.
+        - **date** (datetime.datetime)
+        - **text** (string): assumed to include both the title and the body.
+          When the title, body, or tags of an entry are requested, this text
+          will the parsed to determine the tree.
+        - **starred** (boolean)
+
+Entries also have "advanced" metadata if they are using the DayOne backend, but
+we'll ignore that for the purposes of this demo.
+
+## Custom Importer
+
+If you have a (custom) datasource that you want to import into your jrnl
+(perhaps like a blog export), you can write a custom importer to do this.
+
+An importer takes the source data, turns it into Entries and then appends those
+entries to a Journal. Here is a basic Importer, assumed to be provided with a
+nicely formatted JSON file:
+
+~~~ python
+{%
+  include-markdown "../tests/external_plugins_src/jrnl/contrib/importer/simple_json.py"
+  comments=false
+%}
+~~~
+
+Note that the above is very minimal, doesn't do any error checking, and doesn't
+try to import all possible entry metadata.
+
+Another potential use of a custom importer is to effectively create a scripted
+entry creator. For example, maybe each day you want to create a journal entry
+that contains the answers to specific questions; you could create a custom
+"importer" that would ask you the questions, and then create an entry containing
+the answers provided.
+
+Some implementation notes:
+
+- The importer class must be named **Importer**, and should sub-class
+  **jrnl.plugins.base.BaseImporter**.
+- The importer module must be within the **jrnl.contrib.importer** namespace.
+- The importer must not have any `__init__.py` files in the base directories
+  (but you can have one for your importer base directory if it is in a
+  directory rather than a single file).
+- The importer must be installed as a Python package available to the same
+  Python interpreter running jrnl.
+- The importer must expose at least the following the following members:
+    - **version** (string): the version of the plugin. Displayed to help the
+      user debug their installations.
+    - **names** (list of strings): these are the "names" that can be passed to
+      the CLI to involve your importer. If you specify one used by a built-in
+      plugin, it will overwrite it (effectively making the built-in one
+      unavailable).
+    - **import_(journal, input=None)**: the actual importer. Must append
+      entries to the journal passed to it. It is recommended to accept either a
+      filename or standard input as a source.
+
+## Custom Exporter
+
+Custom exporters are useful to make *jrnl*'s data available to other programs.
+One common usecase would to generate the input to be used by a static site
+generator or blogging engine.
+
+An exporter take either a whole journal or a specific entry and exports it.
+Below is a basic JSON Exporter; note that a more extensive JSON exporter is
+included in *jrnl* and so this (if installed) would override the built in
+exporter.
+
+~~~ python
+{%
+  include-markdown "../tests/external_plugins_src/jrnl/contrib/exporter/custom_json.py"
+  comments=false
+%}
+~~~
+
+Note that the above is very minimal, doesn't do any error checking, and doesn't
+export all entry metadata.
+
+Some implementation notes:
+
+- the exporter class must be named **Exporter** and should sub-class
+  **jrnl.plugins.base.BaseExporter**.
+- the exporter module must be within the **jrnl.contrib.exporter** namespace.
+- The exporter must not have any `__init__.py` files in the base directories
+  (but you can have one for your exporter base directory if it is in a
+  directory rather than a single file).
+- The exporter must be installed as a Python package available to the same
+  Python interpreter running jrnl.
+- the exporter should expose at least the following the following members
+  (there are a few more you will need to define if you don't subclass
+  `jrnl.plugins.base.BaseExporter`):
+    - **version** (string): the version of the plugin. Displayed to help the
+      user debug their installations.
+    - **names** (list of strings): these are the "names" that can be passed to
+      the CLI to invole your exporter. If you specific one used by a built-in
+      plugin, it will overwrite it (effectively making the built-in one
+      unavailable).
+    - **extension** (string): the file extention used on exported entries.
+    - **export_entry(entry)**: given an entry, returns a string of the formatted,
+      exported entry.
+    - **export_journal(journal)**: (optional) given a journal, returns a string
+      of the formatted, exported entries of the journal. If not implemented,
+      *jrnl* will call **export_entry()** on each entry in turn and then
+      concatenate the results together.
+
+### Special Exporters
+
+There are a few "special" exporters, in that they are called by *jrnl* in
+situations other than a traditional export. They are:
+
+- **short** -- called by `jrnl --short`. Displays each entry on a single line.
+  The default is to print the timestamp of the entry, followed by the title.
+  The built-in (default) plugin is at `jrnl.plugins.exporter.short`.
+- **default** -- called when a different format is not specified. The built-in
+  (default) plugin is at `jrnl.plugins.exporter.pretty`.
+
+## Development Tips
+
+- Editable installs (`pip install -e ...`) don't seem to play nice with
+  the namespace layout. If your plugin isn't appearing, try a non-editable
+  install of both *jrnl* and your plugin.
+- If you run *jrnl* from the main project root directory (the one that contains
+  *jrnl*'s source code), namespace plugins won't be recognized. This is (I
+  suspect) because the Python interpreter will find your *jrnl* source directory
+  (which doesn't contain your namespace plugins) before it find your
+  "site-packages" directory (i.e. installed packages, which will recognize
+  namespace packages).
+- Don't name your plugin file "testing.py" or it won't be installed (at least
+  automatically) by pip.
+- For examples, you can look to the *jrnl*'s internal importers and exporters.
+  As well, there are some basic external examples included in *jrnl*'s git repo
+  at `tests/external_plugins_src` (including the example code above).

docs_theme/requirements.txt

@@ -1 +1,2 @@
-mkdocs==1.1
+mkdocs==1.1.2
+mkdocs-include-markdown-plugin==2.8.0

features/data/simple_import.json

@@ -0,0 +1,15 @@
+{
+    "entries" :
+    [
+        {
+            "date" : "2013-06-09 15:39",
+            "title" : "My first entry.",
+            "body" : "Everything is alright"
+        },
+        {
+            "date" : "2013-06-10 15:40",
+            "title" : "Life is good.",
+            "body" : "But I'm better."
+        }
+    ]
+}

features/environment.py

@@ -1,9 +1,17 @@
 import os
+from pathlib import Path
 import shutil
 
 from jrnl.os_compat import on_windows
 
+try:
+    from jrnl.contrib.exporter import flag as testing_exporter
+except ImportError:
+    testing_exporter = None
+
 CWD = os.getcwd()
+HERE = Path(__file__).resolve().parent
+TARGET_CWD = HERE.parent  # project root folder
 
 # @see https://behave.readthedocs.io/en/latest/tutorial.html#debug-on-error-in-case-of-step-failures
 BEHAVE_DEBUG_ON_ERROR = False
@@ -15,6 +23,8 @@ def setup_debug_on_error(userdata):
 
 
 def before_all(context):
+    # always start in project root directory
+    os.chdir(TARGET_CWD)
     setup_debug_on_error(context.config.userdata)
 
 
@@ -27,10 +37,10 @@ def before_all(context):
 
 
 def clean_all_working_dirs():
-    if os.path.exists("test.txt"):
-        os.remove("test.txt")
+    if os.path.exists(HERE / "test.txt"):
+        os.remove(HERE / "test.txt")
     for folder in ("configs", "journals", "cache"):
-        working_dir = os.path.join("features", folder)
+        working_dir = HERE / folder
         if os.path.exists(working_dir):
             shutil.rmtree(working_dir)
 
@@ -46,20 +56,28 @@ def before_feature(context, feature):
         feature.skip("Skipping on Windows")
         return
 
+    if "skip_only_with_external_plugins" in feature.tags and testing_exporter is None:
+        feature.skip("Requires test external plugins installed")
+        return
+
+    if "skip_no_external_plugins" in feature.tags and testing_exporter:
+        feature.skip("Skipping with external plugins installed")
+        return
+
 
 def before_scenario(context, scenario):
     """Before each scenario, backup all config and journal test data."""
     # Clean up in case something went wrong
     clean_all_working_dirs()
     for folder in ("configs", "journals"):
-        original = os.path.join("features", "data", folder)
-        working_dir = os.path.join("features", folder)
+        original = HERE / "data" / folder
+        working_dir = HERE / folder
         if not os.path.exists(working_dir):
             os.mkdir(working_dir)
         for filename in os.listdir(original):
-            source = os.path.join(original, filename)
+            source = original / filename
             if os.path.isdir(source):
-                shutil.copytree(source, os.path.join(working_dir, filename))
+                shutil.copytree(source, (working_dir / filename))
             else:
                 shutil.copy2(source, working_dir)
 
@@ -73,11 +91,22 @@ def before_scenario(context, scenario):
         scenario.skip("Skipping on Windows")
         return
 
+    if (
+        "skip_only_with_external_plugins" in scenario.effective_tags
+        and testing_exporter is None
+    ):
+        scenario.skip("Requires test external plugins installed")
+        return
+
+    if "skip_no_external_plugins" in scenario.effective_tags and testing_exporter:
+        scenario.skip("Skipping with external plugins installed")
+        return
+
 
 def after_scenario(context, scenario):
     """After each scenario, restore all test data and remove working_dirs."""
-    if os.getcwd() != CWD:
-        os.chdir(CWD)
+    if os.getcwd() != TARGET_CWD:
+        os.chdir(TARGET_CWD)
 
     # only clean up if debugging is off and the scenario passed
     if BEHAVE_DEBUG_ON_ERROR and scenario.status != "failed":

features/format.feature

@@ -26,6 +26,7 @@ Feature: Custom formats
         |   basic_folder    |
         |   basic_dayone    |
 
+    @skip_no_external_plugins
     Scenario Outline: JSON format
         Given we use the config "<config>.yaml"
         And we use the password "test" if prompted
@@ -48,6 +49,7 @@ Feature: Custom formats
         | basic_folder    |
         | basic_dayone    |
 
+    @skip_no_external_plugins
     Scenario: Exporting dayone to json
         Given we use the config "dayone.yaml"
         When we run "jrnl --export json"
@@ -91,6 +93,7 @@ Feature: Custom formats
         | basic_folder    |
         | basic_dayone    |
 
+    @skip_no_external_plugins
     Scenario Outline: Exporting using filters should only export parts of the journal
         Given we use the config "<config>.yaml"
         And we use the password "test" if prompted
@@ -112,6 +115,7 @@ Feature: Custom formats
         | basic_folder    |
         | basic_dayone    |
 
+    @skip  # template exporters have been removed
     Scenario Outline: Exporting using custom templates
         Given we use the config "<config>.yaml"
         And we load template "sample.template"