Multiple fixes, feature adds, cleanup. Also partially completes issue…

… 44… (#48)

* Multiple fixes, feature adds, cleanup. Also partially completes issue #44

Fixes:
	Renamed upload function to do_upload as we use the option 'upload' as a boolean elsewhere. Confusing.
	Fixed a missing default for 'upload' in multiparser class that wasn't allowing 'upload' to be used as a config file option.

Cleanup/Housekeeping:
	Changed some errors handling from 'raise error' to simple print error message and exit with a exit code of one (1). Some errors such as invalid extension or file not found don't need the traceback code, they just need to output notification to the user and set the shell exit code are user errors, not code errors.
	Moved a some variables to upload since they are only needed there.
	Moved the story builder inside a conditional to add ability to skip story generation via command line switch (--nostory), or individual story option. Will allow you to temporarily turn off a particulary story without removing it from your config.

New features:

  * Added --nostory option. Allows you to skip all the story generation and process an upload only. In this case the 'output' file is used as the file to upload directly.
  * Added '--noreplace' and '--noupload' as command line only options to allow overriding config file settings (for testing (or other uses ?)) without having to constantly edit config files.
  * Added 'upload' specified in config or global
  * Added 'output' can be specified in config, global, or command line.
  * Added sanity checking if you provide more than one 'output'.
  * Created default config files
  * Added replace to default options as 'False'
  * Added cleanup option (delete target goosepaper after upload success)
  * Added formatter class to allow newlines used in help for command line options. Makes for cleaner reading on some help text (IMO). Mostly mine since my default setting is --verbose --verbose. :)
  * Added a per story 'skip' option (similar to nostory global option but per story source) which will allow you to skip processing a particular story provider without having to remove your entire config. Useful for if the source is having issues but you don't want to remove your entire entry from that Goosepaper.
  * Added an entry to examples to show use of 'skip'
  * Added checks to make sure that either default config files exist or you specified one on the command line.
  * Added global config preference files in order of precedence. Command line options superseded all config file settings.
      ~/.goosepaper.json (HOME environment from os)
      ./goosepaper.json (current working directory when goosepaper called)
      --config <configfile.json> (specified as argument)
  * Added '--showconfig' command line switch to allow both user and dev debugging of config file and command line options since there are four ways to pass settings and options. Shows each file in order parsed as well as the final config
  * Added default goosepaper.json containing all the (current) operational variables set to operate with the least impact to any existing workflow process (i.e. no change to current behavior as of 0.3.1).

* Left in debug code that failed on 3.8 test. Removed

* Removed nneeded module import left over from debugging

* Code format cleanup

* Remove two unimplemented options pending rethink

Co-authored-by: Jordan Matelsky <j6k4m8@users.noreply.github.com>

example-config.json

@@ -30,6 +30,14 @@
         {
             "provider": "reddit",
             "config": { "subreddit": "todayilearned" }
-        }
+        },
+	{
+	    "provider": "rss",
+	    "config": {
+		"rss_path": "https://krebsonsecurity.com/feed/",
+		"limit": 1,
+		"skip": true
+	    }
+	}
     ]
 }

goosepaper/__main__.py

@@ -3,41 +3,52 @@
 
 from goosepaper.goosepaper import Goosepaper
 from goosepaper.util import construct_story_providers_from_config_dict
-from goosepaper.upload import upload
+from goosepaper.upload import do_upload
 from goosepaper.multiparser import MultiParser
 
 
 def main():
     multiparser = MultiParser()
     config = multiparser.config
 
-    story_providers = construct_story_providers_from_config_dict(config)
+    nostory = multiparser.argumentOrConfig("nostory")
 
-    title = config["title"] if "title" in config else None
-    subtitle = config["subtitle"] if "subtitle" in config else None
     filename = multiparser.argumentOrConfig(
         "output",
         default=f"Goosepaper-{datetime.datetime.now().strftime('%Y-%B-%d-%H-%M')}.pdf",
     )
-    replace = multiparser.argumentOrConfig("replace", False)
-    folder = multiparser.argumentOrConfig("folder", None)
-    font_size = multiparser.argumentOrConfig("font_size", 14)
-    print(font_size)
-
-    paper = Goosepaper(story_providers=story_providers, title=title, subtitle=subtitle)
-
-    if filename.endswith(".html"):
-        with open(filename, "w") as fh:
-            fh.write(paper.to_html())
-    elif filename.endswith(".pdf"):
-        paper.to_pdf(filename, font_size=font_size)
-    elif filename.endswith(".epub"):
-        paper.to_epub(filename, font_size=font_size)
-    else:
-        raise ValueError(f"Unknown file extension '{filename.split('.')[-1]}'.")
-
-    if multiparser.argumentOrConfig("upload", folder):
-        upload(filepath=filename, replace=replace, folder=folder)
+
+    if not nostory:  # global nostory flag
+        story_providers = construct_story_providers_from_config_dict(config)
+        font_size = multiparser.argumentOrConfig("font_size", 14)
+
+        title = config["title"] if "title" in config else None
+        subtitle = config["subtitle"] if "subtitle" in config else None
+
+        paper = Goosepaper(
+            story_providers=story_providers, title=title, subtitle=subtitle
+        )
+
+        if filename.endswith(".html"):
+            with open(filename, "w") as fh:
+                fh.write(paper.to_html())
+        elif filename.endswith(".pdf"):
+            paper.to_pdf(filename, font_size=font_size)
+        elif filename.endswith(".epub"):
+            paper.to_epub(filename, font_size=font_size)
+        else:
+            print(f"Unknown file extension '{filename.split('.')[-1]}'.")
+            exit(1)
+
+    if multiparser.argumentOrConfig("upload"):
+        if multiparser.argumentOrConfig("noupload"):
+            print(
+                "Honk! The 'upload' directive was found, but '--noupload' was also specified on the command line. Your goosepaper {0} was generated but I'm not uploading it.".format(
+                    filename
+                )
+            )
+        else:
+            do_upload(filepath=filename, multiparser=multiparser)
 
     return 0
 

goosepaper/auth.py

@@ -1,6 +1,7 @@
 from rmapy.api import Client
 from rmapy.exceptions import AuthError
 
+
 def auth_client():
     client = Client()
 

goosepaper/goosepaper.json

@@ -0,0 +1,12 @@
+{
+    "font-size": "14",
+    "replace": false,
+    "nocase": false,
+    "strictlysane": false,
+    "upload": false,
+    "output": null,
+    "folder": null,
+    "nostory": false,
+    "replace": false,
+    "cleanup": false
+}

goosepaper/goosepaper.py

@@ -22,7 +22,7 @@ def __init__(
         self.story_providers = story_providers
         self.title = title if title else "Daily Goosepaper"
         self.subtitle = subtitle + "\n" if subtitle else ""
-        self.subtitle += datetime.datetime.today().strftime("%B %d, %Y")
+        self.subtitle += datetime.datetime.today().strftime("%B %d, %Y %H:%M")
 
     def get_stories(self, deduplicate: bool = False):
         stories = []
@@ -109,14 +109,15 @@ def to_pdf(
     def to_epub(
         self, filename: str, style: Type[Style] = AutumnStyle, font_size: int = 14
     ) -> str:
+
         """
         Render the current Goosepaper to an epub file on disk
         """
         stories = []
+
         for prov in self.story_providers:
             new_stories = prov.get_stories()
             for a in new_stories:
-
                 if not a.headline:
                     stories.append(a)
                     continue

goosepaper/multiparser.py

@@ -1,7 +1,17 @@
 import argparse
+import os
+from os import getenv
 
 from goosepaper.util import load_config_file
 
+
+class NewLineFormatter(argparse.HelpFormatter):
+    def _split_lines(self, text, width):
+        if text.startswith("||"):
+            return text[2:].splitlines()
+        return argparse.HelpFormatter._split_lines(self, text, width)
+
+
 class MultiParser:
     def __init__(self):
         """
@@ -12,13 +22,15 @@ def __init__(self):
 
         self.parser = argparse.ArgumentParser(
             prog="goosepaper",
-            description="Goosepaper generates and delivers a daily newspaper in PDF format."
+            description="Goosepaper generates and delivers a daily newspaper in PDF format.",
+            formatter_class=NewLineFormatter,
         )
+
         self.parser.add_argument(
             "-c",
             "--config",
             required=False,
-            default=None,
+            default="",
             help="The json file to use to generate this paper.",
         )
         self.parser.add_argument(
@@ -27,35 +39,150 @@ def __init__(self):
             required=False,
             help="The output file path at which to save the paper",
         )
+        self.parser.add_argument(
+            "-f",
+            "--folder",
+            required=False,
+            help="Folder to which the document will be uploaded in your remarkable.",
+        )
         self.parser.add_argument(
             "-u",
             "--upload",
             action="store_true",
             required=False,
+            default=None,
             help="Whether to upload the file to your remarkable using rmapy.",
         )
+        self.parser.add_argument(
+            "--noupload",
+            action="store_true",
+            required=False,
+            default=None,
+            help="Overrides any other 'upload: true' arguments from config files or command line. Useful for testing configs or story generation without having to edit config files.",
+        )
+        self.parser.add_argument(
+            "--showconfig",
+            action="store_true",
+            required=False,
+            default=None,
+            help="Print out all config files and command line options in order loaded and the final config to help finding conflicting options. Needed since there are now four possible ways to pass options.",
+        )
+        self.parser.add_argument(
+            "-n",
+            "--nostory",
+            required=False,
+            default=False,
+            action="store_true",
+            help='||Skip story creation. Combined with "--upload" can be used to\nupload a preexisting output file.\n\n** NOTE ** If used without "--upload" goosepaper will run but\nperform no action.',
+        )
         self.parser.add_argument(
             "--replace",
             action="store_true",
             required=False,
             default=None,
-            help="Will replace a document with same name in your remarkable.",
+            help="||Will replace a document with same name in your remarkable.\nDefault behaviour is case sensitive (ala *nix/Mac).\n\ne.g. 'A Flock of RSS Feeds.epub' and 'a flock of rss feeds.epub'\nare seen as TWO different files. Can be altered with '--nocase'\nor '--strictlysane' switches.",
         )
         self.parser.add_argument(
-            "-f",
-            "--folder",
+            "--noreplace",
+            action="store_true",
             required=False,
-            help="Folder to which the document will be uploaded in your remarkable.",
+            default=None,
+            help="Only valid when specified on command line (ignored if present in any config file). Supersedes any config file setting for 'replace: true', thus ensuring that the file will NEVER be overwritten. Will also supersede command line '--replace' if both are specified regardless of order.",
+        )
+        self.parser.add_argument(
+            "--cleanup",
+            required=False,
+            default=None,
+            action="store_true",
+            help="Delete the output file after upload.",
         )
         self.args = self.parser.parse_args()
 
-        try:
-            self.config = load_config_file(self.args.config)
-        except FileNotFoundError as e:
-            raise FileNotFoundError(
-                f"Could not find the configuration file at {self.args.config}"
-            ) from e
+        # These are in order of precedence, low to highest
+        #  1. Home directory global configs
+        #  2. Local directory from which goosepaper is called
+        #  3. Specified on the command line.
+
+        defaultconfigs = [
+            os.getenv("HOME") + "/.goosepaper.json",
+            "./goosepaper.json",
+            self.args.config,
+        ]
+        foundconfig = False
+        self.config = {}
+        outputcount = 0
+        debug_configs = True if self.args.showconfig else None
+
+        # Debug code for troubleshooting config file and cli override issues.
+        if debug_configs:
+            import pprint
 
+            pp = pprint.PrettyPrinter(indent=3)
+            print(
+                "Command line arguments received:\n(including default values)\n--------------------------------"
+            )
+            pp.pprint(self.args)
+
+        # If passed a config file on the command line, assume it's important so fail
+        # if not readable.
+
+        if self.args.config:
+            try:
+                valid = load_config_file(self.args.config)
+            except FileNotFoundError:
+                print(
+                    "Honk! Honk! Somebody stole my egg! Couldn't find config file ({0}) specified on the command line. Aborting migration.".format(
+                        self.args.config
+                    )
+                )
+                exit(1)
+
+        for defconfigfile in defaultconfigs:
+            try:
+                tempconfig = load_config_file(defconfigfile)
+                if "output" in tempconfig and "output" in self.config:
+                    outputcount = outputcount + 1
+                if "stories" in tempconfig and "stories" in self.config:
+                    for story in self.config["stories"]:
+                        tempconfig["stories"].append(story)
+
+                self.config.update(tempconfig)
+                foundconfig = True
+                if debug_configs:
+                    print(
+                        f"\nConfig options found in {defconfigfile}:\n---------------------\n"
+                    )
+                    pp.pprint(load_config_file(defconfigfile))
+            except FileNotFoundError as e:
+                pass
+
+        if debug_configs:
+            print("\nFinal config values are:\n------------------")
+            pp.pprint(self.config)
+            print("")
+
+        if foundconfig is not True:
+            print(
+                "Honk! Honk! Unable to locate a config file. You must have at least one of the following! '$HOME/.goosepaper.json', './goosepaper.json', or a config file specified on the command line."
+            )
+            exit(1)
+
+        # Not sure if you should able to override the output destination or not.
+
+        # if outputcount > 0:
+        #    print ("Honk! Honk! You've specified more than one output destination in your config files. A flying flock can only have one lead goose in a skein. Please resolve this.")
+        #    exit(1)
+        # if 'output' in self.config and self.args.output:
+        #    print ("Honk! Honk! You have both config file and command line output file options. I don't know which flock with which to fly with so I'm not flying anywhere. Please fix.")
+        #    exit(1)
+
+        # --noreplace only makes sense to me on the command line, so if present in a config file ignore it.
+        # Same with --noupload.
+
+        if "noreplace" in self.config:
+            del self.config["noreplace"]
+        if "noupload" in self.config:
+            del self.config["upload"]
 
     def argumentOrConfig(self, key, default=None, dependency=None):
         """
@@ -86,5 +213,3 @@ def argumentOrConfig(self, key, default=None, dependency=None):
             value = default
 
         return value
-
-

goosepaper/rss.py

@@ -40,4 +40,3 @@ def parallelizable_request(self, entry):
         story = Story(doc.title(), body_html=doc.summary(), byline=source)
 
         return story
-