pypyr.steps.pathcheck & deprecate multi-input fetches

README.rst

@@ -555,9 +555,9 @@ Built-in steps
 +-------------------------------+-------------------------------------------------+------------------------------+
 | `pypyr.steps.envget`_         | Get $ENVs and use a default if they don't exist.| envget (list)                |
 +-------------------------------+-------------------------------------------------+------------------------------+
-| `pypyr.steps.fetchjson`_      | Loads json file into pypyr context.             | fetchJsonPath (path-like)    |
+| `pypyr.steps.fetchjson`_      | Loads json file into pypyr context.             | fetchJson (dict)             |
 +-------------------------------+-------------------------------------------------+------------------------------+
-| `pypyr.steps.fetchyaml`_      | Loads yaml file into pypyr context.             | fetchYamlPath (path-like)    |
+| `pypyr.steps.fetchyaml`_      | Loads yaml file into pypyr context.             | fetchYaml (dict)             |
 +-------------------------------+-------------------------------------------------+------------------------------+
 | `pypyr.steps.fileformat`_     | Parse file and substitute {tokens} from         | fileFormat (dict)            |
 |                               | context.                                        |                              |
@@ -574,6 +574,8 @@ Built-in steps
 +-------------------------------+-------------------------------------------------+------------------------------+
 | `pypyr.steps.filewriteyaml`_  | Write payload to file in yaml format.           | fileWriteYaml (dict)         |
 +-------------------------------+-------------------------------------------------+------------------------------+
+| `pypyr.steps.pathcheck`_      | Check if path exists on filesystem.             | pathCheck (string or dict)   |
++-------------------------------+-------------------------------------------------+------------------------------+
 | `pypyr.steps.py`_             | Executes the context value `pycode` as python   | pycode (string)              |
 |                               | code.                                           |                              |
 +-------------------------------+-------------------------------------------------+------------------------------+
@@ -1306,14 +1308,19 @@ Loads a json file into the pypyr context.
 
 This step requires the following key in the pypyr context to succeed:
 
-- fetchJsonPath
+.. code-block:: yaml
+
+  fetchJson:
+    path: ./path.json # required. path to file on disk. can be relative.
+    key: 'destinationKey' # optional. write json to this context key.
 
-  - path-like. Path to file on disk. Can be relative.
+If ``key`` is not specified, json writes directly to context root.
 
-- fetchJsonKey
+If you do not want to specify a key, you can also use the streamlined format:
+
+.. code-block:: yaml
 
-  - Optional. Write json to this context key. If not specified, json writes
-    directly to context root.
+  fetchJson: ./path.json # required. path to file on disk. can be relative.
 
 All inputs support `Substitutions`_.
 
@@ -1323,7 +1330,7 @@ overwrite existing values if the same keys are already in there.
 I.e if file json has ``{'eggs' : 'boiled'}``, but context ``{'eggs': 'fried'}``
 already exists, returned ``context['eggs']`` will be 'boiled'.
 
-If *fetchJsonKey* is not specified, the json should not be an array [] at the
+If ``key`` is not specified, the json should not be an array [] at the
 root level, but rather an Object {}.
 
 pypyr.steps.fetchyaml
@@ -1332,14 +1339,19 @@ Loads a yaml file into the pypyr context.
 
 This step requires the following key in the pypyr context to succeed:
 
-- fetchYamlPath
+.. code-block:: yaml
+
+  fetchYaml:
+    path: ./path.yaml # required. path to file on disk. can be relative.
+    key: 'destinationKey' # optional. write yaml to this context key.
 
-  - path-like. Path to file on disk. Can be relative.
+If ``key`` not specified, yaml writes directly to context root.
 
-- fetchYamlKey
+If you do not want to specify a key, you can also use the streamlined format:
 
-  - Optional. Write yaml to this context key. If not specified, yaml writes
-    directly to context root.
+.. code-block:: yaml
+
+  fetchYaml: ./path.yaml # required. path to file on disk. can be relative.
 
 All inputs support `Substitutions`_.
 
@@ -1355,7 +1367,7 @@ I.e if file yaml has
 but context ``{'eggs': 'fried'}`` already exists, returned ``context['eggs']``
 will be 'boiled'.
 
-If *fetchYamlKey* is not specified, the yaml should not be a list at the top
+If ``key`` is not specified, the yaml should not be a list at the top
 level, but rather a mapping.
 
 So the top-level yaml should not look like this:
@@ -1754,6 +1766,76 @@ the last line will substitute like this:
 See a worked `filewriteyaml example here
 <https://github.com/pypyr/pypyr-example/tree/master/pipelines/filewriteyaml.yaml>`_.
 
+pypyr.steps.pathcheck
+^^^^^^^^^^^^^^^^^^^^^
+Check if a path exists on the filesystem. Supports globbing. A path can point
+to a file or a directory.
+
+The ``pathCheck`` context key must exist.
+
+.. code-block:: yaml
+
+  - name: pypyr.steps.pathcheck
+    in:
+      pathCheck: ./**/*.py # single path with glob
+
+If you want to check for the existence of multiple paths, you can pass a list
+instead. You can freely mix literal paths and globs.
+
+.. code-block:: yaml
+
+  - name: pypyr.steps.pathcheck
+    in:
+      pathCheck:
+        - ./file1 # literal relative path
+        - ./dirname # also finds dirs
+        - ./**/{arbkey}* # glob with a string formatting expression
+
+After *pathcheck* completes, the ``pathCheckOut`` context key is available.
+This contains the results of the *pathcheck* operation.
+
+.. code-block:: yaml
+
+  pathCheckOut:
+      # the key is the ORIGINAL input, no string formatting applied.
+      'inpath-is-the-key': # one of these for each pathCheck input
+          exists: true # bool. True if path exists.
+          count: 0 # int. Number of files found for in path.
+          found: ['path1', 'path2'] # list of strings. Paths of files found.
+
+Example of passing a single input and the expected output context:
+
+.. code-block:: yaml
+
+  pathCheck: ./myfile # assuming ./myfile exists in $PWD
+  pathCheckOut:
+    './myfile':
+      exists: true,
+      count: 1,
+      found:
+        - './myfile'
+
+The ``exists`` and ``count`` keys can be very useful for conditional
+decorators to help decide whether to run subsequent steps. You can use these
+directly in string formatting expressions without any extra fuss.
+
+.. code-block:: yaml
+
+  - name: pypyr.steps.pathcheck
+    in:
+      pathCheck: ./**/*.arb
+  - name: pypyr.steps.echo
+    run: '{pathCheckOut[./**/*.arb][exists]}'
+    in:
+      echoMe: you'll only see me if ./**/*.arb found something on filesystem.
+
+All inputs support `Substitutions`_. This means you can specify another context
+item to be an individual path, or part of a path, or the entire path list.
+
+See a worked
+example for `pathcheck here
+<https://github.com/pypyr/pypyr-example/tree/master/pipelines/pathcheck.yaml>`_.
+
 pypyr.steps.py
 ^^^^^^^^^^^^^^
 Executes the context value `pycode` as python code.

pypyr/steps/envget.py

@@ -50,6 +50,8 @@ def run_step(context):
     else:
         get_items = [context['envGet']]
 
+    get_count = 0
+
     for get_me in get_items:
         (env, key, has_default, default) = get_args(get_me)
 
@@ -59,17 +61,21 @@ def run_step(context):
 
         if formatted_env in os.environ:
             context[formatted_key] = os.environ[formatted_env]
+            get_count += 1
         else:
             logger.debug(f"$ENV {env} not found.")
             if has_default:
                 logger.debug(f"Using default value for {env} instead.")
                 formatted_default = context.get_formatted_iterable(default)
                 context[formatted_key] = os.environ.get(formatted_env,
                                                         formatted_default)
+                get_count += 1
             else:
                 logger.debug(
                     f"No default value for {env} found. Doin nuthin'.")
 
+    logger.info(f"saved {get_count} $ENVs to context.")
+
 
 def get_args(get_item):
     """Parse env, key, default out of input dict.

pypyr/steps/fetchjson.py

@@ -20,9 +20,13 @@ def run_step(context):
     Args:
         context: pypyr.context.Context. Mandatory.
                  The following context key must exist
-                - fetchJsonPath. path-like. Path to file on disk.
-                - fetchJsonKey. string. If exists, write json structure to this
-                                context key. Else json writes to context root.
+                - fetchJson
+                    - path. path-like. Path to file on disk.
+                    - key. string. If exists, write json structure to this
+                      context key. Else json writes to context root.
+
+    Also supports a passing path as string to fetchJson, but in this case you
+    won't be able to specify a key.
 
     All inputs support formatting expressions.
 
@@ -31,17 +35,28 @@ def run_step(context):
 
     Raises:
         FileNotFoundError: take a guess
-        pypyr.errors.KeyNotInContextError: fetchJsonPath missing in context.
-        pypyr.errors.KeyInContextHasNoValueError: fetchJsonPath exists but is
+        pypyr.errors.KeyNotInContextError: fetchJson.path missing in context.
+        pypyr.errors.KeyInContextHasNoValueError: fetchJson.path exists but is
                                                   None.
 
     """
     logger.debug("started")
-    context.assert_key_has_value(key='fetchJsonPath', caller=__name__)
 
-    file_path = context.get_formatted('fetchJsonPath')
+    deprecated(context)
+
+    context.assert_key_has_value(key='fetchJson', caller=__name__)
 
-    destination_key_expression = context.get('fetchJsonKey', None)
+    fetch_json_input = context.get_formatted('fetchJson')
+
+    if isinstance(fetch_json_input, str):
+        file_path = fetch_json_input
+        destination_key_expression = None
+    else:
+        context.assert_child_key_has_value(parent='fetchJson',
+                                           child='path',
+                                           caller=__name__)
+        file_path = fetch_json_input['path']
+        destination_key_expression = fetch_json_input.get('key', None)
 
     logger.debug(f"attempting to open file: {file_path}")
     with open(file_path) as json_file:
@@ -65,3 +80,21 @@ def run_step(context):
 
     logger.info(f"json file written into pypyr context. Count: {len(payload)}")
     logger.debug("done")
+
+
+def deprecated(context):
+    """Create new style in params from deprecated."""
+    if 'fetchJsonPath' in context:
+        context.assert_key_has_value(key='fetchJsonPath', caller=__name__)
+
+        context['fetchJson'] = {'path': context['fetchJsonPath']}
+
+        if 'fetchJsonKey' in context:
+            context['fetchJson']['key'] = context.get('fetchJsonKey', None)
+
+        logger.warning("fetchJsonPath and fetchJsonKey "
+                       "are deprecated. They will stop working upon the next "
+                       "major release. Use the new context key fetchJson "
+                       "instead. It's a lot better, promise! For the moment "
+                       "pypyr is creating the new fetchJson key for you "
+                       "under the hood.")

pypyr/steps/fetchyaml.py

@@ -18,12 +18,16 @@ def run_step(context):
     Args:
         context: pypyr.context.Context. Mandatory.
                  The following context key must exist
-                - fetchYamlPath. path-like. Path to file on disk.
-                - fetchYamlKey. string. If exists, write yaml to this context
-                                key. Else yaml writes to context root.
+                - fetchYaml
+                    - path. path-like. Path to file on disk.
+                    - key. string. If exists, write yaml to this context key.
+                      Else yaml writes to context root.
 
     All inputs support formatting expressions.
 
+    Also supports a passing path as string to fetchYaml, but in this case you
+    won't be able to specify a key.
+
     Returns:
         None. updates context arg.
 
@@ -35,11 +39,22 @@ def run_step(context):
 
     """
     logger.debug("started")
-    context.assert_key_has_value(key='fetchYamlPath', caller=__name__)
 
-    destination_key_expression = context.get('fetchYamlKey', None)
+    deprecated(context)
+
+    context.assert_key_has_value(key='fetchYaml', caller=__name__)
 
-    file_path = context.get_formatted('fetchYamlPath')
+    fetch_yaml_input = context.get_formatted('fetchYaml')
+
+    if isinstance(fetch_yaml_input, str):
+        file_path = fetch_yaml_input
+        destination_key_expression = None
+    else:
+        context.assert_child_key_has_value(parent='fetchYaml',
+                                           child='path',
+                                           caller=__name__)
+        file_path = fetch_yaml_input['path']
+        destination_key_expression = fetch_yaml_input.get('key', None)
 
     logger.debug(f"attempting to open file: {file_path}")
     with open(file_path) as yaml_file:
@@ -64,3 +79,21 @@ def run_step(context):
 
     logger.info(f"yaml file written into pypyr context. Count: {len(payload)}")
     logger.debug("done")
+
+
+def deprecated(context):
+    """Create new style in params from deprecated."""
+    if 'fetchYamlPath' in context:
+        context.assert_key_has_value(key='fetchYamlPath', caller=__name__)
+
+        context['fetchYaml'] = {'path': context['fetchYamlPath']}
+
+        if 'fetchYamlKey' in context:
+            context['fetchYaml']['key'] = context.get('fetchYamlKey', None)
+
+        logger.warning("fetchYamlPath and fetchYamlKey "
+                       "are deprecated. They will stop working upon the next "
+                       "major release. Use the new context key fetchYaml "
+                       "instead. It's a lot better, promise! For the moment "
+                       "pypyr is creating the new fetchYaml key for you "
+                       "under the hood.")