toml & text+binary file read/write

Finally, toml, has come, to pypyr! 🎆

• pypyr.parser.tomlfile to initialise pipeline context with a toml file
• pypyr.steps.fetchtoml to read toml file into context as an object with typing
• pypyr.steps.filewritetoml to write specified context to output toml file - lets you create toml files on-the-fly programmatically.
• pypyr.steps.fileformattoml to read input tomls specified by path, glob or list of paths, replace {formatting expressions} and write to output file(s).

Working with files:

• pypyr.steps.fileread to read files in text or binary mode into context.
• pypyr.steps.filewrite to write or append to file in text or binary modes.

What's Changed

• pypyr.steps.fileread by @yaythomas in #244
• pypyr.steps.filewrite by @yaythomas in #245
• toml by @yaythomas in #246

Full Changelog: v5.0.0...v5.1.0

Relative pipelines + API breaking changes.

Implement adr2 relative pipelines + api changes.

In brief, this release lets pipelines reference custom modules & child pipelines relative to the pipeline itself, rather than the current directory. This lets you create portable, re-usable & composable pipeline libraries.

Breaking Changes

This is a major version increment because it comes with BREAKING CHANGES:

1. API: pipelinerunner.run() replaces both pipelinerunner.main() and pipelinerunner.main_with_context()
2. API: def get_pipeline_definition(pipeline_name, working_directory) signature for custom pype loaders changes to def get_pipeline_definition(pipeline_name, parent)
3. CLI: the —dir flag now only sets the directory for ad hoc custom Python modules, it does NOT also set the directory for pipelines anymore
4. Final removal of deprecated get_formatted_iterable, get_formatted_string #195 & pypyr.steps.contextset #184. Where previously these would just give deprecation warnings, they are now completely removed.
5. pypyr.pypeloaders.fileloader renamed pypyr.loaders.file

Non-Breaking Changes

• You can now access the current pipeline’s metadata & loader information from within a pipeline with context.current_pipeline
• Improve handling of absolute paths in file loader only to search path once, rather than unnecessarily go through the same relative path lookup sequence with the same path.
• Typing support added for the pypyr API entrypoint.

Detailed Technical Breakdown:

• Introduce new classes to model pipeline payload, rather than just using the bare dict-like yaml directly.
  • PipelineInfo - pipeline metadata set by loader. This maintains a pipeline’s parent/path info so that child pipelines can load relative to the parent.
  • PipelineDefinition - this wraps the pipeline payload and its metadata (PipelineInfo) to allow pypyr to cache it all with one reference
• Add new Pipeline class for the run-time properties of a single run.
  • The Pipeline references the shared cached PipelineDefinition.
  • Move run + load_and_run logic from pypyr.pipelinerunner to the new Pipeline class. This massively streamlines the pipeline invocation process, since run/load_and_run can just operate on the shared Pipeline state rather than sling a bunch of args between different functions as before.
• Add a call-stack of running Pipeline instances on Context. i.e Parent -> child1 -> child2 where the root pipeline calls child pipelines via pype
  • Add current_pipeline attribute to Context, controlled with a context manager to scope itself to an individual pipeline run’s lifespan.
    • This means that steps can access current pipeline’s properties.
    • This allows pypyr.steps.pype to find the current (i.e parent) pipeline’s metadata such as path, to load child pipeline relative to the calling pipeline’s location.
    • When a child pipeline completes, the calling pipeline (i.e the previous Pipeline in the call-stack) becomes the current pipeline
• Amend pypyr.steps.pype to instantiate Pipeline object to load_and_run() child.
  • pype now deals the context.current_pipeline.pipeline_definition.info metadata to work out whether to cascade parent path down to child, so child can load relative to the parent.
  • Notably, the parent loader now cascades to the child, so pipeline authors don’t need explicitly to set the same custom loader repeatedly for each child.
  • Given the context manager controlling current pipeline scope in Pipeline.load_and_run_pipeline remove the clumsy side-shuffle for pipeline_name, working_dir to swap out these values as child pipe runs and swap these back when it completes/errors.
• Remove global PipelineCache. Replace with distinct pipeline cache per loader.
  • This resolves a long standing limitation where pypyr assumed unique pipeline names across all loaders.
  • The per-loader pipeline cache stores PipelineDefinition objects.
  • Introduce Loader class, which wraps loader & its pipeline cache. Loader is what the LoaderCache caches.
  • Thus LoaderCache -> Loader -> _pipelineCache -> PipelineDefinition
• File loader has a private file cache keyed on absolute path of file
  • This is to prevent >1 load+parse where the same underlying pipeline.yaml file has different names in the loader’s pipeline cache
    • e.g (name=‘dir/mypipe’, parent=None) and (name=‘mypipe’, parent=‘dir’) both resolve to dir/mypipe.yaml
  • Caching a reference to the PipelineDefinition object, so not duplicating memory
• Improve handling of absolute paths in file loader only to search path X1, rather than unecessarilly go through the same relative path lookup sequence with the same path.
• File loader get_pipeline_definition now returns a PipelineDefinition with PipelineFileInfo to store file-system specific metadata for the loader pipeline
• Remove working_dir global. The py_dir input on run() now refers ONLY to module paths, NOT pipeline locations.
  • The CLI —dir flag, or py_dir input on run() basically adds the specified directory to sys.path.
• Add current pipeline’s parent directory to sys.path on load. This allows child pipelines to resolve custom modules relative to itself.
• pypyr.dsl.Step does not need StepsRunner anymore, because it can get it from the context.current_pipeline instead.
• Recode (some) integration tests to take advantage of list pypyr.steps.append step and checking that for output on return context rather than intercepting logger.NOTIFY.
• Rename master branch to main in CI/CD GitHub actions
• Add typing annotations to the public run() function and the Pipeline class public accessors. The idea is NOT to type pypyr exhaustively, just to provide annotations for the sensible/likely entrypoint to enhance API user experience. Include py.typed in pypyr package.
• Remedy packaging snafu where tests.common was deploying alongside pypyr because exclude condition in find_packages didn't include wildcard for sub packages.

What's Changed

• Relative pipelines & API run() replaces main/main_with_context by @yaythomas in #243

Full Changelog: v4.6.0...v5.0.0

set, add, append.

What's Changed

• Fix failing ops/build (failing on linting issues and locale dependent failure) by @vlcinsky in #226
• Evaluate step skip only if the run check returned true by @Reskov in #234
• pypyr.steps.append & pypyr.steps.add by @yaythomas in #235
• shorter alias set for contextsetf. py 3.10. by @yaythomas in #238

New Contributors

• @vlcinsky made their first contribution in #226

Full Changelog: v4.5.0...v4.6.0

retry backoff & cleaner api init

• add retry backoff strategies. #216.
• prevent duplicates in sys.path (#218) on main* api entry points. thread-safe existence check on sys.path list.
• prevent redundant multiple addition of notify log-level on main* api entry. Move initialization code to package __init__ (#219).

context parser initialise to empty rather than None

• Context parsers that create an entry in context now initialise to empty rather than None. This means you can directly use something like {argList}, {argDict} and argString (initialising respectively to [], {}, '') directly for things like foreach loops without having to worry about None checks. Your existing truthy checks for these values will work as before. closes #214.

simplified py step syntax & imports for !py strings.

• New step pypyr.steps.pyimport to import references to the py-string namespace.
  • This includes an underlying api signature change by removal of pypyr.utils.expressions.eval_string(), but this is sufficiently far down the call-chain that it shouldn’t affect any normal pipeline operator or api consumer.
  • See #110 for details.
• pypyr.steps.contextclearall wipes pyimport imported references in addition to the key/values inside context.
• Simplify pypyr.steps.py syntax by allowing a new py (rather than pycode) input. This allows pipeline authors to use context key names directly as variables, rather than have to specify them as keys in context (my_var vs context[‘my_var’]).
  • see #204 for details on simplifying the pypyr.steps.py syntax.
  • the old pycode will keep on working in the same way, so no need to worry about backwards compatibility for your existing pipelines.
• Allow substitutions on Retry max. Resolves #207. Excellent bug find & fix by @Reskov, much thanks 🙌 🙌 🙌 as ever for a superb contribution! 🔥 🔥 🔥
• foreach can now use any iterable, including generators. Closes #209

streamlined api main() entry-point

• Streamline main entrypoint API. close #201.
  • main() allows consumer to set pype loader, rather than having to drop further down into api to load_and_run_pipeline()
  • new main_with_context() to input dict to initialise context, and bypass context_parser entirely. Also returns the Context object after pipeline run completes.
  • make all non-essential args optional to allow minimal calls to main entrypoint without having to add optional=None style inputs.
  • This is fully backwards compatible.
• pypyr.steps.pype
  • defaults useParentContext to False is pipeArgs specified.
  • pipeArgs shlex-es input string
  • set pipeline_name on child pipeline rather than use parent pipeline name
• working_dir uses Path object rather than string
• pypyr.steps.echo remove redundant string check.

flat & recursive format, !jsonify, parsejson.

• python 3.9 compatibility officially confirmed. All good. Woo!
• New Flat ff and Recursive rf formatting specifiers. #195
  • Maintain backwards compatibility for all current formatting functionality.
  • The get_formatted_string, get_formatted_iterable & get_processed_string methods on Context() are now all deprecated. Use get_formatted_value instead. All of the deprecated functions will keep on working as before, but will print a WARN to the output.
  • get_formatted and get_formatted_value remain the official best ways of getting formatted values.
• New !jsonify yaml special tag directive. This converts context object to a json string. #197.
• New pypyr.steps.jsonparse step. Parse and deserialise a json string into Context. #199
• Bare pypyr.steps.assert syntax. #196
• Add new utils.asserts for extra flexibility with nested key validation.
• PyStrings optimised. Not so that you'd notice, but still, it'll be a smidge faster.
• Friendlier error messages when yaml/json context parsers reject documents that do NOT have a mapping at root.

step description (skipping) when skipping.

• Better description output - add (skipping) to output if the step is not running because run is False or skip is True. Ref #158.
• First release published from shiny new GitHub Action CI/CD!
• Add License to wheel published to pypi

Stop in failure handlers. in context cleanup default.

• BREAKING CHANGE: final deprecation of in args persisting after step execution. For full discussion please see #177. Preview functionality where setting environment variable $PYPYR_IN_CLEAN = 1 is now the default. You can remove the $PYPYR_IN_CLEAN variable from this release onwards. Henceforth pypyr removes in args from context after step completes. If you want to persist values in context, use pypyr.steps.contextsetf, pypyr.steps.contextcopy or pypyr.steps.default.
• BREAKING CHANGE: pypyr.steps.assert raises an AssertionError rather than a ContextError. Ref #188.
• The cli now defaults to a simplified log format that is way less noisy. If you still want the full, old style logs, run pypyr with --log 25 set explicitly. pypyr my pipe --log 25. Ref #187
  • description decorator output also simplified to echo only your actual custom text.
• Failure Groups now support Stop instructions. This allows you to exit the failure handler reporting success, so pypyr can report success if you consider the error condition handled. Ref #175.
• pypyr.steps.contextset renamed pypyr.steps.contextcopy. The old name will keep on working, but with a deprecation warning. Ref #184.
• internal: improvements to integration testing error reporting.