DO NOT MERGE: Experimental rewrite of ops to invoke

[vlcinsky]

To learn pypyr, I rewrote existing ops/* content to invoke as I am using invoke often and feel quite familiar with it.

This pull requests is not supposed to be accepted, I put it here to notify about possible comparison between pypyr and invoke implementation only.

For quick review, read the ops/README.rst file.

All stuff from ops/*.yaml is implemented in tasks.py and ops/*.py files.

GitHub actions are not modified and I am not planning to change that.

Comparing amount of code:

• pypyr ops/*.yml files: 259 lines
• tasks.py + ops/*.py + invoke.yml: 399 lines.

If I would try to have less lines, it seems feasible to get the invoke part down to 300 lines
This makes pypyr 259 lines rewritten to invoke 300 lines what seems a bit more, but still pretty close.

Lessons learned

• pypyr turns to be a bit more succinct comparing to invoke code.
• some people might prefer coding in python as this does not require to learn any new syntax
• invoke solution allows more granular set of actions to call and is quite self-explanatory
• learning invoke in depth also requires some time
• invoke required some customization to report called tasks and steps and their results in more readable way
• the most daunting task is to design actual actions to do for build, tests and other actions and this applies to both pypyr and invoke implementation.
• pypyr surprised me by wide range of constructs usable for given purpose. First impression was it is not so easy to read but after this exercise I like the pypyr syntax more

[vlcinsky]

Sorry for the messy "bump version 4.5.x -> 4.5.x+1` commits - testing CI/CD tool might have side effects like this.

[yaythomas]

This is AMAZING work @vlcinsky! Wow! 🥇 🥇 🥇

Thank you so much! You did a lot and you wrote a fair & balanced assessment and I think it's super useful for anyone looking to learn more of pypyr, pyinvoke or just general ci/cd of python projects.

I've not particularly optimised the pypyr ops pipelines for number of lines. . . and you've probably noticed I'm a bit wordy with my comments/descriptions which also take up more space in the pipelines. But I'm not particularly interested in starting a over-the-top minimisation competition - what's better is what you did here, which is a real-world comparison of code-in-practice, as it exists in the wild for actual useful tasks, without using fancy tricks to get the line-count down.

I would be very interested to hear if you have some thoughts on streamlining & possible improvements to pypyr, whether in the core or in pipelines, now that you've been through the process? Feel free on the Discussions board https://github.com/pypyr/pypyr/discussions

I don't want your valuable work to be lost in a PR here that'll end up closed & forgotten. . . How about I create a repo for you in the pypyr organisation where you can put this work and all the great documentation/notes you've written? I've made the repo here https://github.com/pypyr/pyinvoke-comparison-example and I made you a maintainer, so you've got the power to do what you want :-) I've left a little toy/hello-world python package in there for the example rather than the full pypyr build, so for the sake of testing/building you don't have to wait for a big project with >1,000 tests to run 1st :-)

By the way, I have a very similar test repo here https://github.com/pypyr/python-ci-cd-example that I use to test a bare-bones python project with fully automated ci/cd implemented in pypyr, including GitHub actions. It's very similar to what you see here in the pypyr repo, but it's a sandpit where I can be more free to make experimental messes that I don't want to make in the main production pypyr repo. I am at fault for not advertizing/documenting this experimental repo more widely, though - I didn't write anything up as thoroughly as you have done here!

P.S Regarding a note you made:

Challenges

Can we minimize installing dependencies of these tools into actual python
virtual env used for the package?

From the pypyr side, at least, you can do the following to create a virtualenv with all the necessary build/test dependencies in "one step": https://pypyr.io/docs/contributing/developers-guide/#if-tox-takes-too-long

Alternatively you can just use tox, but for day-to-day testing that takes more time to run, so while I'm still furiously hacking & slashing I tend to run tests more selectively in the venv just for quicker turnaround :-)

[vlcinsky]

@yaythomas I put the code into https://github.com/pypyr/pyinvoke-comparison-example repository.

[yaythomas]

Much thanks, looks great! 🙌 🙌 🙌

[yaythomas]

new example repo renamed to https://github.com/pypyr/pypyr-to-invoke-comparison-example