Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tasks #591

Closed
wants to merge 5 commits into from

Conversation

@michaeldel
Copy link

commented Nov 2, 2018

Task

I find it pretty useful for Poetry to be able to run tasks in development, similarly to NPM's scripts (that can be launched with npm run).These tasks will not be available in end-user package installation.

They are defined in a tool.poetry.tasks section, pretty similar to tool.poetry.scripts:

[tool.poetry.tasks]
foo = "echo 'bar'"

And may be run like so

poetry task foo

They are automatically run within the project's virtualenv (the same way poetry run runs scripts).

This new feature should also address some concerns expressed here in #241. make is already suited to take care of tasks running but:

  • it is not available on Windows by default (and might sometimes be painful to install and work with, especially in CI environments)
  • it requires an additional file whereas this tool.poetry.tasks feature will be consistent with the new pyproject.toml project files.
  • make still has to be called through poetry run in order to be integrated within the current virtualenv, hence missing the "shorter command" initial benefit

Current tool.poetry.scripts are currently not that well suited for such operations as well, indeed:

Use cases

Here are some common use cases where such feature may come in handy.

Lint and test code

[tool.poetry.tasks]
lint = "black --check ."
test = "pytest tests/"

Run development or production server

[tool.poetry.tasks]
dev = "FLASK_APP=app.py flask run"
start = "gunicorn app.wsgi --log-file -"

Clean environment

[tool.poetry.tasks]
clean = "poetry run manage.py flush && rm -rf media"

Improvements

I have first chosen tasks as name in order not to mismatch tasks with scripts (which can be run from an end-user perspective). It may feel a bit of a duplication among some so these might be renamed to "development scripts" (tool.poetry.dev-scripts) in order to be consistent (also poetry run probably remains a better command name than poetry task for that sort of thing).

It does also currently not support passing arguments to tasks, maybe adding an option for this might be an interesting idea (similarly to npm run task -- args).

Some other options might also be considered such as --silent or --pwd.

Current implementation is of course very rudimentary and will probably be further developed later if merged.

Checklist

  • Added tests for changed code.
  • Updated documentation for changed code.

@michaeldel michaeldel force-pushed the michaeldel:tasks branch from c35ff74 to 5452423 Nov 2, 2018

@jacebrowning

This comment has been minimized.

Copy link
Contributor

commented Nov 2, 2018

Placing these in [tool.poetry.dev-scripts] to be run as $ poetry run <name> seems preferable to me because:

  • It mirrors the syntax of the existing [tool.poetry.scripts]
  • One does not have to know which section the script is in to run it
@auvipy

auvipy approved these changes Nov 3, 2018

@ZeeD26

This comment has been minimized.

Copy link

commented Nov 7, 2018

@jacebrowning Personally, I'd rather have $ poetry run <shell command> reserved for shell commands... Someone like me might have the bright idea to name a "task" test and then I can not access the builtin shell command anymore 😁

@ZeeD26

This comment has been minimized.

Copy link

commented Nov 7, 2018

On another note, if I understood it correctly this is also some direction the @tox-dev team wants to go, i.e., general automation. Is it really necessary for poetry to also include this functionality, then?

@michaeldel

This comment has been minimized.

Copy link
Author

commented Nov 9, 2018

@ZeeD26 even though @tox-dev is working on similar features does not mean we should not implement it for Poetry, indeed not everyone uses Tox and adding it ot a project for the sake of tasks sounds overkill to me.
I also think that it makes much more sense to integrate a task system to a tool like Poetry that I find more generalist when it comes to "project management" (kindof).

@jacebrowning I would also prefer such syntax but I am very worried about the aliasing it could involve, maybe it would not be that bad though (@ZeeD26 you may still run the test command without poetry run since there is no dependency from the virtualenv 😄, but I understand your concern)

One other "issue" with [tool.poetry.dev-scripts] would be the need to remain consistent with [tool.poetry.scripts], that is setting a new specific syntax for "shell tasks" in order not to conflict with the current syntax, something along the lines of:

[tool.poetry.dev-scripts]
foo = 'foo:main'
test = { task = "pytest tests/" }

This might get pretty cumbersome 🤔

@Woile

This comment has been minimized.

Copy link

commented Nov 15, 2018

I'm really looking forward to see this feature merged. Most of the time adding a Makefile to a small project feels like too much, but I still want to have the commands documented, and it's much easier to keep it up to date if I use them constantly instead of writing them once in the README and forgetting.

It would be awesome if I could just remember poetry run test and forget about the underlying technology (pytest, pytest + codecov, unittest, nose, etc) and the args (if they are included in the future).

I really like the way yarn behaves. Where any script added to scripts can be executed directly using yarn.

E.g:

{
  "scripts": {
    "build": "node build.js"
  }
}

and then yarn build.

This could be really useful. If I redefine publish instead of executing the default behavior from poetry, I could do something completely different, this would help adoption of the tool.

@purificant

This comment has been minimized.

Copy link

commented Nov 15, 2018

I would really like this feature. Having this built in makes it easier to share knowledge amongst developers when adopting poetry as a tool of choice for dependency management on a project.
All typical activities such as test / lint / build / ci / deploy / publish can be automated through clearly defined scripts without further tooling.

Do not mind if it's poetry run or poetry task, although the first one is one less character. dev-scripts sounds good too, it's clear that they are intended for package developers and that they are scripts, which reduces cognitive load for users already familiar with the concept of npm scripts for example.

Thank you @michaeldel

@erksch

This comment has been minimized.

Copy link

commented Feb 18, 2019

What's the status of this? Do you need any help on getting this done?

@pawamoy pawamoy referenced this pull request Feb 18, 2019

Closed

Thanks #124

@purificant
Copy link

left a comment

There are conflicts in the PR that need to be resolved

@erksch

This comment has been minimized.

Copy link

commented Feb 19, 2019

I was able to merge the current master in this branch without any conflicts (the particular part in poetry/utils/env.py did not change much).

@michaeldel I opened a pull request on your tasks branch with the updates.

@michaeldel michaeldel force-pushed the michaeldel:tasks branch from cda3a4e to 5452423 Feb 22, 2019

michaeldel added some commits Feb 22, 2019

@michaeldel

This comment has been minimized.

Copy link
Author

commented Feb 22, 2019

Codebase has been updated, also resolved conflicts.

@Kamforka

This comment has been minimized.

Copy link

commented Feb 23, 2019

Is there any documentation available on this implementation yet?

So far it's not clear if we ended up using tools or dev-scripts?

@Woile

This comment has been minimized.

Copy link

commented Feb 23, 2019

It would be nice to have this instead:

[tool.poetry.dev-scripts]
foo = 'foo:main'
test = { task = "pytest tests/" }

Makes it easy to remember being able to reuse poetry run instead of having 2, one for python scripts and one for shell scripts.

I think tasks and dev-scripts will generate confussion.

@michaeldel

This comment has been minimized.

Copy link
Author

commented Feb 23, 2019

@Kamforka there is minimal documentation in cli.md. At the moment the section used is still [tool.poetry.tasks].

The subcommand name (currently task) has still not be settled, run feels terser but may cause issues because of name conflicts (e.g. a task and a script both called start). Considering we change it so, how do we handle those name conflicts?

  • Should one take priority on the other? (e.g. scripts considered higher priority than tasks) Also raising a warning telling about the conflict?
  • Should an error be raised instead?

Implementing tasks that way (using the run subcommand) will also probably involve more work when it comes to modifying the scripts operation, this sounds to be considered.

@Woile of course dev-scripts would fully replace tasks to avoid confusion.

@erksch

This comment has been minimized.

Copy link

commented Mar 1, 2019

@michaeldel do you mind update tasks to dev-scripts then?

@brycedrennan

This comment has been minimized.

Copy link
Collaborator

commented Mar 6, 2019

I'd rather direct people to use Makefiles, which are cross-language compatible, than to build yet another language-specific task runner. Unless poetry is managing python versions as well (with pyenv) then I'll still have to have a Makefile anyway.

But yeah I don't know about windows support.

@purificant

This comment has been minimized.

Copy link

commented Mar 7, 2019

What's with the holy crusade of trying to teach everyone about Makefiles? We get it, they've been around since early 1970s. They solve some problems, most of those problems have been solved again by many other tools many more times since then. If you want to use Makefiles they will give you a gun and enough rope to shoot / hang yourself with, whichever you prefer.

@michaeldel

This comment has been minimized.

Copy link
Author

commented Mar 9, 2019

@erksch I am taking care of it as soon as I got time, in some days probably.

@brycedrennan make is pretty cumbersome to use within a Python project using poetry (cf first post), finding a way to "activate" the Poetry environment might simplify this, but I have not studied all the work implementing this would involve. Windows support also sounds like an issue to me, it is still very widely used as a development environment despite its flaws.
I personally find getting development scripts into pyproject.toml the thing to do since it is striving to become the standard Python projects' main management file.

@brycedrennan

This comment has been minimized.

Copy link
Collaborator

commented Mar 9, 2019

@michaeldel yeah good points. I’ve got no answer for windows support so I must admit this will be helpful for a lot of projects.

@Kamforka

This comment has been minimized.

Copy link

commented Mar 9, 2019

Unfortunately I'm also doomed to develop under Windows and deploy to Linux and I know a lot others who have to deal with this.

Having poetry as a single source of truth might ease the pain a little bit though.

@hangtwenty

This comment has been minimized.

Copy link

commented Mar 12, 2019

another voice against tasks in pyproject.toml, alternative suggestion

I'm empathetic to the intent, but strongly opposed to rushing to an implementation

I am still wondering if there is a way to do this that would encourage people to use other tools that are dedicated to running tasks, like make or invoke, and having pyproject.toml have a way to point to those or help track those ... without absorbing all of the shell commands directly into the toml.

I, for one, do not look forward to pyproject.toml becoming a kitchen-sink mess. Lots of reasons people find npm's run thing to be a problem, no need to re-hash here. TLDR unix philosophy and separation of concerns...

Also this young in Poetry's life it is likely sensible to defer commitment / defer critical decisions. i.e. even if there is a concept of tasks in the TOML later, it doesn't feel ready to commit to one way yet.

the Windows thing

It's true that Windows may need something besides make but it's not Poetry's job to fill in for the endless limitations of Windows. There be dragons that way.

It also does not seem like a very Pythonic way to address Windows portability issues; "some string command" leaves it implicit what shell should be used, etc.

While there are multiple shells in Linux and Linuxlike environments, they tend to be compatible enough for most users at this level of abstraction. That can't be said of the different shells in Windows environments.

And while poetry run and poetry shell just use whatever shell... Somehow it seems more explicit and Pythonic than the stringy tasks case.

there cannot be any tool that is best as 'single source of truth' for every single part of your workflow

There is discussion of Poetry being a 'single source of truth,' and I understand the desire.

but I think that's not exactly the best way to look at it. Poetry is not well-suited to replace your .gitignore. Poetry is not well-suited to replace pre-commit. Poetry is not an IDE and it's not even an EditorConfig. Those things are all great at what they do and Poetry doesn't need to reinvent those wheels or overload the toml with that.

Yes Poetry is about the SDLC and being about build/packaging inherently means it covers some ground of dev/test. But we have to draw the line somewhere on what it does itself, and what it points to or integrates with. While some tools in other languages have made some other choices, we should give a lot of thought to what seems the most Pythonic for this. (Though I know everyone has their own interpretations on that.) Again we should defer critical decisions and mull this over (and try alternatives, and compare them)


alternative approach, that could address all of those concerns

if support is added for "concise run" ( #940 ) then for example you could use invoke and tasks.py, and then poetry inv my_task is going to work (inv is invoke (http://docs.pyinvoke.org/en/1.2/#the-invoke-cli-tool).

This is no less concise than poetry task my_task; it's easy to understand what's happening; and it pushes the complexities of a task runner to ... A task runner!

Something like this alternative could satisfy all of the desired outcomes in this thread (and the #241) ... including convenience ... while avoiding some serious yak-shave distractions for Poetry (the inevitable edge-cases that would happen from supporting inlined tasks)

@erksch

This comment has been minimized.

Copy link

commented Mar 12, 2019

@hangtwenty
I think the alternative approach you are proposing would not add so much value to poetry. If you set up invoke tasks it's not that hard to just call them with invoke. I see no benefit in moving the call to peotry.

Important is the difference between something like a make, invoke or shovel task and the proposed tasks we (or I) would like to have in poetry. I would write a shovel task for something like seeding my database or setting up constraints or whatever but I would not want to write a shovel task for executing my tests or linting my code. The difference is that the first ones are tasks and the second ones are convenience aliases for executing a tiny bit of code.

And why would adding scripts to the toml result in a mess? Making task runner tasks for everything would be the greater mess.

@Askir

This comment has been minimized.

Copy link

commented Mar 12, 2019

I'm not sure if everyone is actually scared of a result that would look like this. And even if that is the case you'd just move that mess from the pyproject.toml to another file depending on what task runner you use.


I do agree with @erksch on the fact that these scripts/tasks are mostly aliases. As soon as the scripts/tasks become complicated they are moved to different files (see the npm example).
They abstract from the modules that are actually used in the project. Because every project might use a different test runner but poetry run tests would always stay the same.

I think the dream of yarn and npm is that you can start developing simply after running git clone and yarn install. And I personally would love if that was the same for poetry. And installing dependencies is just one of the hassles that i have to go through as a developer.

If I want to test I need to know the testing library that the repository uses, if it is pytest I run poetry run pytest if it is pyunit I run poetry run pyunit if it is mamba I run poetry run mamba then I still need to know the folder where the testing files are saved.

Now the same steps happen when i want to lint. Does the project use prospector? or simply pylint?
All these questions would not have to be asked if I could simply run poetry run lint or poetry run tests.


Now the alternative solutions that are suggested are:

  • Use Make!
    Well now I have to run poetry install and install Make for this project.
    Just to again call poetry commands from the Makefile?

  • Use invoke!
    This makes a bit more sense as it will load the environment already, but again I need to know that behind the scenes the project uses invoke and not a different task runner. It doesn't streamline the process as well as tasks/scripts would do. You could still use invoke to test you'd just have to add invoke tests behind your custom poetry run tests command.

@hangtwenty

This comment has been minimized.

Copy link

commented Mar 13, 2019

@erksch You mention aliases - I understand what you mean but then let’s compare to aliases in shell scripting. In bash for example aliases can still have further arguments added to them. Would pyproject.toml aliases support that? Or do people have to create redundant aliases with slight variations? Or start using environment variables a bunch? Would pyproject.toml tasks be able to call each other? Do we want to deal with pyproject.toml tasks calling multiple layers of “poetry run” subshells (and does that have any weirdness to consider?)

These questions are answerable but the point I’m trying to make is that aliases are not as simple as you are implying. They only seem simple to you because they are your personal preference.

To others of us, this kind of hardcoding of shell invocations into text is not simple. Instead it proliferates custom work. It does not have less cognitive overhead than the alternative(s), it’s just less consistent.

Contrast with the case where the tasks are executed by a well-defined task runner or sure you can do your own custom approach visible right there. Either way people don't start digging into poetry when they need to debug. The dig into the thing they chose. It's more explicit, and explicit is better than implicit in Pythonic code.

Also I’m not trying to get tit for tat, my higher-level point here is that any approach to this should probably be considered slowly and carefully.

@hangtwenty

This comment has been minimized.

Copy link

commented Mar 13, 2019

The issue #940 could be widely useful regardless of whether there is something like pyproject.toml in-line classes.

So then why not do that first, and see how it goes, and let this incubate - instead of immediately adding more semantics to the pyproject.toml?

@hangtwenty

This comment has been minimized.

Copy link

commented Mar 13, 2019

@Askir It looks like something needs clarifying. You said

This makes a bit more sense as it will load the environment already, but again I need to know that behind the scenes the project uses invoke and not a different task runner.

What do you mean? I mentioned the example of poetry inv my_task. inv in that case is invoke, and that has nothing to do with poetry; when you install invoke it creates both entrypoints (see invoke docs: http://docs.pyinvoke.org/en/1.2/#the-invoke-cli-tool )

Per #940 this would be a general concept for running entrypoints. It could be implemented almost trivially, i.e. if poetry <subcommand> is not known, it poetry would check if <subcommand> is an existing executable (like which but portable), if yes then it proceeds.

So you may have thought I was suggesting a way to glue a specific task runner to poetry.

There is no magic and there is no 'behind-the-scenes' to understand. Literally the only difference from current behavior is that you could save typing three characters. poetry run inv my_task would be the same thing today. poetry inv my_task would be the convenience.

To further illustrate the point,

# equivalent to 'poetry run inv my_task'
poetry inv my_task        

# equivalent to 'poetry run invoke my_task'
poetry invoke my_task  

# because it's just 'poetry run' the semantics for adding extra arguments are obvious
poetry inv my_task --flag --blah

# equivalent to 'poetry run flake8 .'
poetry flake8 .

# equivalent to 'poetry run pytest tests/'
poetry pytest tests/

Those examples so far are Python entrypoints. it'd probably make sense if it just worked for anything run worked for though. which means it's anything in your shell.

poetry echo "hello world"              # equivalent to: poetry run echo "hello world"

and so on.

@hangtwenty

This comment has been minimized.

Copy link

commented Mar 13, 2019

I only used invoke as an example so far because it is an extremely mature option (when you consider its previous life as Fabric), and it is a go-to on many projects.

http://docs.pyinvoke.org/en/1.2/concepts/configuration.html#setup
http://docs.pyinvoke.org/en/1.2/getting-started.html#declaring-pre-tasks
http://docs.pyinvoke.org/en/1.2/concepts/invoking-tasks.html#how-tasks-run


It's just poetry run though. You could use whatever you want.

Using doit would the least difference from 'hardcoding a buncha strings to invoke as commands', yet like invoke, it has a clear path forward once your project is not "just" simple anymore. And we don't overload poetry with it forcing choices about how "tasks" should look. It's a modular approach.

http://pydoit.org/usecases.html#simplify-cumbersome-command-line-calls
https://github.com/pydoit/doit#sample-code
http://pydoit.org/tutorial_1.html#doit-task
http://pydoit.org/tasks.html#task-selection
http://pydoit.org/task_creation.html#custom-task-definition
http://pydoit.org/usecases.html#speed-up-by-parallel-task-execution

lint = "black --check ."

👇

# dodo.py
def task_lint():
    return {'actions': ['black --check .']}

poetry run doit lint would do this currently; then after #940 it'd be poetry doit lint - poetry doit lint is no less clean than poetry task lint so it's quite compatible with the proposed usages in this thread


Here is another example, implementing all of the OP's tasks with doit in a way similar to the "flat hardcoded aliases" idiom that you'd have if you inlined all this stuff to the TOML.

aliases = dict(
    lint    = ["black --check **/*.py"],
    test    = ["pytest tests/"],
    dev     = ["FLASK_APP=app.py flask run"],
    start   = ["gunicorn app.wsgi --log-file -"],
    clean   = ["poetry run manage.py flush", "rm -rf media"]
)

# small bit of boilerplate, this is a way that doit lets us generate task functions.
# http://pydoit.org/task_creation.html#custom-task-definition
class Aliases(object):
    @staticmethod
    def create_doit_tasks():
        return (({'actions': actions, 'basename': name, 'targets': [name]}) for name, actions in aliases.items())

Even though there is that bit of boilerplate, don't lose sight of the important part: once any one of your tasks outgrows hardcoded-string-aliases, you can use the full features of your chosen task-runner.

Personally I don't care about 'one-liners' so I'd make them all "real" tasks from the start. But that's the thing ... I can make the choices for my project, and you can make your choices.


In both examples given, invoke and doit, you have the tools to have tasks depend on each other, to run 'as needed', to watch your files and re-run (inotify), and so on and so on. doit has features where it can parallelize task execution, but keeps sequential dependencies running in parallel within that.

Yes Poetry can, and should be, opinionated about some things. This is an opinionated stance too, and it encourages users of poetry to keep their dev tasks cohesive. That means you don't have a split between your stringy tasks and your "real" tasks when your needs get complicated.


TLDR ... modular / separation of concerns / tools doing their primary things well. Poetry doesn't have to reinvent any wheels for this -- or at least we can address #940 now (simple) and consider doing other things later.

@actionless

This comment has been minimized.

Copy link

commented Jun 12, 2019

it seems the people who proposing alternative solutions (make, pyinvoke, pydoit, etc) not following the main motivation behind this idea -- to avoid having extra dependencies or extra files in project for this more than trivial purpose

@funkyfuture

This comment has been minimized.

Copy link

commented Jun 20, 2019

i don't see any value in this proposal for poetry, but only maintenance cost.

@maxsu

This comment has been minimized.

Copy link

commented Jun 21, 2019

Can we merge this code please? This feature is a big part of why I fell in love with NPM/Yarn, and I really need it to help onboard my team with a minimum of cognitive burden.

No, I will not be teaching them to use make. Ever.

Anyone saying to use other tools please ask yourself whether you've actually made a PR on this project, and if the answer is no, go ahead and defer to and support the folks who have contributed code or have a genuine vision for what this project can do if it defies the pythonic status quo.

The old pythonic ways lead to stagnant, schizophrenic designs, high cognitive loads, and incredibly slow progress. To wit, we have been flogging this issue for over half a year using meaningless and counterproductive rationale such as pythonicity and "do one thing and do it well", when in fact this feature and a complementary cluster of others were already solved and integrated into a coherent whole with resounding success in another community over 5 years ago. If you'd like a more genuine opportunity to contribute, stop pulling the handbrake on this project and go triage issues on the tools you're recommending. I assure you they exist. And are miserable, if you dare face them.

To conclude my screed, @sdispater are there any gaps preventing the integration of the current PR? If so, what are they? Can we move forward here? I'd really like to show this feature to the team :)

@pmav99

This comment has been minimized.

Copy link

commented Jun 21, 2019

old
stagnant,
schizophrenic designs
high cognitive loads
incredibly slow progress
flogging
meaningless and counterproductive rationale

@maxsu Regardless of the issue at hand, reserving this sort of toxicity for your self, or, if that is not possible, to communities that may welcome such behaviour would be a great first step towards contributing.

@brycedrennan

This comment has been minimized.

Copy link
Collaborator

commented Jun 21, 2019

@maxsu

It would be nice to have a single tool that could “do it all”, I’m not opposed to that in principle but I am opposed to adding features when core functionality isn’t complete. Last I checked (admittedly a few months ago) there were several blockers to rolling poetry out in my corporate environment. (#524 #697 #800).

NPM is a great example. It tried to do a lot and was so bad at its core functionality that it had to be replaced by a Yarn. Not the kind of “success” I’d hope for poetry.

I also think some people here are underestimating the complexity of task runners. It’s easy to get the simple cases working but we will inevitably also want the more complex cases working. That’s why there are whole libraries to deal with solving that problem.

As with any open source software of decent size, there are lots of competing visions for what the software “should” be. It’s generally valuable to have respectful debates about the merits of new features. It’s up to the maintainers, as informed by the input of everyone, to decide what fits the vision. I don’t think it’s productive to tell people with differing opinions they aren’t “genuinely” contributing.

I’m hesitant to post my contact info here but it’s pretty easy to find. Id be happy to learn more about your python pain points (outside of this thread) and see if we can find ways to make things easier for your team (for free). I do have a bit of experience with this sort of thing. Let me know!

@sdispater

This comment has been minimized.

Copy link
Owner

commented Jun 23, 2019

Poetry is a package and dependency manager, not a task manager. This feature is beyond the scope and the original purpose of Poetry and will likely never be integrated into it. I want to keep Poetry simple and intuitive so I carefully think before adding new features to the core codebase since the weight of maintaining it will fall on the shoulders of the maintainers.

That being said, when the planned plugin system is fully implemented (which will be after the release of the 1.0 version) anyone will be free to add features via plugins. This is the best tradeoff I can find and will hopefully help build an ecosystem around the core project and keep its simplicity.

@maxsu I am grateful of anyone willing to take some of their free time to make a PR for Poetry. I really do. However, that does not mean I will blindly accept any PR, especially when they add new features, if they do not align with the purpose of the project. I know this can be sometime disappointing to see your PR rejected after spending time on it but as a project maintainer it's my responsibility to determine what's best for the project. And that means, sometimes, to turn down good and thorough suggestions like this one.

@michaeldel Thanks a lot for taking the time to make this PR! I really appreciate it. However, for the reasons mentioned above, I don't think this align with the purpose of Poetry. Nonetheless, like I said this would be a perfect fit for a plugin once the plugin system is ready. So keep an eye on the project for when this will be announced.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.