[BUG] invoke --list results in IndexError: list index out of range

[mfekadu]

Description of the bug

When running invoke --list I receive a traceback containing an IndexError due to having too many tasks in one task.py

To Reproduce

Preconditions

1. make an empty folder and change into it

mkdir test_pyinvoke_issue_704 && cd test_pyinvoke_issue_704

2. download the task.py that I was working with

wget https://gist.githubusercontent.com/mfekadu/8b35a73776e0297e317fdca6382bd898/raw/073291138ff49f6cd70768a63e4d377160004d95/tasks.py

Steps To Reproduce

1. run invoke --list
2. notice the IndexError

$ invoke --list

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/bin/invoke", line 8, in <module>
    sys.exit(program.run())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/invoke/program.py", line 377, in run
    self.parse_cleanup()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/invoke/program.py", line 516, in parse_cleanup
    self.list_tasks()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/invoke/program.py", line 788, in list_tasks
    getattr(self, "list_{}".format(self.list_format))()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/invoke/program.py", line 791, in list_flat
    pairs = self._make_pairs(self.scoped_collection)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/invoke/program.py", line 837, in _make_pairs
    pairs.append((full, helpline(task)))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/invoke/util.py", line 172, in helpline
    return docstring.lstrip().splitlines()[0]
IndexError: list index out of range

Workaround

Removing some of the tasks does resolve the IndexError but the expected behavior of invoke when running invoke --list is to simply list out all @tasks without error.

Additional details

tasks still work fine

No other tasks are affected. The tasks work just fine. Only invoke --list fails.

For example, you can still run

invoke hi PyInvoke
invoke go-to-invoke-docs
invoke hiscala

here is the list of all tasks within that file

$ cat tasks.py | grep def | grep "(c"

def _dash_dash_list(c):
def github(c, username="INSERT_USERNAME", repo="INSERT_REPOSITORY_NAME"):
def gist(c, edit=False, username="mfekadu", gist_hash="8b35a73776e0297e317fdca6382bd898"):
def github_desktop(c):
def go_to_invoke_docs(c):
def hi(c, name, help=False):
def black_auto_format(c, verbose=True):
def hello_scala(c, verbose=True, name="hello_scala"):
def copy_tasks_py_to_clipboard(c):
def copy_ssh(c):

Cleanup after reproducing bug

cd .. && rm -i -r test_pyinvoke_issue_704

• "examine...?" yes
• "remove...?" yes
• "remove...?" yes

[mfekadu]

I have identified the bash command pipeline above as an amusing workaround

@task(aliases=("list", "lsit", "ist", "-list", "lis", "li", "slit", "slist"))
def _dash_dash_list(c):
    """
    because i forget --list often and fixz my ttypos
    """
    try:  # because pyinvoke issue #704
        c.run("invoke --list", hide="err")
    except Exception as e:
        print("uh oh, https://github.com/pyinvoke/invoke/issues/704")
        print("but here, try this...\n")
        cmd = 'cat tasks.py | grep def | grep "(\c"'  # \c avoid self-reference
        print(f"$ {cmd}\n")
        c.run(cmd)

It's a custom task that is invoked by invoke list which somewhat covers the features of invoke --list.

Honestly, this almost feels like a meme, "yo dawg, I heard you have an invoke bug. So I used invoke to make a workaround for your invoke bug. So you can invoke while working with invoke."

The above code allows the following...

$ invoke list

uh oh, https://github.com/pyinvoke/invoke/issues/704
but here, try this...

$ cat tasks.py | grep def | grep "(\c"

def _dash_dash_list(c):
def github(c, username="INSERT_USERNAME", repo="INSERT_REPOSITORY_NAME"):
def github_desktop(c):
def go_to_invoke_docs(c):
def hi(c, name, help=False):
def black_auto_format(c, verbose=True):
def hello_scala(c, verbose=True, name="hello_scala"):
def copy_tasks_py_to_clipboard(c):
def copy_ssh(c):

[mfekadu]

It turns out that the above task is an insufficient regex that

1. depends on the user being in the same directory as tasks.py

2. does not capture aliases

3. does not capture multiline function argument definitions

example of what I mean by "multiline function argument definitions"

@task
def foo(
        c, arg="some_really_really_really_really_long_string"
):
    pass

improved, yet still amusing workaround...

import re

@task(aliases=("list", "lsit", "ist", "-list", "lis", "li", "slit", "slist"))
def _dash_dash_list(c):
    """
    because i forget --list often and fixz my ttypos
    """
    BAD_KEY_MSG = "uh oh, maybe pyinvoke changed the Config object?"
    # it would be nice to have a simple `c.config.get_project_prefix()`
    assert c.config.__dict__.get("_project_prefix", None) != None, BAD_KEY_MSG
    PROJECT_DIR = c.config.__dict__.get("_project_prefix", None)
    TASK_FILE = os.path.join(PROJECT_DIR, "tasks.py")

    try:  # because pyinvoke issue #704
        c.run("invoke --list", hide="err")
    except Exception as e:
        print("uh oh, https://github.com/pyinvoke/invoke/issues/704")
        print("but here, try this...\n")
        # https://unix.stackexchange.com/a/37316
        with open(TASK_FILE, "r") as f:
            # https://regex101.com
            regex = r"@task\(?.*\)?(?=\s*.*def)\sdef \w*\(\s*.*\s*\):"
            code = f.read()
            for match in re.findall(regex, code):
                print(match, "\n")

$ inv list

➜  test_pyinvoke_issue_704 inv list
uh oh, https://github.com/pyinvoke/invoke/issues/704
but here, try this...

@task
def foo(
        c, arg="some_really_really_really_really_long_string"
):

@task(aliases=("list", "lsit", "ist", "-list", "lis", "li", "slit", "slist"))
def _dash_dash_list(c):

@task(aliases=("gh", "repo", "remote", "origin"))
def github(c, username="INSERT_USERNAME", repo="INSERT_REPOSITORY_NAME"):

@task(aliases=("gsit", "gst", "sgit", "gis", "gsi", "giat", "gisr", "gsot", "gost"))
def gist(c, edit=False, username="mfekadu", gist_hash="8b35a73776e0297e317fdca6382bd898"):

@task(aliases=("ghd", "desktop"))
def github_desktop(c):

@task(aliases=("invoke", "wtf", "huh", "what", "umm", "uhh", "idk"))
def go_to_invoke_docs(c):

@task(help={"name": "Name of the person to say hi to."})
def hi(c, name, help=False):

@task(aliases=("format", "black", "lint"))
def black_auto_format(c, verbose=True):

@task(aliases=("sc", "scala", "hi-scala", "hiscala", "helloscala"))
def hello_scala(c, verbose=True, name="hello_scala"):

@task(aliases=("copy", "pbcopy"))
def copy_tasks_py_to_clipboard(c):

@task(aliases=("ssh",))
def copy_ssh(c):

➜  test_pyinvoke_issue_704

[draganbobas]

The bug is not due to having too many tasks it's because there is an empty docstring for some of them.
From the error output, this line raises the error:
return docstring.lstrip().splitlines()[0]
It assumes that something is inside the docstring.
The workaround is actually removing the empty docstring.
Still a bug though.

[mfekadu]

Interesting! Thanks @draganbobas

Sounds like an easy bug to fix with some smarter indexing