list_format parameters %s and %k broken

[MinchinWeb]

When I run: topydo ls -F "%s" or topydo ls -F "%k" it returns Error while parsing format string (list_format config option or -F)

However, when combined with -N, both work (i.e. topydo ls -F "%s" -N)

Windows 10, topydo version 0.13 and HEAD, Python 3.7 and 3.9.

This used to work (it's in my config file from a long time ago).

"%s" is particularly annoying, as that's the "text" of the todo item.

[mruwek]

I could not reproduce that bug at Cygwin (Python 3.8) under Win7 nor on Linux (Fedora 33 with Python 3.9). It may be some windows-specific issue. Can You post full error output?

[MinchinWeb]

That's all that is returned (topydo 0.13, Python 3.8.6, Powershell Core 7.1.0 on Windows 10) :(

[mruwek]

1. ls -f is for output format (json, ical, or text) not for list_format so there must be something in your config. Can you post it here?
2. This exception is typically used when list_format breaks regexp rules.
3. To pinpoint this problem change raise ListFormatError to simply raise at this line here. Compile and try again. At least we will see some more output from re.error. Maybe this will lead us to the source of the error.

[MinchinWeb]

The error message is the same for both -f (as in my image) and -F... that's probably why I missed it. Sorry!

I changed the line as you suggested, and yes, it is an re error:

C:\> topydo ls -F "%s"
Traceback (most recent call last):
  File "C:\Python39\lib\sre_parse.py", line 1039, in parse_template
    this = chr(ESCAPES[this][1])
KeyError: '\\R'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\Code\topydo\.venv\Scripts\topydo-script.py", line 33, in <module>
    sys.exit(load_entry_point('topydo', 'console_scripts', 'topydo')())
  File "e:\code\topydo\topydo\ui\UILoader.py", line 62, in main
    CLIApplication().run()
  File "e:\code\topydo\topydo\ui\cli\CLI.py", line 61, in run
    if self._execute(subcommand, args) == False:
  File "e:\code\topydo\topydo\ui\CLIApplicationBase.py", line 271, in _execute
    if command.execute() != False:
  File "e:\code\topydo\topydo\commands\ListCommand.py", line 209, in execute
    self._print()
  File "e:\code\topydo\topydo\commands\ListCommand.py", line 156, in _print
    self.out(self.printer.print_list(self._view().todos))
  File "e:\code\topydo\topydo\lib\printers\PrettyPrinter.py", line 87, in print_list
    return [self.print_todo(todo) for todo in p_todos]
  File "e:\code\topydo\topydo\lib\printers\PrettyPrinter.py", line 87, in <listcomp>
    return [self.print_todo(todo) for todo in p_todos]
  File "e:\code\topydo\topydo\lib\printers\PrettyPrinter.py", line 77, in print_todo
    todo_str = ppf.filter(todo_str, p_todo)
  File "e:\code\topydo\topydo\lib\prettyprinters\Format.py", line 32, in filter
    p_todo_str = self.parser.parse(p_todo)
  File "e:\code\topydo\topydo\lib\ListFormat.py", line 292, in parse
    substr = re.sub(r'(?<!\\)%({ph}|\[{ph}\])'.format(ph=placeholder), repl, substr)
  File "C:\Python39\lib\re.py", line 210, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "C:\Python39\lib\re.py", line 327, in _subx
    template = _compile_repl(template, pattern)
  File "C:\Python39\lib\re.py", line 318, in _compile_repl
    return sre_parse.parse_template(repl, pattern)
  File "C:\Python39\lib\sre_parse.py", line 1042, in parse_template
    raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \R at position 91

Or with "%k":

C:\> topydo ls -F "%k"
Traceback (most recent call last):
  File "C:\Python39\lib\sre_parse.py", line 1039, in parse_template
    this = chr(ESCAPES[this][1])
KeyError: '\\A'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\Code\topydo\.venv\Scripts\topydo-script.py", line 33, in <module>
    sys.exit(load_entry_point('topydo', 'console_scripts', 'topydo')())
  File "e:\code\topydo\topydo\ui\UILoader.py", line 68, in main
    CLIApplication().run()
  File "e:\code\topydo\topydo\ui\cli\CLI.py", line 61, in run
    if self._execute(subcommand, args) == False:
  File "e:\code\topydo\topydo\ui\CLIApplicationBase.py", line 271, in _execute
    if command.execute() != False:
  File "e:\code\topydo\topydo\commands\ListCommand.py", line 209, in execute
    self._print()
  File "e:\code\topydo\topydo\commands\ListCommand.py", line 156, in _print
    self.out(self.printer.print_list(self._view().todos))
  File "e:\code\topydo\topydo\lib\printers\PrettyPrinter.py", line 87, in print_list
    return [self.print_todo(todo) for todo in p_todos]
  File "e:\code\topydo\topydo\lib\printers\PrettyPrinter.py", line 87, in <listcomp>
    return [self.print_todo(todo) for todo in p_todos]
  File "e:\code\topydo\topydo\lib\printers\PrettyPrinter.py", line 77, in print_todo
    todo_str = ppf.filter(todo_str, p_todo)
  File "e:\code\topydo\topydo\lib\prettyprinters\Format.py", line 32, in filter
    p_todo_str = self.parser.parse(p_todo)
  File "e:\code\topydo\topydo\lib\ListFormat.py", line 292, in parse
    substr = re.sub(r'(?<!\\)%({ph}|\[{ph}\])'.format(ph=placeholder), repl, substr)
  File "C:\Python39\lib\re.py", line 210, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "C:\Python39\lib\re.py", line 327, in _subx
    template = _compile_repl(template, pattern)
  File "C:\Python39\lib\re.py", line 318, in _compile_repl
    return sre_parse.parse_template(repl, pattern)
  File "C:\Python39\lib\sre_parse.py", line 1042, in parse_template
    raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \A at position 3

One other oddity: if I replace the line you mentioned with pass, it works as expected (i.e. it lists my todo items).

(This is Python 3.9.0, and the HEAD of topydo, reported version 0.14).

[mruwek]

I'm starting to think that the problem is in one or more of your tasks. Are there any \ signs in any line of your todo.txt?
You can bisect todo.txt by yourserlf for exaample by starting anew with blank todo.txt and adding lines one-by-one from the old copy. Or you can try splitting your todo.txt in smaller files (let's say 5-10 per file) and check witch files shows error or not.
You can also try some debugging fun and:

diff --git a/topydo/lib/ListFormat.py b/topydo/lib/ListFormat.py
index 8b990f0..8c6ca9f 100644
--- a/topydo/lib/ListFormat.py
+++ b/topydo/lib/ListFormat.py
@@ -221,6 +221,8 @@ class ListFormatParser(object):
             'z': lambda t: color_block(t) if config().colors() else ' ',
         }
         self.format_list = self._preprocess_format()
+        with open('re-error-dbg.txt', 'w') as f:
+            f.write(str(self.format_list) + '\n')

     def _preprocess_format(self):
         """
@@ -276,6 +278,8 @@ class ListFormatParser(object):
         """
         parsed_list = []
         repl_trunc = None
+        with open('re-error-dbg.txt', 'a') as f:
+            f.write(str(p_todo.source()) + '\n')

         for substr, placeholder, getter in self.format_list:
             repl = getter(p_todo) if getter else ''

Then please post re-error-dbg.txt contents here. We shall find that way the task that triggers the exception and we'll see if pre-processed format_list is properly built.

You can also send your todo.txt to me and I'll do my best to hunt this issue :)

[MinchinWeb]

Yes, there are \ on some of the entries. I'm on Windows and they are used for filepaths.

I tired the changes you mentioned, and I think these are the two lines (of interest) from the log file:

[('|', None, None), ('I| ', 'I', <function ListFormatParser.__init__.<locals>.<lambda> at 0x00000148B585EAF0>), ('x ', 'x', <function ListFormatParser.__init__.<locals>.<lambda> at 0x00000148B5861280>), ('{(}p{)} ', 'p', <function ListFormatParser.__init__.<locals>.<lambda> at 0x00000148B585EDC0>), ('c ', 'c', <function ListFormatParser.__init__.<locals>.<lambda> at 0x00000148B5689790>), ('s ', 's', <function ListFormatParser.__init__.<locals>.<lambda> at 0x00000148B585EF70>), ('k ', 'k', <function ListFormatParser.__init__.<locals>.<lambda> at 0x00000148B585EB80>), ('{due:}d ', 'd', <function ListFormatParser.__init__.<locals>.<lambda> at 0x00000148B578E670>), ('{t:}t', 't', <function ListFormatParser.__init__.<locals>.<lambda> at 0x00000148B58610D0>)]

and

2013-12-14 read 'Design and Development of a Statewide Land Use Transport Model for Alberta' at 'S:\Archive\Documents\Research\Sources\Personal Interest' +reading_list

[mruwek]

I can replicate %k error with the task you posted. Moreover, I think I know why this error happens (topydo still looks for tags despite quotation marks and it treats S as tag key and \Archive\Documents\Research\Sources\Personal as tag value.

Yet I can't reproduce %s error. Can you try once more ls -F %s with aforementioned debug patch and check which task is this time on the last line of re-error-dbg.txt?

[MinchinWeb]

It seems to break on the same line. Here's the traceback:

C:\> topydo ls -F %s
Traceback (most recent call last):
  File "C:\Python39\lib\sre_parse.py", line 1039, in parse_template
    this = chr(ESCAPES[this][1])
KeyError: '\\R'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\Code\topydo\.venv\Scripts\topydo-script.py", line 33, in <module>
    sys.exit(load_entry_point('topydo', 'console_scripts', 'topydo')())
  File "e:\code\topydo\topydo\ui\UILoader.py", line 68, in main
    CLIApplication().run()
  File "e:\code\topydo\topydo\ui\cli\CLI.py", line 61, in run
    if self._execute(subcommand, args) == False:
  File "e:\code\topydo\topydo\ui\CLIApplicationBase.py", line 271, in _execute
    if command.execute() != False:
  File "e:\code\topydo\topydo\commands\ListCommand.py", line 209, in execute
    self._print()
  File "e:\code\topydo\topydo\commands\ListCommand.py", line 156, in _print
    self.out(self.printer.print_list(self._view().todos))
  File "e:\code\topydo\topydo\lib\printers\PrettyPrinter.py", line 87, in print_list
    return [self.print_todo(todo) for todo in p_todos]
  File "e:\code\topydo\topydo\lib\printers\PrettyPrinter.py", line 87, in <listcomp>
    return [self.print_todo(todo) for todo in p_todos]
  File "e:\code\topydo\topydo\lib\printers\PrettyPrinter.py", line 77, in print_todo
    todo_str = ppf.filter(todo_str, p_todo)
  File "e:\code\topydo\topydo\lib\prettyprinters\Format.py", line 32, in filter
    p_todo_str = self.parser.parse(p_todo)
  File "e:\code\topydo\topydo\lib\ListFormat.py", line 296, in parse
    substr = re.sub(r'(?<!\\)%({ph}|\[{ph}\])'.format(ph=placeholder), repl, substr)
  File "C:\Python39\lib\re.py", line 210, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "C:\Python39\lib\re.py", line 327, in _subx
    template = _compile_repl(template, pattern)
  File "C:\Python39\lib\re.py", line 318, in _compile_repl
    return sre_parse.parse_template(repl, pattern)
  File "C:\Python39\lib\sre_parse.py", line 1042, in parse_template
    raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \R at position 91

[mruwek]

Alright. Try that branch from my repo. It fixes aforementioned bug with looking for tags inside quotes and escapes backslashes if needed.

If this will be sufficient resolution I would suggest to fix it this way in proper PR next week.

[MinchinWeb]

It's still error-ing out on your new branch, although a new one this time. I tried adding back in your debugging patch from above, but it's not creating the log file this time.

Traceback (most recent call last):
  File "e:\code\topydo\topydo\ui\UILoader.py", line 44, in main
    if args[0] == 'prompt':
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\Code\topydo\env\Scripts\topydo-script.py", line 33, in <module>
    sys.exit(load_entry_point('topydo', 'console_scripts', 'topydo')())
  File "e:\code\topydo\topydo\ui\UILoader.py", line 64, in main
    CLIApplication().run()
  File "e:\code\topydo\topydo\ui\cli\CLI.py", line 50, in run
    self.todolist = TodoList.TodoList(self.todofile.read())
  File "e:\code\topydo\topydo\lib\TodoList.py", line 72, in __init__
    super().__init__(p_todostrings)
  File "e:\code\topydo\topydo\lib\TodoListBase.py", line 55, in __init__
    self.add_list(p_todostrings)
  File "e:\code\topydo\topydo\lib\TodoListBase.py", line 149, in add_list
    todos = [Todo(src) for src in p_srcs]
  File "e:\code\topydo\topydo\lib\TodoListBase.py", line 149, in <listcomp>
    todos = [Todo(src) for src in p_srcs]
  File "e:\code\topydo\topydo\lib\Todo.py", line 35, in __init__
    TodoBase.__init__(self, p_str)
  File "e:\code\topydo\topydo\lib\TodoBase.py", line 44, in __init__
    self.set_source_text(p_src)
  File "e:\code\topydo\topydo\lib\TodoBase.py", line 180, in set_source_text
    self.fields = parse_line(self.src)
  File "e:\code\topydo\topydo\lib\TodoParser.py", line 98, in parse_line
    for word in shlex.split(rest, posix=False):
  File "C:\Python39\lib\shlex.py", line 315, in split
    return list(lex)
  File "C:\Python39\lib\shlex.py", line 300, in __next__
    token = self.get_token()
  File "C:\Python39\lib\shlex.py", line 109, in get_token
    raw = self.read_token()
  File "C:\Python39\lib\shlex.py", line 191, in read_token
    raise ValueError("No closing quotation")
ValueError: No closing quotation

[mruwek]

The shlex.split() usage in 1717a51 probaly messes with this mode. You have to have some task with unclosed quotation mark. shlex properly treats apostrophes like in gate's, but it will crash on something like: This 'task has opening quote without closing-one.

I'll try to address that this week.

[mruwek]

Ok, try to update my branch. It shouldn't crash anymore. TodoParser will now add additional quotes if lone quotation mark is detected.

[MinchinWeb]

It works! Thank you, good sir, in tracking that down and fixing it!

On thing I note is that it highlights the path (at least until the first space) as if it were a tag. Without a whitelist of tags, there's probably no way around that, and it doesn't bother me.

[mruwek]

On thing I note is that it highlights the path (at least until the first space) as if it were a tag.

Yep. I see it too. Maybe we can test if strings destined to be coloured are actually in Todo.tags()? I'll test such resolution but I'm not sure if this is not an overkill.

This branch grows thicker and thicker. I would love any feedback from someone smarter than me before posting it as a PR. @bram85, can You take a quick look at the code?