Convert REDIRECTION_CHARS to REDIRECTION_TOKENS

[kotfu]

Currently we look at REDIRECTION_CHARS in many places in the code: parsing, tab completion, etc. REDIRECTION_CHARS is a constant and is set to ['>', '|'. When we use this constant, we have to do additional code to make sure that we catch >> as a valid redirection, but not ||.

We should really convert REDIRECTION_CHARS into a constant called REDIRECTION_TOKENS, which would be set to ['>', '>>', '|']. This would force code that uses this constant to be generalized so that if we change the constant, the code still works.

Related to #381.

[kmvanbrunt]

This gets a little tricky in setting up readline delimiters, which are supposed to be single characters. The current code is the following.

# Break words on whitespace and quotes when tab completing
completer_delims = " \t\n" + ''.join(constants.QUOTES)

if self.allow_redirection:
    # If redirection is allowed, then break words on those characters too
    completer_delims += ''.join(constants.REDIRECTION_CHARS)

readline.set_completer_delims(completer_delims)

If we allow multicharacter redirectors (#381 suggests '->'), each character of that string will need to be a delimiter. As long as Cmd2.tokens_for_completion() and StatementParser._split_on_punctuation() are updated to support multicharacter redirectors, I think tab completion should still work OK.

I will say updating Cmd2.tokens_for_completion() and StatementParser._split_on_punctuation() like this will add complexity to their fairly simple code. It was nice just examining each character of the tokens. The current code also matches nicely with how shlex treats punctuation as individual characters in Python 3.6+.

[kotfu]

Agreed. I like keeping the logic similar to shlex. I'll close this issue with a wontfix label. This means that we will have to deprecate Cmd.redirector or modify it's behavior so it only accepts a single character, in issue #381.