Tokenizer tests

[JoshMcguigan]

I wrote some tests for the tokenizer to understand how it works. In the process I believe I found a bug in the handling of a command like long-running-process & date.

I was actually hoping to get to adding pipes, but looking more closely it seems adding more complexity to this would benefit from restructuring the parsing, perhaps to use something like conch-parser, but that really starts to move this away from being an educational project.

[psinghal20]

@JoshMcguigan Thanks for adding tests!
I saw that test case, didn't knew & can also act as command delimiter. I thought a ; or && would be required for multiple commands.
Yes, It seems logical to use a well-written shell command parser. I was planning to write a command parser. Don't know when I would be able to get to that.
Merging this for now. If you plan on fixing parser for that test case, please raise another PR. 😄