Fix a bunch of issues

[stroncium]

Added tests, fixed incorrect treatment of rows which masked problem with tokenizer to proper way to treat tokens.

---

Fixes #23
Fixes #24
Fixes #25
Fixes #26
Fixes #27

[stroncium]

Turned out, #23 was depending on both #26 and #25(which also fixes #27), so I merged them all into one PR.

[stroncium]

Proper tests for #24 (part about leading spaces) require previous fixes in this PR, so I merged this fix here too.

[vadimdemedes]

Hey @stroncium, great work! Would you be able to copy test cases from https://github.com/chalk/wrap-ansi/pull/29/files#diff-1dd241c4cd3fd1dd89c570cee98b79dd into your PR to verify your fixes cover more stuff, so that I can close my PR?

[stroncium]

@vadimdemedes Most of them were already there, added remaining except this one:

test('trim beginning of string if it starts with a space and not wrapped into background color escape', t => {
	const res = t.is(m(chalk.green(' hello '), 10, {hard: true, trim: false});
	t.is(res, chalk.green('hello '));
});

as I do believe that there is a mistake and it shouldn't be trimmed with trim:false.

[vadimdemedes]

Hey @stroncium, PR works great overall, there's only one issue I was able to find:

const chalk = require('chalk')
const figures = require('figures')
const wrapAnsi = require('.')

const str = [
	`${chalk.green('?')} ${chalk.bold('select packages to update')}`,
	'',
	`${chalk.yellow('other')}`,
	`${figures.pointer} ${figures.circle} standard-version ${chalk.green('devDep')} ${chalk.bold('MISSING > 5.0.0')} ${chalk.blue('https://github.com/conventional-changelog/standard-version#readme')}`,
	''
].join('\n')

console.log(wrapAnsi(str, 64, {hard: true, trim: false}))

Produces the following output:

Link is moved over to the next line, instead of being wrapped as well. Here's the expected output:

Would you be able to look into it?

[stroncium]

@vadimdemedes It actually isn't a regression, and it was there before and this behavior is locked in testcases. It boils down to following expectation failing:

t.is(m('AAA WWWWWWWW', 7, {hard: true, trim: false}), 'AAA WWW\nWWWWW'); // produces 'AAA \nWWWWWWW\nW'

// CC @sindresorhus
However, I looked into it and it actually seems to make more sense to do it that way. So I've implemented a simple heuristics on how to break words longer than columns in hard mode(minimizing number of breaks) using which would probably be a good idea.
It broke test breaks strings longer than "cols" characters(line 53) in part of exact breakdown of strings, so I propose to change it
Old expectation:

...
fox
jumpe
d
over
...

New expectation:

fox j
umped
over

Arguably, in this particular case neither option is looking quite pretty, but for real life purposes it should work good, and it fixes the problem stated above for example.

[vadimdemedes]

Works great, awesome work @stroncium!

[sindresorhus]

@stroncium Yes, the change makes sense to me.

[sindresorhus]

Really great work on this, @stroncium 🙌