Refactor Eex Tokenizer

[feliperenan]

This commit moves the line and column to the last argument to be handled as a metadata as you can see in the example below:

    assert T.tokenize('foo', 1, 1, @opts) ==
             {:ok, [{:text, 'foo', %{line: 1, column: 1}}, {:eof, %{line: 1, column: 4}}]}

[josevalim]

Looks great to me, we just need to get CI green! :)

[josevalim]

Ah, don't forget to update the docs at the top of EEx.Tokenizer. :)

[feliperenan]

Yeah. I'm trying to debug what I'm missing here to make it green.

Also, we are going to expose token(contents, opts), right?

[josevalim]

We will expose EEx.tokenize(contents, opts), yes!

[feliperenan]

@josevalim I've updated the docs and exposed Eex.tokenize/2. I couldn't figure what is missing to get tests green though, I'm still investigating, but let me know if you have any clues - I guess it is something silly I haven't changed 🤔

[josevalim]

@feliperenan can you also please change the tokenizer errors to be of the shape {:error, reason, meta}?

[feliperenan]

@josevalim Done!. Let me know if there is something else 🙌🏻

[josevalim]

💚 💙 💜 💛 ❤️