use sedlex for utf-8 aware lexing (closes #5163)

[nadako]

As we discussed on Slack, I updated my sedlex branch from #5163 (comment). The only thing changed in the lexer since then is the "cache friendly debug line calculus" and the following fix by @ncannasse which were straightforward to add here as well.

[nadako]

@andyli Hey, it looks like our ocaml setup on Travis/Linux is not opam-aware, so I have no idea how to install sedlex there? Could we have opam there?

[andyli]

It's a bit complicated.
Ubuntu Trusty do not have ocaml 4.02, so I've backported one and put it in ppa:haxe/ocaml such that we can install it without compiling from source.
opam, similar to other ocaml lib/bytecode program, only works with the exact ocaml version. That means we cannot use the opam Ubuntu Trusty package. I looked into backporting one just like how I did for ocaml 4.02, but opam itself has a lot of dependencies that need backporting too...

I will see what I can do.

[andyli]

Turn out we can still install the opam binary and let it use the system ocaml by adding the system param at the end as follows:

wget https://raw.github.com/ocaml/opam/master/shell/opam_installer.sh -O - | sh -s /usr/local/bin system

And then all we need to do is to

opam install sedlex
opam config exec -- make

Tested it locally and it worked well.

[nadako]

Thanks, I'll try adding that!

UPD: that worked!

[Simn]

I'm fine with merging this as soon as the #6143 people are in a good place.

[nadako]

Rebased the branch and force-pushed to trigger Travis/Appveyor.

[nadako]

I'm fine with merging this as soon as the #6143 people are in a good place.

It's not clear to me what to do to move this forward...

[Simn]

Basically ask @hughsando if he can run opam install sedlex.

[Simn]

Your .merlin change conflicts.

[nadako]

Fixed.

[Simn]

I'll go ahead and merge this now. Let me know if there are any problems.