Use a strategy pattern and inject encoder to process semantic tokens

[wildmaples]

Motivation

Related to: https://github.com/Shopify/ruby-dev-exp-issues/issues/518

As suggested by @paracycle on Slack:

So, I have a different opinion on this altogether: Both of these methods are only needed for the particular encoding that is needed to satisfy the LSP spec and are not (and should not be) a part of the SH processing. Both of these methods should be implemented as a decorator on top of the SH request handler that transforms the SH processed list of tokens to the proper encoding. That decorator class can be in an Encoder namespace, for example.

Because #compute_delta is one of the possible encoding for our semantic tokens, we can use a decorator dedicated to encoding to modify the tokens gathered from Requests::SemanticHighlighting.

This PR is to demonstrate the refactor to use the decorator pattern in the semantic highlighting request.

@paracycle Let me know if this was what you envisioned.

Implementation

The primary concern I have is that the requests are not set up to use the decorator pattern. I had to go against the request (BaseRequest) calling convention to make it work.

This is how a request is usually run:

Requests::SemanticHighlighting.run(document)

This is how SemanticHighlighting needs to be run using the decorator pattern:

Encoder::Relative.new(Requests::SemanticHighlighting.new(document)).run

---

Vini previously had a concern for performance, since this would mean we're doing a two pass on the tokens array. However, I think this pattern is more maintainable, understandable and would allow us to have better tests. I think we can optimize later on if it becomes an issue. Wdyt?

Automated Tests

This is a proof of concept. If this is 👍, there is still additional work to be done to refactor the tests. The SemanticHighlighting class can be tested separately from Encoder::Decorator, which is nice.

Manual Tests

N/A

[paracycle]

I like this refactor a lot, thanks for working on this. I've put in some comments to push the refactor forward and clean it up a little bit more.

[paracycle]

This looks really nice. One, final, round of comments.

[wildmaples]

There are still tests to be written for the encoder. I'd prefer to merge this PR first and make those changes in a separate PR, if you are all good with that.

[paracycle]

Looks great, thanks for working on this 🙏