Rename PretokenizationPipeline and PosttokenizationPipeline

[FilipBolt]

PosttokenizationPipeline and PretokenizationPipeline sounds too similar to pipeline.py module and should be renamed to be more related to hooks.

Suggestions:

• PretokenizationHookManager / PosttokenizationHookManager
• PretokenizationQueue / PosttokenizationQueue

Further, hook code could be extracted to a separate module and unified using a general HookQueue interface that defines add_hook and clear methods to avoid copying of those.

[mariosasko]

I don't like the names ending with Manager because the logic that these components implement is too simple.

I think it's best to remove PosttokenizationPipeline and PretokenizationPipeline altogether and transfer all the logic inside Field and MultioutputField. Rn, Field.{add, remove}_hook just delegate the calls and it's fine to have a few lines of logic there.

The bigger issue is that MultioutputField is not a subclass of Field, so we are duplicating some logic.

[FilipBolt]

True, I would also like for MultioutputField to be somehow unified with Field (perhaps not directly as MultioutputField contains Fields).

I believe we had some offline discussion on this, and the way forward is to separate hooks out (and potentially rename them to some name, undecided yet), and leave Field and MultioutputField as it for the time being (as they require a larger overhaul).

[ivansmokovic]

I don't think MultioutputField should be looked at as a Field, as it's more or less just a wrapper to signify special handling during preprocessing.

[ivansmokovic]

Maybe simplify MultioutputField down to the bone, making it just a holder for fields/tokenizers and moving the logic into Field. This would allow us to remove the annoying [(name, (raw, tokenized))] return type for preprocess.

[mttk]

Why not just do the python thing and drop all suffixes, going with Pretokenization, Posttokenization?

[mariosasko]

Actually, I think the naming is not that relevant in this case. Both PosttokenizationPipeline and PretokenizationPipeline are only used internally by Field so keeping them private is the way to go imo.

[mttk]

I'd agree with @mariosasko 's previous comment as they are only used internally. We can underscore them and be OK with this.