refactor: make TransformersDocumentClassifier output consistent between different types of classification

[anakin87]

Related Issues

• fixes TransformersDocumentClassifier: inconsistent output between ordinary and zero-shot classification #3167

Proposed Changes:

TransformersDocumentClassifier output structure was different between ordinary and zero-shot classification.
Now the output is consistent. For doc.meta['classification'] we always have a structure like this:

{"label": "love",
"score": 0.9608993530273438, 
"details": {"love": 0.9608993530273438, "joy": 0.032584577798843384, ...}}}

After the discussion in #3167, I decided to keep the score attribute in the first level of doc.meta['classification']: I think that it can be useful for filtering.

How did you test it?

Manual verification.
I can add some tests if you think that's the case.

Notes for the reviewer

Other refactoring aspects:

• replaced the attribute return_all_scores with top_k: it mimicked the same attribute of the HF pipeline, which is now deprecated in favor of the latter.
• simplified batched predictions accumulation
• removed document classifier from a test, where it was unused

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added tests that demonstrate the correct behavior of the change
• I've used the conventional commit convention for my PR title
• I documented my code
• I ran pre-commit hooks and fixed any issue

[ZanSara]

Hey @anakin87 ! Code looks good! Can you add a small test to ensure that the details field is there in both cases, and that it gets populated with the right amount of labels specified on top_k? Once that is done, this is ready to merge 👍

[anakin87]

Hey @ZanSara!

I added two tests, since:

• for ordinary classification, the number of labels in details is determined by top_k
• for zero-shot classification, the number of labels in details is just the number of labels specified in the constructor

[ZanSara]

Thank you! I found a couple of improvements but it's already good to go. I'll go ahead and approve, feel free to commit or resolve the suggestions as you see fit. I'll merge it around EOD or tomorrow morning.