Huggingface integration

[pacifikus]

Closes #83

[daavoo]

Thanks for the contribution @pacifikus ! It looks great.

Could you:

• Open a P.R. in https://github.com/iterative/dvc.org adding HuggingFace to the list of https://dvc.org/doc/dvclive/user-guide/ml-frameworks . You should be able to use some of the other pages as reference

[daavoo]

Is there a reason why the Callbacks that are part of HuggingFace use on_log instead of on_evaluate:

https://huggingface.co/transformers/_modules/transformers/integrations.html#CometCallback

This callback does look simpler than those other implementations, but I'm just curious about that

[pacifikus]

Thank you for your comment!

PR in dvc.org: iterative/dvc.org#2718

Yes, there is a reason why the Callbacks that are part of HuggingFace use on_log instead of on_evaluate. As you can see in https://huggingface.co/transformers/main_classes/callback.html#trainercallback, metrics are only available in the on_evaluate event. Thus, we can get the loss values both at the train and eval from on_log, but eval metrics only from on_evaluate.

The on_evaluate event has its disadvantage: we cannot get the training loss during this event, only the evaluation loss.

[daavoo]

Thanks for the explanation. Would it make sense to extend DvcLiveCallback to use both on_log (for train and eval losses) and on_evaluate (for eval metrics) ?

[pacifikus]

Yes, I think it's a good idea. I did it

[daavoo]

I think that the huggingface docs might be a little misleading. I just debug the test locally and it seems that the same values available as metrics inside on_evaluate are previously being passed as logs to on_log.

[pacifikus]

Yes, you're right, it's strange, but metrics logs are also written in on_log event. Thank you for your comment!

[pacifikus]

Hmm, now for different operating systems, different len(first(logs.values())) values in the test are obtained

[daavoo]

Well that is strange, I will debug locally the macOS test case which is the one failing

[daavoo]

Still don't understand what the bug was. If I have to guess, it might be related with first behaving differently in MacOS. Anyway I made the test more explicit and it now passes 🎉

[codecov-commenter]

Codecov Report

Merging #137 (c236003) into master (01c0513) will increase coverage by 0.43%.
The diff coverage is 100.00%.

❗ Current head c236003 differs from pull request most recent head 9b48440. Consider uploading reports for the commit 9b48440 to get more accurate results

@@            Coverage Diff             @@
##           master     #137      +/-   ##
==========================================
+ Coverage   90.00%   90.43%   +0.43%     
==========================================
  Files          12       13       +1     
  Lines         310      324      +14     
==========================================
+ Hits          279      293      +14     
  Misses         31       31              

Impacted Files	Coverage Δ
dvclive/huggingface.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 01c0513...9b48440. Read the comment docs.