Roadmap post 1.0

[Evizero]

I think its time to revisit this package again. It has been around for a while now, which also means that some of the design aspects are overdue for a bit of cleanup.

• Port documentation to Documenter (done in Port documentation from Readthedocs to docstrings + Documenter #100)
• make Losses support broadcasting via Ref (done in Port documentation from Readthedocs to docstrings + Documenter #100)
• Add benchmarks to track regressions
• use new Broadcast which supports lazy to reduce code complexity
• Rename AvgMode to AggMode (aggregate mode) Rename "Average Modes" #91 (done in Rename AvgMode to AggMode #104)
• Integrate use of label encodings Handling non-float targets #87
• Separate MLMetrics.AvgMode from new LossFunctions.AggMode

---

About: Separate AvgMode from AggMode

Basically I realised that what we call AvgMode right now is used to describe two orthogonal problems.

1. LossFunctions.jl: It is used to specify if a loss should be

   • computed element-wise with AvgMode.None (i.e. same shape as outputs), or
   • aggregated somehow to one number (AvgMode.Sum, AvgMode.Mean), or
   • aggregated somehow to one number but with observation weights (AvgMode.WeightedSum, AvgMode.WeightedMean, etc),
   • aggregated somehow to a vector (one number per observation) (e.g. AvgMode.Sum + ObsDim.Last)

2. MLMetrics.jl: It is used to specify if an aggregated metric (precision, recall, etc) should be

   • computed per class with AvgMode.None (i.e. one number per class), or
   • micro averaged to one number (with AvgMode.Micro), or
   • macro averaged to one number (with AvgMode.Macro)

Both of these scenarios should also support weighting observations and weighting classes. Now in the first case for LossFunctions.jl, we do class re-weighting using the loss itself, and observation weighting using the special AvgMode. I am not yet sure how to address this for the second case, but the interface should be similar

With all this in mind it makes sense to split the two problems into an AggMode for the first and an AvgMode for the later. While AggMode would be placed here in LossFunctions, the new AvgMode would be placed in MLMetrics

[juliohm]

Thank you @Evizero for sharing the roadmap. From what I understand most items have been accomplished, and only the items regarding benchmarking and label encodings are missing. The latter one is somewhat outdated with the efforts in CategoricalArrays, and so we are only left with infrastructural issues. I will close this issue, and update the roadmap moving forward.