Trial Empirical Measurements: LB Solvers

[smiths]

Measure 3 to 5 LB solvers using the empirical measures available in GitStats. You should use the same 3 to 5 solvers as for #15. Think about the story that the data tells. Which of the metrics are meaningful? Which ones should we continue to collect for other projects?

[peter-michalski]

Meaningful metrics not visible on GitHub:
Authors
Total files and Total lines (may also be used to gauge avg file size) - maintainability
Commits per day - maintainability

\
\
\

Comments on My 5 qualities (bold signifies measured using GitStats, italicized signifies measured with a different tool or found in the repo):

Robustness - measured through manipulation of the system, repo stats will not measure this.

Performance - measured through documentation, other measures will not be nicely comparable between solvers

Maintainability - open/closed issues, update frequency, number of maintainers, age, total size (lines of code), number of files, avg file size, number of modules, percentage of commented lines, quality of comments, quality of naming conventions,

Reusability - avg file size, modularization, documentation_/naming/commenting(are modules and interfaces defined)_, quality of comments

Portability - number of external files needed to be manually added, os platforms, os versions

The metrics that we will need another tool for: percentage of commented lines - this can be gathered using https://github.com/boyter/scc

[smiths]

Thank you for your list @peter-michalski. I like your list. However, I'm not clear on how you are going to measure all of the italicized numbers. For instance, you mention quality of comments, and quality of naming conventions. Is there a tool to assess these?

The tool that you linked looks good. If we can get all of these measures from two tools, I think we'll be in good shape.

[peter-michalski]

@smiths unfortunately there is no tool for measuring quality of comments and quality of naming conventions. A score of some sort would have to be assigned by an expert after reviewing the repo.

I believe that we can get all required measurements using the two tools and information readily accessible in git repositories.

[peter-michalski]

I believe we have what we need from this issue and am closing it.

[smiths]

Sounds good.