Measures 3 to 5 LB Solvers

[smiths]

Following the template at:

https://github.com/adamlazz/DomainX

Measure 3 to 5 LB Solvers. While doing this, critically evaluate the previous template. Think about what we can add and what we can take out.

This file explains the entries in the template:

https://github.com/adamlazz/DomainX/blob/master/TemplateToGradeSCSoft.pdf

You might also find this spreadsheet helpful:

https://gitlab.cas.mcmaster.ca/SEforSC/mmsc/-/blob/master/DomainX/ProcessForSoftwareReview/SoftwareGrading.xlsx

The DomainX repo has several examples:

https://github.com/adamlazz/DomainX

[Ao99]

Some Q&A during the meeting:
Q: as for the 3-5 packages, do we choose from the most promising ones, or choose randomly?
A: pick more promising ones, but at least try 1 or 2 randomly.

Q: do we choose only the ones from GitHub, or also include the ones on BitBucket and SourceForge?
A: it depends on whether the empirical tools are git-specific or GitHub-specific, because the latter two might also use git. Besides, it's also OK to include 1 or 2 packages which cannot be analyzed by the empirical tools.

[peter-michalski]

Thank you for posting this @Ao99

[smiths]

Yes, thank you for posting this @Ao99.

[peter-michalski]

Results of 4 solvers in commit c4e4f15

[peter-michalski]

Comments on the old template in 37dbcff

[peter-michalski]

Updated results in 857e435

[peter-michalski]

Updated LUMA and sailfish comments 4810360

[smiths]

The measurements in the old template look good. I can tell that you have spent time investigating the software. Were you able to install all of the examples you selected? Maybe the template should have a specific entry for whether the installation was successful?

[smiths]

Comments on the old template in 37dbcff

Thank you for the comments on the old template @peter-michalski. We will use those when we revise the template.

[peter-michalski]

@smiths Of the 4 I had initially set out to install:

• 2 worked after a repeat attempt
• 1 did not work due to the requirement of NVIDIA CUDA GPU - I confirmed this with developers
• 1 does not appear to be working correctly (when trying a test case) - The test case is not clear. Reaching out to the developer did not help. The developer pointed out that a separate tool needs to be independently developed in Visual Studio in order to view the visual aspects of the output. Even non visual output does not appear to be working. I would mark this as an installation success but test case failure.

I did a shorter "test" of an additional 6 solvers simply trying to install them and run a case without filling out all of the details of the template. I did this to get a better feeling for how difficult running these programs may be for this domain:

• 2 worked without help from developers
• 2 I have decided to label dead - Not maintained, poor/no docs, difficult to figure out how to install/run, no clear way to contact devs, ignored issues in the repo asking the devs simple questions. These were smaller independent projects anyway.
• 1 worked with additional documents sent from developers
• 1 is not working, reached out to a developer but no response, professional looking project so do not want to discard/mark dead yet

[smiths]

Thanks for the update @peter-michalski. I'm glad that you are investigating this. Installability is definitely an important issue, especially in your domain. I'm glad that you have some of them working. 😄 I'm sure you are already doing this, but keep your notes on who you have contacted and how much time you have spent.

Our goal has been to measure qualities (like installability), but you should keep track of ideas you have for improving installability. This might be an avenue for us to explore when writing up your report/paper.

[peter-michalski]

I have added notes on installability in 5caad72 and 5811a47.

I will mark this issue closed.