What defines "awesome"?

[banesullivan]

The description in https://github.com/softwareunderground/awesome-open-geoscience/blame/master/awesome.md#L9 is vague to me... I'm curious if there is a checklist of what might make software "awesome" and ready to be added to this list?

I'm asking because I've seen a few projects show up in PRs that maybe aren't as awesome as some of the projects listed here.

Example

Compare GemPy (on the awesome list) and a proposed change like that in #74

GemPy is pretty awesome in my opinion - it has excellent documentation, a new-user-friendly README, lots of tutorials for new users to get up an running, and it's generally accessible (installable via a package manager like pip).

Meanwhile I’m struggling to find these aspects of the project in #74 which proposes adding MB-System. This software has a difficult to read/understand README, no API documentation (or at least after clicking around their website, I did not immediately see it), and no(?) existing examples for new users to get their hands dirty. Although it does appear to be generally accessible (installable using Homebrew)

[JesperDramsch]

In the beginning we left it intentionally vague, trusting the community to be self regulating. Now that the list has grown to what appears to be a critical site, I totally agree with your point that we may want to specify this further.

However, #74 also shows that our PR checklist is working quite well as a filter.

I would like to see a project with good documentation. That's something easy to check.

Installable... Well... How do we check? Right now @leouieda is doing a ton of work and I chip in when I can. I figure for both installing and trying out these submissions is near impossible.

Contributing guide like JOSS needs? Awesome projects are open to contributions, right? Or is that over the top?

[JesperDramsch]

How about tests as well?

[leouieda]

I agree with both of you that we should probably have clearer guidelines. What about adopting the JOSS criteria? We could start with those and modify the language a bit. This could be part of the PR checklist and should be included in the README somewhere.

From https://joss.readthedocs.io/en/latest/review_criteria.html#review-items

Software license

There should be an OSI approved license included in the repository. Common licenses such as those listed on choosealicense.com are preferred. Note there should be an actual license file present in the repository not just a reference to the license.

Acceptable: A plain-text LICENSE file with the contents of an OSI approved license
Not acceptable: A phrase such as ‘MIT license’ in a README file

Documentation

There should be sufficient documentation for you, the reviewer to understand the core functionality of the software under review. A high-level overview of this documentation should be included in a README file (or equivalent).

Installation instructions

There should be a clearly-stated list of dependencies. Ideally these should be handled with an automated package management solution.

Good: A package management file such as a Gemfile or package.json or equivalent
OK: A list of dependencies to install
Bad (not acceptable): Reliance on other software not listed by the authors

Example usage

The authors should include examples of how to use the software (ideally to solve real-world analysis problems).

API documentation

Reviewers should check that the software API is documented to a suitable level.

Good: All functions/methods are documented including example inputs and outputs
OK: Core API functionality is documented
Bad (not acceptable): API is undocumented

Community guidelines

There should be clear guidelines for third-parties wishing to:

Contribute to the software
Report issues or problems with the software
Seek support

Functionality

Reviewers are expected to install the software they are reviewing and to verify the core functionality of the software.

Tests

Authors are strongly encouraged to include an automated test suite covering the core functionality of their software.

Good: An automated test suite hooked up to an external service such as Travis-CI or similar
OK: Documented manual steps that can be followed to objectively check the expected functionality of the software (e.g. a sample input file to assert behaviour)
Bad (not acceptable): No way for you the reviewer to objectively assess whether the software works

[JesperDramsch]

I agree with both of you that we should probably have clearer guidelines. What about adopting the JOSS criteria? We could start with those and modify the language a bit. This could be part of the PR checklist and should be included in the README somewhere.

Including things in the PR checklist is really good, I agree.

From https://joss.readthedocs.io/en/latest/review_criteria.html#review-items

Software license

There should be an OSI approved license included in the repository. Common licenses such as those listed on choosealicense.com are preferred. Note there should be an actual license file present in the repository not just a reference to the license.

Acceptable: A plain-text LICENSE file with the contents of an OSI approved license
Not acceptable: A phrase such as ‘MIT license’ in a README file

I agree with OSI approved. But cheatsheets can just have the approved CC-BY tag I think.

Documentation

There should be sufficient documentation for you, the reviewer to understand the core functionality of the software under review. A high-level overview of this documentation should be included in a README file (or equivalent).

Agree. And it should be looking awesome.

Installation instructions

There should be a clearly-stated list of dependencies. Ideally these should be handled with an automated package management solution.

Good: A package management file such as a Gemfile or package.json or equivalent
OK: A list of dependencies to install
Bad (not acceptable): Reliance on other software not listed by the authors

Agree. Requirements and installable seem really good.

Example usage

The authors should include examples of how to use the software (ideally to solve real-world analysis problems).

I think that is like a bonus points point.

API documentation

Reviewers should check that the software API is documented to a suitable level.

Can we shoehorn that into general documentation?

Good: All functions/methods are documented including example inputs and outputs
OK: Core API functionality is documented
Bad (not acceptable): API is undocumented

Like it.

Community guidelines

There should be clear guidelines for third-parties wishing to:

Yeah that would clearly be awesome.

Cotribute to the software
Report issues or problems with the software
Seek support

Functionality

Reviewers are expected to install the software they are reviewing and to verify the core functionality of the software.

I think it should only be suggested if it's working.

Tests

Authors are strongly encouraged to include an automated test suite covering the core functionality of their software.

Probably a good idea... Didn't check for that yet.

Good: An automated test suite hooked up to an external service such as Travis-CI or similar
OK: Documented manual steps that can be followed to objectively check the expected functionality of the software (e.g. a sample input file to assert behaviour)
Bad (not acceptable): No way for you the reviewer to objectively assess whether the software works

I think it could be a bit more software agnostic, to invite other contributions like datasets and reference material.

[JustinGOSSES]

I like the JOSS suggestions. Suggest they should be clearly worded as guidelines of what "Awesome" looks like more than requirements for entry? Both due to the wider range of what is accepted here compared to JOSS and the fact that we probably want to be a step below JOSS in terms of what's allowed. Additionally, if people feel they have to do a lot of work to submit a change request to this repo, they're less likely to do so.

Maybe have these criteria under a separate "need more help determining if something is awesome, go here". That way they can be used for clarification, and back up when declining pull requests, but don't pour cold water on enthusiastic participation.

[JesperDramsch]

I've made a stab at it in #88. Would love for you folks to just go in and edit it to make it certified awesome.

We can then go on to modify the PR list, when we have this done. I see it being something like:

---

<!-- Thank you for contributing to our list! -->
<!-- Please write a DESCRIPTIVE TITLE for the pull request and commits. -->

Project name:

Project website/repository:

License:

<!-- If adding a project to the list, make sure it fulfills the following criteria. -->
<!-- An empty checkbox is [ ], a checked one is [x], you can also click them once submitted. -->

Checklist for new projects:

<!-- Make sure it's "certified awesome"! -->
- [ ] Project is [`certified awesome`](awesome.md)

<!-- General Requirements -->
- [ ] The project is [open-source](https://opensource.org/licenses/alphabetical) and accessible.
- [ ] Searched the existing entries to make sure this is not a duplicate.
- [ ] Contains only a single addition (make separate PRs if adding more than one).
- [ ] Added the link to the bottom of the relevant category.
- [ ] Created a new category only if necessary.

<!-- For Software -->
- [ ] Link the repo rather than the website/documentation
- [ ] Installation instructions present
- [ ] Documentation looks great
- [ ] Examples or Tutorials to follow
- [ ] Tests and Travis CI running

<!-- For Datasets, Cheatsheets and Publications -->
- [ ] Easily Accessible
- [ ] Comprehensive and Appealing

<!-- Formatting Critarie -->
- [ ] Add icon from the `media/icon/` folder if applicable (like `![Python](media/icon/python.png)` ).
- [ ] Checked spelling and grammar.
- [ ] Removed trailing whitespace and periods.
- [ ] Confirm the dash – is not a minus -.
- [ ] Used [title-casing](http://titlecapitalization.com) (AP style) for project name.

[gsakkis]

Can new or obscure projects (as in, with little real-world usage) that satisfy all/most other criteria qualify as "awesome"? If yes I'd like to plug tilesegy, a small Segyio-like project I published recently.

[JustinGOSSES]

I'm leaning yes? Would be great to hear opinions of anyone else too.

Can you add a bit more in the README about why someone would use it and what they would use it to do? That's probably a lot more obvious to you then others.