Permalink
Branch: master
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
77 lines (57 sloc) 4.15 KB

Policy Simulation Library Project Criteria

Summary

  1. Models must be transparant.
  2. Interface recommendations fascilitate interoperability.
  3. Organizational recommendations fascilitate community.

Introduction

Open-source modeling ensures that model results are reproducible, that experts around the world can collaborate to make models better, and that users can explore the parameter space for themselves. These outcomes systematically improve public policy decisionmaking.

We have developed the criteria in this document to fascilitate the growth of an open source public policy modeling ecosystem.

Criteria

PSL establishes three kinds of project Criteria. A project are required to (MUST) conform to Acceptance Criteria to be accepted in PSL. Projects are recommended to (SHOULD) or optionally (MAY) conform to various Community Criteria and Interoperability Criteria.

Acceptance Criteria for Transparancy and Quality

  1. Models MUST be released under an OSI-approved open source license or the Creative Commons Public Domain Dedication (CC0).
  2. Data MUST be publicly available, unless release is restricted by a third party.
  3. For any data that SHOULD not be disclosed, provided MUST be:
    • A complete descriptive list of all data variables;
    • Descriptive statistics for all data variables for such data (including averages, standard deviations, number of observations, and correlations to other variables), to the extent that the descriptive statistics do not violate the rule against disclosure;
    • Contact information for the individual or entity who has unrestricted access to the data.
  4. At least one test MUST generate key outputs from source materials, the test MUST be run with every new version, and the outputs of the test MUST be checked into the repository.
  5. Projects MUST have unit tests and SHOULD report code coverage.
  6. Projects MUST report names and contact information for at least one maintainer.
  7. Projects MUST have a suggested citation.
  8. Projects MUST have a project overview.
  9. Projects MUST have installation directions.
  10. Project MUST be mirrored in the same GitHub organization as PSL, and therefore they MUST be under version control.
  11. Projects MUST use a consistent versioning scheme, which SHOULD be semantic versioning. If projects want to use the PSL Package-Builder tool to distribute packages via the Anaconda Cloud PSLmodels channel, there are additional MUST criteria.

Community Criteria

  1. Projects SHOULD have a public roadmap.
  2. Projects SHOULD have contributor documentation and guidelines.
  3. Projects SHOULD have regular office hours, webinars, or standing meetings.
  4. Projects SHOULD list technical contributors.
  5. Projects SHOULD list funders.
  6. Projects SHOULD list user citations and case studies.
  7. Projects SHOULD include subject matter tags, chosing from ...
  8. Projects SHOULD include a disclaimer.
  9. Projects SHOULD have a public issues tracker.
  10. Projects SHOULD have a changelog.
  11. Projects MAY have a Stack Overflow channel.
  12. Projects MAY include a "News" translation of the changelog for users.
  13. Projects MAY include criteria for participating in cross-model PSL initiatives.
  14. Projects MAY include a link to a webapp verison.
  15. Projects MAY include a list of consultants.

Interoperability Criteria

  1. The source code SHOULD be written in an open source language.
  2. All input and output variables SHOULD be documented like...
  3. All policy parameters and assumptions SHOULD be documented like...
  4. Meta information about your project SHOULD be documented like...
  5. A PSL_catalog.json configuration file to be used for cataloging these criteria MUST be included in the project's repository. Specifc instructions for creating this file can be found in the Catalog-Builder Documentation.