sustainable-pkg-stats

This is a set of scripts to get statistics on the scikit-surgery library and turn them into a nice webpage

Sustainability dashboard template for community

We provide a repository template for creating a sustainability dashboard for your own library/ecosystem of interest. It automatically creates the scripts and files needed to run the analysis needed for deployment of dashboard showing metrics of libraries existing within a given base Python package/ecosystem. It includes the Github Action that deploys the produced html files to the gh-pages branch of a target repository, which triggers a deployment every 12 hours, using the cron scheduler.

Using the template

1. You can use your preferred virtual python envirohment to install cookieninja.

pip install cookieninja

For instance, you can create conda or mamba environment with basic packages as follows

conda create -n susdbVE pip cookieninja -c conda-forge
conda activate susdbVE

2. Run cookieninja in the desired location

cookieninja gh:scikit-surgery/sustainable-pkg-stats

If you have this repo locally (this may be the case if you are developing, or you cloned this repository before), you can alternatively run the following:

cookieninja /path/to/your/checkout/of/python-template

3. A series of questions will pop up to configure the project. Type the answer or hit return to use the default option (shown in square brackets). Note that these project variables are defined in the cookiecutter.json file. It is important that you enter a value for base_library_name as the dashboard analysis scripts will be configured for this base package. There is a script the cookieninja runs placed under hooks/pre_gen_project.py that checks if the name given returns package entries in pypi search.

author_name [John Smith]:
author_email [temp@gmail.com]:
project_name [Community Dashboard]:
project_slug [dashboard_for_scikit-surgery]:
base_library_name [scikit-surgery]:
project_short_description [A dashboard template from scikit-surgery]:
funder [JBFC: The Joe Bloggs Funding Council]:
Select licence:
1 - MIT
2 - BSD-3
3 - GPL-3.0
Choose from 1, 2, 3 [1]:

4. This will create a directory with the following configuration: For example, for a project with the following variables:

project_name : Community Dashboard
base_library_name : scikit-surgery

We will get a project folder named after dashboard_for_scikit-surgery, structured like this:

├── assets
│   └── logo-dashboard.svg
├── _config.yml
├── get_badges.py
├── get_github_repos.py
├── get_loc.py
├── get_pypi_repos.py
├── html
│   ├── dashboard.html
│   ├── dashboard.html.in.head
│   ├── dashboard.html.in.tail
│   ├── excluded.html.in.head
│   ├── excluded.html.in.tail
│   └── exclusions.html
├── index.html
├── libraries
│   ├── exclusions
│   └── lines_of_code
├── LICENSE
├── loc
│   ├── CMakeCatchTemplate.html
│   └── PythonTemplate.html
├── pypi-simple-search
├── README.md
├── requirements.txt
├── sksurgerystats
│   ├── common.py
│   ├── from_github.py
│   ├── from_pypi.py
│   ├── html.py
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── common.cpython-310.pyc
│   │   ├── html.cpython-310.pyc
│   │   └── __init__.cpython-310.pyc
│   └── pypi_downloads.py
├── static
│   └── loc_plot.js
├── templates
│   ├── dashboard.css
│   └── loc_plot.html
├── tests
│   ├── conftest.py
│   └── test_template_workflow.py
├── update_dashboard.py
├── update_github_stats.py
└── update_pypi_stats.py

Important configurations to note:

1. get_github_repos.py and get_pypi_repos.py will take base_library_name as the base name to search packages in https://pypi.org/search/ and github
2. project_name will appear in the README.md as the human-readable name of the project.
3. html/dashboard.html will take project_name as the main title, Community Dashboard, and also use project_slug for a description below the logo, as shown below:

5. Create a new repository with the same project_slug name (e.g. https://github.com/$GITHUB_USER_ID/dashboard_for_scikit-surgery).
6. Few [optional] things to set before you can run the pipeline!

1. You can specify a list for the libraries you want to exclude from your dashboard deployment, under libraries/exclusions

   Similar to libraries folder, this (as shown below) has a dict entry for each package, such as in this example from scikit-surgery:

   libraries/exclusions

   ├── scikit-surgeryoverlay

   ├── scikit-surgerytorsosimulator

   └── scikit-surgeryvideoutils

   Each file entry (ex. scikit-surgeryoverlay) is a .json file that has : an obsolete key and a value that is a sentence describing why they are obsolete, such as: `{"obsolete" : "Became <a href='https://github.com/UCL/scikit-surgeryvtk'>sikit-surgeryvtk.</a>"}`

2. You can save the logo of your base package (a .svg file) under assets/logo-dashboard.svg for it to show up in your deployment header

7. Github Configurations a. You need to initialise github pages in your repository and set the deployment source from branch main: You can find the instructions here

You might need admin rights from your organisation to use your organisation's base name. You can also use your username as the domain.

Your configuration will need to look like this (In the example below, our domain name is the scikit-surgery organisation):

8. Setting up git repository. Push your first commit to your package repository

cd dashboard_for_scikit-surgery
git init
git add .
git commit -m "first commit"
git branch -M main
git remote add origin git@github.com:$GITHUB_USER_ID/dashboard_for_scikit-surgery.git
git push -u origin main

9. Running the pipeline on Github actions

The Github Actions workflow will run this pipeline, so you do not need to do anything.

10. Running the pipeline locally

Locally, you can check if the pipeline works correctly, by running the python scripts ordered and referenced in the Makefile file of this repository. Note for checking if things work properly. While running get_badges.py you should notice that under libraries folder, there are .json files of dictionary entries for each package.

To run the scripts locally, you need a personal access token for Github API generated from here Save it in the base directory in a text file named github.token Be careful not to commit the token file to your git repository. If you do it will be submitted to github in your publicly visible repository.

Instructions for developers

Clone repository

• (Optional) Generate your SSH keys as suggested here
• (Optional) GitHub CLI as suggested here
• Clone the repository by typing (or copying) the following line in a terminal at your selected path in your machine:

git clone git@github.com:SciKit-Surgery/sustainable-pkg-stats.git

Creating and activating the environment either with venv or conda

Using conda

conda create -n susdbVE pip -c conda-forge
conda activate susdbVE
pip install -r requirements.txt

Using venv

mkdir env
python -m venv env/
source env/bin/activate
pip install -r requirements

Running the pipeline in your local machine

Running the pipeline that generates dashboard.html and associated files needed by Github Pages

bash Makefile

You can also run the individual python scripts to check outputs:

Search for relevant packages on pypi and githib

python get_pypi_repos.py
python get_github_repos.py

update stats

python update_pypi_stats.py
python update_github_stats.py

get coverage/docs/etc badges

python get_badges.py

update html files

python update_dashboard.py

Inspect libraries with pypi

./pypi-simple-search scikit-surgery > scikit-surgery-onpypi.txt
python get_github_repos.py > scikit-surgery-ongithub.txt

We can use pypinfo to get data for things on pypi

pypinfo --auth snappy-downloads-3d3fb7e245fd.json
pypinfo scikit-surgeryvtk country