New release (2.2.0)

[EwoutH]

I would suggest tagging a new release soon. Changes since 2.1.2 have made the code base more readable, added the annotation of plot_tradeoff graphs, removed a lot of dependency warnings (which add noise when running the code), have improved the testing and release process and improved documentation. I provided a list of changes below.

Ping @quaquel.

2.2.0 Changelog

Functional

• prim.py: Make PrimBox.peeling_trajectory["id"] int instead of float #121
• analysis: Allow optional annotation of plot_tradeoff graphs #123
• evaluators.py: Allow MultiprocessingEvaluator to initialize with cpu_count minus N processes #140
• 6d83a6c

Code quality

• Improve code quality with static analysis #119
• a83533a

Dependency depreciations

• feature_scoring: fix Regressor criterion depreciation #125
• feature_scoring.py: Change max_features in get_rf_feature_scores to "sqrt" #129
• 8b8bfe4
• 255e3d6
• e8fbf6f
• b5a9ca9

Testing and CI

• CI: Use Pytest instead of Nose, update default build to Python 3.10 #131
• Release CI: Only upload packages if on main repo #132
• CI: Split off flake8 linting in seperate job #133
• CI: Add weekly scheduled jobs and manual trigger #137

Documentation

• Docs: Installation.rst: Add how to install master or custom branch #122
• 28569bd
• 28569bd
• Docs: Replace all http links with secure https URLs #134
• ebdbc9f

[quaquel]

Yes, I agree.

Would also be good to have a place for this changelog somewhere in the documentation.

[EwoutH]

Maybe a changelog.rst file, and include it in the ReadTheDocs documentation?

[quaquel]

see ebdbc9f for a first step towards this. The basic setup is now in place. There is a changelog.md which is automatically included within the documentation when it is being built. The next steps are adding some compare links to GitHub and converging on headings within the changelog. I used a standard template for now but I am open to discussing these. For example, I find the Jupyter changelog also quite useful and easy to understand.

[EwoutH]

I think for clarity you should split up a commit like ebdbc9f into multiple commits the next time. Most is about the changelog, but a lot of other stuff is also happening, like cleanup and documentation in cart.py, unrelated requirements in requirements.txt, and the removal of a tutorial. To keep a clean and usable commit log, those should al be separate commits.

Aside from that,keep a changelog is a nice format. However, if we're going to use a semantic versioning changelog, we should also adhere to semantic versioning itself. Since we have introduced new functionality, the next tag should be 2.2.0.

Last point, I would like to get #140 in before the next release, if that's okay with you. I added docs, but maybe we should also add a few example of it's possible usage somewhere.

[quaquel]

I would like to adhere to semantic versioning, so you are right that the next version will be 2.2.0. Next to #140, I would at least also like to address at least #124 and possibly #104.

[EwoutH]

The GitHub Milestones is a very easy built-in tool in GitHub that lets you assign issues and PRs to certain release targets. Creating a Milestone for the 2.2.0 release (and 2.3.0) might provide a nice overview of what still needs to be done.

[EwoutH]

Would a 2.2.0 tag today or this weekend be possible?

There is no shortage of minor (or any) tags btw, we can go to 2.9999.0 if we want.

[quaquel]

I still need to finalize addressing #124. Once that is done, going to 2.2 is fine with me.

[quaquel]

Last step before this can be closed will be to fill in the changelog and thus agree on the format.

[EwoutH]

I think the most important question if we want to sort by change type (added, removed, etc.) or by change category (Feature, Docs, testing, etc.). Both have their advantages and disadvantages.

I think it's also nice if we can include the changelog in the Readthedocs somehow.

Maybe we can ask that each commit/PR contains a changelog entry from v2.2.0 onward.

[quaquel]

The changelog is already automatically included in the read the docs: https://emaworkbench.readthedocs.io/en/latest/changelog.html. This was part of the large commit you complained about.

I like the basic idea of including a changelog as part of a commit or PR. However, often a change is spread over several commits. For example, I have already started changing some stuff in the callback and related unit tests as part of a future change in how data is being stored in the workbench. These are non-user-facing changes and would be thus not as relevant to cover in a changelog.

I like the jupyter lab changelog which is a bit of a hybrid between change type and change category. They use change type for the API, and change category for docs, testing, building and distribution).

[EwoutH]

I like this style proposed used by JupyterLab! I think something like that might also work for us.

[quaquel]

That would give us something like the following. Still need to link in all the relevant commits.

new features added

• PrimBox.inspect now can take an integer or list of integers. In the case of a list of integers, the table or graph will be shown for each item.
• PrimBox.inspect now has a third option, labeled 'data', which returns the statistics and box_lims.

enhancements made

• new keyword argument in show trade_off for annotating points with their ID
• MultiprocessingEvaluator now also excepts a negative integer in which case the number of workers is the max number of cores - the provided number
• MultiprocessingEvaluator will throw an exception if you try to start more workers than the maximum number of available cores.

bugs fixed

• Make PrimBox.peeling_trajectory["id"] int instead of float
• MultiprocessingEvaluator now starts at most 61 processes for Windows OS. For details behind this check concurrent.futures

maintenance and upkeep improvements

• shift to using pytest instead of nose for running unit tests
• CI: flake8 linting is moved into a separate job
• CI: weekly scheduled job and manual triggering of jobs
• Release CI: only upload packages if on the main repo
• test_callback has been automatically reformated with unittest2pytest. All other tests still need reformatting
• scikit-learn requirement has been updated to >= 1.0.0
• setup.py: Add project_urls for documentation and issue tracker links
• code quality improvements using static analysis

documentation improvements

• installation page now also explains how to install master or custom branch
• all links in the documentation have been updated to use https instead of http

contributors to this release

[EwoutH]

I was looking into how to give this the lowest maintenance workload for keeping a changelog. I think GitHub now has some nice features to automatically get the relevant PRs and list them in a category based on a label. See Automatically generated release notes.

We once need to define a file .github/release.yml which list label corresponds to which category. For example, the new-feature label would correspond to new features added, etc.

Then for each new release, this process would go as follows:

1. Label each PR with the fitting label
2. Draft the release through GitHub Releases
3. (optionally) add commits that weren't added through a PR (maybe we need to stop with direct commits to the master branch)
4. Add descriptions for PRs that need some extra explanation or are worth spotlighting
5. Copy the release notes over to the changelog

This eases the workload between versions to only labeling the PRs, instead of continually needing to adjust the changelog. What do you think?

[quaquel]

Thanks for looking into this. Indeed, anything that makes our life easier is welcome. I now manually did a diff with the latest tag to see all commits and next manually sorted them into the right categories. The more of this process can be supported, the better.

One question, however, is how creating a new tag interacts with the GitHub action that uploads to PyPI. At the moment, once I commit a new tag, this triggers a GitHub action resulting in a new release on PyPI. Logically speaking, it would make sense to separate the tag from the actual release. So, add the tag to trigger the creation of new release notes. Curate this into a human-readable form and add it to the changelog. Next, create the release which then triggers the action to upload to PyPI.

Moving away from direct commits to the master would not be a bad idea. I would need to force myself to do development in separate branches, and when merging a branch into master, I can probably add a fitting label and descriptive comment.

[EwoutH]

@quaquel I updated the draft release notes (in GitHub Releases) with the commits in the list above. I also wrote a highlights section, let me know what you think of it!

If it's good, you can publish it. GitHub will then create a tag automatically, and if all goes well, the CI will upload the package to PyPI.

We can then also add the notes to the CHANGELOG.md.

[quaquel]

I slightly modified the highlights and published it. It seems however that this does not yet create the tag that triggers uploading to CI, so we either need to modify CI to use releases instead of tags, or figure out what is going on. I would personally be fine with using releases instead of tags. Also, we still need to do the changelog.

[EwoutH]

I looked into it, and the workflow worked fine. We only forgot to change the version number, so it was published as 2.1.3!

In #159 I updated the version number (in intit.py) and changelog, and moved the tag using git tag --force to trigger the CI again (not best practice, but since no code change were made okay I think). PyPI now succesfully published 2.2.0.

I will include not forgetting to bump the version in the releases part of the contibuting.md docs.

Congratulations on the 2.2.0 release, I think this issue can be closed!

[quaquel]

thanks for looking into this. I recall that there also might be a way of automatically updating the version number with a new release, so that might be worth checking at some point.

[EwoutH]

There are tools like Versioneer that do this indeed. Good suggestion, I will look into it!