release v4.4.0

[jameslamb]

Release checklist:

Copied from #6277, with a few changes.

before merge

• Update VERSION.txt number.
• Update version in Appveyor config file.
• Update version in configure file of R-package: /gha run r-configure.
• Change development.mode from unreleased to release in pkgdown config file.
• Update version in python-package/pyproject.toml
• Add release branch to RTD versions, trigger a new build, check docs
• All docs for new behavior have Sphinx versionadded:: annotations (docs on those)
• All new parameters in config.h have *New in version {version}* comments added
• Run the valgrind checks with /gha run r-valgrind (docs link)
• test Python and R packages on arm64 Mac

Items to be addressed before merging:

• [R-package] always name the shared library 'lightgbm', not 'lib_lightgbm' #6432
• [c++] remove uses of '..' in headers #6409
• [python-package] clarify max_depth warning and limit when it is emitted #6402
• [python-package] Allow to pass early stopping min delta in params #6274
• [python-package] NumPy 2.0 support #6454
  • [python-package] Add support for NumPy 2.0, test against nightly versions of dependencies (fixes #6454) #6467
• [R-package] ensure use of interaction_constraints does not lead to features being ignored #6377
• [R-package] skip integer categorical feature check when building dataset subset (fixes #6412) #6442
• deprecation warnings for these:
  • [RFC] remove 'categorical_feature' and 'feature_name' parameters in cv() and train() #6435
  • [RFC] remove HDFS support? #6436

I'd really like for this to be the first release with arm64 macOS wheels, if we can get there. But let's not delay the release waiting on that if everything else above is done.

• [ci] [python-package] add arm64 macOS wheels #6391

after merge

• Push v* tag to trigger GitHubRelease action at Azure Pipelines.
• Update stable tag at GitHub.

code to update tags (click me)

NEW_VERSION="4.4.0"
git checkout master
git pull upstream master
git fetch upstream --tags
git tag -d stable
git push upstream :refs/tags/stable
git tag stable
git tag "v${NEW_VERSION}"
git push upstream stable "v${NEW_VERSION}"

• Add new tag to RTD versions and trigger a new build.
• Remove the release branch release/v{n}.{n}.{n} from RTD versions

after the release is created by Azure DevOps job

• Add release notes from release-drafter to the automatically-created GitHub release, convert it from Draft to published.
• Upload release to CRAN
• Upload release to test PyPI
• Upload release to PyPI.

code to publish to PyPI (summary)

mkdir /tmp/lgb-release
cd /tmp/lgb-release

NEW_VERSION="4.4.0"

# NOTE: requires a GitHub personal access token with "repo" scope
gh release download \
    --repo microsoft/LightGBM \
    --dir ./artifacts \
    --pattern 'lightgbm*-py3-*.whl' \
    --pattern "lightgbm-${NEW_VERSION}.tar.gz" \
    "v${NEW_VERSION}"

# NOTE: manually downloaded arm64 wheels from CI

# config docs: https://packaging.python.org/en/latest/specifications/pypirc/
twine upload \
    -r testpypi \
    ./artifacts/*

(gh is the GitHub CLI, see https://cli.github.com/manual/gh_release_download)

Then confirmed that installing the latest wheel works.

pip install -i https://test.pypi.org/simple/ "lightgbm==${NEW_VERSION}"
python ./examples/python-guide/logistic_regression.py

Then pushed them to real PyPI.

twine upload \
    ./artifacts/*

• Update cran-comments.md when new release is accepted on CRAN.
• Upload release to NuGet.
• Update version and commit hash in Homebrew formula (docs)
• put up another PR incrementing development version here in GitHub (example: bump development version to 4.1.0.99 #6090)

code to open that dev version PR (click me)

RELEASE_PR_NUMBER="6439"
RELEASE_VERSION=$(cat ./VERSION.txt)
DEV_VERSION="${RELEASE_VERSION}.99"

git checkout -b ci/dev-version

echo "${DEV_VERSION}" > ./VERSION.txt
sed \
    -i .bak \
    "s|${RELEASE_VERSION}|${DEV_VERSION}|g" \
    .appveyor.yml

sed \
    -i .bak \
    "s|version = \"${RELEASE_VERSION}\"|version = \"${DEV_VERSION}\"|g" \
    ./python-package/pyproject.toml

sed \
    -i .bak \
    's|mode\: release|mode\: unreleased|g' \
    ./R-package/pkgdown/_pkgdown.yml

docker run \
    --rm \
    -v $(pwd):/opt/LightGBM \
    -w /opt/LightGBM \
    ubuntu:22.04 \
    ./R-package/recreate-configure.sh

git add \
    ./.appveyor.yml \
    ./R-package/configure \
    ./R-package/pkgdown/_pkgdown.yml \
    ./VERSION.txt \
    ./python-package/pyproject.toml

commit_msg="bump development version to ${DEV_VERSION}"
git commit -m "${commit_msg}"
git push upstream ci/dev-version

gh pr create \
    --repo microsoft/LightGBM \
    --base 'master' \
    --label 'maintenance' \
    --title "${commit_msg}" \
    --body "now that v${RELEASE_VERSION} has been released: #${RELEASE_PR_NUMBER}"

Notes for Reviewers

I believe this should be v4.4.0 instead of v4.3.1 because of some small breaking changes:

• [cmake] [c++] use .dylib extension unconditionally on macOS #6418
• ENH more stable gradient of CrossEntropy #6327
• [R-package] Rename slice() to lgb.slice.Dataset() #6293

See https://github.com/microsoft/LightGBM/releases.

[jameslamb]

@shiyu1994 @guolinke @jmoralez @borchero @btrotta to help with the release, could you please do the following?

• comment here if you want to change anything about the proposed list of issues/PRs to be addressed before merging
• comment on the open Request for Comment issues:
  • [RFC] remove 'categorical_feature' and 'feature_name' parameters in cv() and train() #6435
  • [RFC] remove HDFS support? #6436
• share any other thoughts here about this release (e.g. the version number, the timing, the testing plan)

Thank you!

[borchero]

Plan sounds great to me, thanks a lot for taking care of the release @jameslamb!

I think that v4.4.0 makes sense as the next version since we're introducing a few new features :)

[shiyu1994]

Thank you @jameslamb for the releasing. I don't think we have anything urgent to be included in v4.4.0 for now.

[celestinoxp]

Please add verbose fix in this new version...

[jameslamb]

I've added #6454 to the list of issues to resolve for this release. I'll prioritize that this week.

Added that because numpy 2.0 will be released June 16th: numpy/numpy#26191 (comment). I think we should try to have a lightgbm release out by then that's compatible.

[jameslamb]

Alright I think we are close!

I'm sorry for the rush, but could we please try to finish the release in the next 3 days, so we could hopefully put up a release of the Python package prior to numpy 2.0 going out on June 16th?

I think we can get it done.

I can do all of the release tasks to get this PR ready, but could use some help on reviews of the following:

• [python-package] Add support for NumPy 2.0, test against nightly versions of dependencies (fixes #6454) #6467
• [ci] [python-package] add arm64 macOS wheels #6391

@guolinke @shiyu1994 @borchero @jmoralez if you have time in the next few days, could you help with reviews on those?

While that's going on, I'll continue trying to help @jmoralez unblock R CI in #6442, and I can review that PR and #6377.

[jameslamb]

/gha run r-valgrind

Workflow R valgrind tests has been triggered! 🚀
https://github.com/microsoft/LightGBM/actions/runs/9499960462

Status: success ✔️.

[jameslamb]

test Python and R packages on arm64 Mac

LightGBM now has a CI job that does this on every commit (as of #6391).

I tested the R package on my M2 Mac tonight, and put up an issue documenting the need to add a CI job in the future: #6481

[jameslamb]

Alright I think this is ready!

@guolinke @shiyu1994 @jmoralez @borchero could you please take a look when you have time?

Once at least 2 people have approved (and at least one of @guolinke and @shiyu1994 ), I'll merge this and proceed with the release (pushing tags, publishing release notes, etc.).

[borchero]

Nice, thanks for getting this over the line @jameslamb! Sorry for being inactive the last few days... 😬

[jameslamb]

Sorry for being inactive the last few days...

Don't worry about it @borchero ! We're mostly all volunteers here, with availability that varies week-to-week. Appreciate all your help on the other contributions that led up to this release being almost-ready today 😁

[jameslamb]

Thanks everyone! Whenever CI passes here, I'll merge this and continue with the release.

[jameslamb]

Ran the following to create the v4.4.0 tag and update the stable tag.

git checkout master
git pull upstream master
git fetch upstream --tags
git tag -d stable
git push upstream :refs/tags/stable
git tag stable
git tag v4.4.0
git push upstream stable v4.4.0

(NOTE: I alias this repo to upstream and my fork to origin in my git settings)

https://github.com/microsoft/LightGBM/tags

That triggered an Azure DevOps build which should create the release automatically: https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=16384&view=results

This takes around 90 minutes (because of the QEMU CI job).

I'll check back in a few hours and continue the release then.

[jameslamb]

Add new tag to RTD versions and trigger a new build.
Remove the release branch release/v{n}.{n}.{n} from RTD versions

These are done.

• build: https://readthedocs.org/projects/lightgbm/builds/24700459/
• docs: https://lightgbm.readthedocs.io/en/v4.4.0/Parameters.html

And v4.4.0 in the selector:

[jameslamb]

Add release notes from release-drafter to the automatically-created GitHub release, convert it from Draft to published.

The release is now published here on GitHub:

https://github.com/microsoft/LightGBM/releases/tag/v4.4.0

[jameslamb]

Submitted to CRAN:

[jameslamb]

Uploaded to test PyPI and tested installing from it (including running lightgbm's tests) on my M2 Mac. That worked!

So published to real PyPI.

v4.4.0 is now up on PyPI

[jameslamb]

put up dev version PR: #6485

[jameslamb]

upload release to NuGet

done: https://www.nuget.org/packages/LightGBM

[jameslamb]

The R package has passed all checks at https://cran.r-project.org/web/checks/check_results_lightgbm.html

[jameslamb]

Update cran-comments.md when new release is accepted on CRAN.

Done in #6499

Update version and commit hash in Homebrew formula (docs)

This was done in Homebrew/homebrew-core#174635

---

And with that, this release is complete! Thanks everyone!!