UltraJSON 2.0.0 release checklist

[hugovk]

Seeing as it's been over 4 years since the last release, and because support some EOL Python versions have been dropped in that time, I propose the next release should bump to version 2.0.0. Python 2.7 support should be retained for this one.

Some things that come to mind to make a new release:

• Move https://github.com/esnme/ultrajson to https://github.com/ultrajson/ultrajson

  • @cgbystrom Please could you do the transfer? Instructions here.

• Review and merge open PRs

  • Reviews welcome from everyone, please comment those which are most important

• Get access to https://pypi.org/project/ujson/

  • I've emailed previous maintainer Jonas to ask him to add @cgbystrom, @rstms and me to PyPI (same usernames)

• Get access to https://test.pypi.org/project/ujson/

  • @segfault I see you've been using this for experimenting, would you mind adding @cgbystrom, @rstms and @hugovk too?
  • https://test.pypi.org/manage/project/ujson/collaboration/

• Set up Travis CI to deploy to TestPyPI on merge to master (to test the release machinery) and to PyPI on tags

  • I can set this up, it depends on previous two items

• Anything else?

[segfault]

I'm happy to help. I've added @rstms and @hugovk to test.pypi.org. @cgbystrom needs to create an account on that site before they can be added.

[goanpeca]

Set up Travis CI to deploy to TestPyPI on merge to master (to test the release machinery) and to PyPI on tags
I can set this up, it depends on previous two items

I suggest we use Github Actions with supports Win, Mac and Linux and is nicely integrated with Github. A single file can setup and test everything.

I can help in this direction.

I would add to the list that wheels are needed! right now is one of the hurdles we are experiencing with Spyder and PyLS (this project is also used by many other IDEs)

We really want to help and contribute to bringing this project up to date so if you could also add me as contributor I could start the work on the CI.

Thanks!

[segfault]

I went through the PR listed when I forked and established the ujson-segfault build. It should have most of the mergeable PRs included and a functional build with wheels. I'm not tied to any of it, but hopefully it helps save some time.

[hugovk]

@goanpeca GHA would be great! As would wheels!

There's already a GHA workflow for testing here: .github/workflows/test.yml, and a PR to add a lint workflow in #354. So just a new thing for deploying is needed.

I don't have access to add anyone to PyPI or TestPyPI, @segfault please could you add @goanpeca to TestPyPI? Looks like same username. Then we get things moving.

For reference, the wheel building/uploading in @segfault's fork is done here: https://github.com/segfault/ultrajson/blob/master/.travis.yml

[hugovk]

We can discuss autodeploy in #355 (and there's an older #219 for wheels).

[cgbystrom]

I got access from Jonas. Added @hugovk and @rstms as maintainers on PyPI.

[hugovk]

I've created #357 to autodeploy to TestPyPI for on non-tagged merges to master, and to PyPI for tags.

I wanted to get something ready so we can get a release out soon, and I've used Travis for this before, but I'd still be very happy to change this over to GHA when ready. And it lays some useful groundwork for that.

It will generate an sdist and a bdist_wheel. I'm not sure exactly what sort of wheel it will create, I suggest we have a look at TestPyPI when it's merged, and we can decide whether to keep it, or remove it pending #219.

Wheels are important for UltraJSON, but I'd suggest we don't hold off an sdist-only release until we've got that set up. Even if we do an sdist-only release first, that's on par with the last release 4 years ago and we can make a new release (or just upload wheels) as soon as we're ready with that. (Release early, release often!)

---

@segfault Do you think you could delete the test releases at https://test.pypi.org/project/ujson/#history? It might be confusing to see 2.1.x versions when #357 will be putting 1.36.x and 2.0.x files there. Thanks!

[hartwork]

Just an idea:

Based on the commit message of eb7d894…

In addition to fixing our precision problems this will improve double
encoding by 4-5x. Decoding is however slightly slower according to the
benchmarks - but accurate at least.

…the upcoming 2.0.0 may have different characteristics with regard to both correctness and performance compared to ancestor 1.35.

I see a window in Git history that could make for a release 1.36 — in case you want to make some — that would be close to 1.35 and usable, e.g. in benchmarks.
The #346 link errors that trouble 1.35 are fixed starting 6cf6c7f and there are no link errors up to right before eb7d894 (where #277 link errors are introduced that PR #312 seems to be about to fix).

Here's a complete list of candidate commits — most recent on top — to base a 1.36 release on in case you want to make some:

• 2f1d487 — Skip OrderedDict test for < 2.7
• c9f8318 — Fix for incorrect order when using OrderedDict
• 3a6ba52 — Removed blist tests
• 7b5dc31 — New major version because of breaking changes
• 50181f0 — Removed serialization of date/datetime objects
• 5f98f01 — Removed support for __json__ method on str
• 53f85b1 — Removed generic serialization of objects/iterables
• ac4637f — Following std json handling of None dict key
• 409c6d4 — Fix for overflowing long causing invalid json
• 870ee48 — Fixes for sort_keys bug and a typo.
• ab6b6f8 — Merge pull request #217 from orivej/indent-dict-values
• d25e024 — Merge pull request #238 from WGH-/static-functions
• 6cf6c7f — added "static" to C functions, where possible

What do you think?

[hugovk]

Thanks for the suggestion, and for checking this.

My feeling is, that at this stage of the project, it would be simpler and better to move ahead with 2.0.0. I'm not sure of the benefits of making an incremental 1.35, as the earlier maintainers had already merged breaking changes (as seen by the major version bump in 7b5dc31), and efforts are best put in moving forward.

[goanpeca]

I also think moving to 2.0 would be the best course of action for bringing this project back to life :-)

[hugovk]

OK, looks like everything that can be merged right now is merged. We don't have wheels yet, but I think I think we're ready to do a release.

Let's say version 1.36.dev87 on TestPyPI is the release candidate, to be released as 2.0.0.

Please could you give it a go? If there's no major regressions in a few days, let's do the release.

• https://test.pypi.org/project/ujson/1.36.dev87/

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple ujson==1.36.dev87

---

Draft release notes:

Added

• Include license file in manifest (Include license file in manifest #288) @kszucs
• added "static" to C functions, where possible (added "static" to C functions, where possible #238) @WGH-
• bring back benchmark introduction (bring back benchmark introduction #234) @rdpate
• Add syntax highlighting to README (Add syntax highlighting to README #255) @adamchainz
• Test on new Travis CPUs (Test on new Travis CPUs #350) @hugovk
• Test on GitHub Actions (Test on GitHub Actions #348) @hugovk
• Test on Python 3.9-dev to avoid surprises (Test on Python 3.9-dev to avoid surprises #349) @hugovk
• Use Release Drafter to draft releases (Use Release Drafter to draft releases #365) @hugovk
• Deploy from GitHub Actions instead of Travis CI (Deploy from GitHub Actions instead of Travis CI #364) @hugovk
• Autodeploy to PyPI from Travis CI (Autodeploy to PyPI from Travis CI #357) @hugovk
• Lint with pre-commit: Black, isort, Flake8 (Lint with pre-commit: Black, isort, Flake8 #354) @hugovk

Changed

• Reduce default buffer on stack size (Reduce default buffer on stack size #281) @ncopa
• Update ultrajsondec.c (Update ultrajsondec.c #260) @markpiekarz
• Indent dict values (Indent dict values #217) @orivej
• Test with pytest (Test with pytest #363) @hugovk
• Less Travis (Less Travis #361) @hugovk
• Update links (Update links #356) @hugovk
• Tests: Remove unused, unmaintained blist (Tests: Remove unused, unmaintained blist #362) @hugovk
• Only deploy sdist from Travis CI (Only deploy sdist from Travis CI #359) @hugovk

Removed

• Drop EOL Python 2.5-2.6, 3.2-3.4 and add 3.7-3.8 (Drop EOL Python 2.5-2.6, 3.2-3.4 and add 3.7-3.8 #298) @hugovk

Fixed

• Fix reference counting bug for dict values (Fix reference counting bug for dict values #353) @richafrank
• Fix a couple of memory leaks. (Fix a couple of memory leaks. #257) @borman
• Correct JSON decoding implementation to die on trailing commas in Objects (Correct JSON decoding implementation to die on trailing commas in Objects #323) @timdawborn
• fixup dconv_wrapper linkage (fixup dconv_wrapper linkage #312) @tclh123
• Fix Warning: 'classifiers' should be a list, got type 'filter' (Fix Warning: 'classifiers' should be a list, got type 'filter' #351) @hugovk
• Remove redundant license parameter (Remove redundant license parameter #341) @blag
• do not forcefully remove the build directory manually (fix Install with --skip-build don't install C extensions #179) (do not forcefully remove the build directory manually (fix #179) #207) @anthraxx
• Merged pull request to fix memory leak. (API compatibility with other Python JSON libraries #1) @Adriandorr
• http -> https (http -> https #299) @hugovk

[goanpeca]

We don't have wheels yet, but I think I think we're ready to do a release.

This is the main reason why the Spyder team started the thread on the status of the project. So this release unfortunately will not solve our issues :-( (on windows)

[hartwork]

OK, looks like everything that can be merged right now is merged. We don't have wheels yet, but I think I think we're ready to do a release.

Sounds great! 🎉

Let's say version 1.36.dev87 on TestPyPI is the release candidate, to be released as 2.0.0.

Please could you give it a go? If there's no major regressions in a few days, let's do the release.

* https://test.pypi.org/project/ujson/1.36.dev87/

pip install -i https://test.pypi.org/simple/ ujson==1.36.dev87

Does not work (neither with 2.7 nor 3.7):

# pip install -i https://test.pypi.org/simple/ ujson==1.36.dev87
[..]
  ERROR: No matching distribution found for setuptools>=40.8.0
[..]

Prior pip install 'setuptools>=40.8.0' does not help.
My impression is that https://test.pypi.org/ is both broken and dangerous to use (..).
I would recommend testing with this command, instead:

# pip install git+https://github.com/ultrajson/ultrajson && python -c 'import ujson; print(ujson.loads("[1, 2]"))'

Draft release notes:
[..]

Looks pretty good to me 👍

[hugovk]

Thanks, let's double check the TestPyPI installation in #366.

[hugovk]

OK, this is the correct command, need to let it fetch other dependencies from the main PyPI:

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple ujson==1.36.dev87

Re: https://packaging.python.org/guides/using-testpypi/#using-testpypi-with-pip

I've fixed the command in #352 (comment). Thanks for checking!

[hartwork]

Sorry for cross-posting, but:
I would like to repeat that I advise against use of test.pypi.org for anything touching end-users, like test installs, for security reasons (..).

[KapJI]

Amazing to see this project resurrected! When are you planning to release the new version? Your checklist looks done 🙂

[hugovk]

🚀 UltraJSON 2.0.0 released!

• https://pypi.org/project/ujson/2.0.0/
• https://github.com/ultrajson/ultrajson/releases/tag/2.0.0

Thanks everyone!

Let's try and get wheels included for the next one: #219.